The real story behind the 360/PS3 failures

Deleted User wrote: »

any chance of a copy and paste for us in work?

System Failure: Why PS3 and 360 drop dead for the same reasons

August 29th, 2009

In a week where Xbox 360 production boss Aaron Greenberg stated that Microsoft's hardware issues were "well behind us", I found myself finally facing up to the notorious unreliability of the older 360 consoles, and attempting to do something about it.

In many ways, this feature is an off-shoot of a personal story. Readers of the Digital Foundry Twitter feed will know that both of my retail Xbox 360s died of RROD in quick succession. While Microsoft was nice enough to send me a new PAL "Jasper" model, swapping out my prized NTSC 360 Elite, originally purchased for a Eurogamer hardware test would be much more difficult. Over and above the luxuries of having the ability to play region-locked games, the concept of binning off two consoles that cost me the best part of £600 was basically wasteful and unacceptable. Something had to be done.

Looking for a more permanent resurrection for my unit, I'd heard that the best fix involved the rather manly-sounding process of "reballing" the GPU - resoldering the joints from the motherboard to the graphics chip. That being the case, I found myself at my nearest independent console workshop, Colchester Computers, staring at their impressive BGA rework/reflow station. Working from an industrial unit just off the Essex town's bizarre Magic Roundabout, this was an interesting opportunity to find out from the experts first-hand why the average games console ceases to function, and how they are fixed. Upon arrival at the workshop, the vast stack of dead consoles up against one wall ("spare parts") was somewhat eye-opening.

Talking to the company's engineer, Darren Thickbroom, it instantly became apparent that many of the Internet truths surrounding the console failures were anything but, and that the heat dissipation issues that plagued every revision of the Xbox 360 up until the most recent Jasper version were hardly exclusive to the Microsoft console. Slowly but surely, just like its competitor, the issue of PlayStation 3 reliability is being brought into question.

While the scale of the so-called YLOD issue is difficult to judge in context of the all-pervasiveness of RROD, the fact is that what I learned on my visit was pretty shocking: whether you own a Microsoft or Sony console, it seems that the act of simply using our consoles for the job they were designed can cause cumulative damage, with the very real danger that our games machines may go "pop" after the manufacturer's warranty expires.

"Your Xbox might last two or three years - it's as much down to the environment as the hardware itself," says Darren Thickbroom. "You open up some machines and you can understand why it might have broken down or over-heated, maybe there's tons of dust or fluff that came in from the intake. It depends from person to person, we speak to some people over the phone who've been through three to five units since the launch period. If you're a serious player, I'd recommend you change or upgrade your Xbox after a year."

Thickbroom also deals with many dead Xbox 360s that have had the so-called "X-Clamp" fix and reckons that it is essentially a complete waste of time - something worth bearing in mind if your out of warranty 360 suddenly bites the dust and you fancy tackling it yourself. "The Internet" has decreed that the clamps, attached to the base of the motherboard and securing the heatsinks, do the job too effectively, causing the motherboard to warp in concert with the heat generated by the CPU and graphics chip. However, over four years into the lifespan of the machine and many hardware revisions later, the clamps remain in Microsoft's design, and Thickbroom will replace any of these homebrew fixes with the original securing mechanism. The blame lies elsewhere, he reckons.

"It's just the general design and the heat factor," he says. "Everything's combined into such a small space, the heatsinks on the GPU are relatively small, there's a lot of heat to dissipate and it can't do it. The trapped heat warps the boards and that's what causes the problems over a long-term period."

Based on Thickbroom's experience, it would seem that the entire X-Clamp replacement industry is effectively a waste of money. With the DIY procedure, the real "fix" comes from the process of reflowing the solder after the clamps are replaced. In effect, you remove the fans from the 360, and allow it to massively overheat. Let it cool down again and in most cases service, post-RROD, will be resumed. But unless you're particularly lucky, it is a short-term fix, if it works at all. Reflowing is a precision job requiring precision tools, and the DIY method is akin to attempting to paint the Mona Lisa with fingerpaints. Professional repair shops will use a somewhat more involved method to get the job done.

"We remove the heatsinks, clean off any thermal compound left on the boards, cleaning the GPU and CPU," says Thickbroom. "Then we apply a BGA-based gel flux around the GPU and pre-heat this under a dark infra-red base BGA re-balling machine, which pre-heats the boards up to a set profile temperature, keeps that heat, then increases it again to a uniform temperature that the solder reflows at. When the reflow occurs, the flux is introduced to it. When the flux gets to the point that it becomes active, it fixes the problem caused by the heat: dry joints and poor connections are resolved."

Et Voila, my black and white 360-shaped doorstops are back in action as fully armed and operational games machines, fit for whatever abuse I would care to put them through in the Digital Foundry lair.

Whether you have RROD or its single-light variant, E74, it's the same root cause that is responsible - poor heat dissipation from the GPU - and the same fix is applied. What is intriguing however, is that it does appear as though the initial release of Xbox 360 has proved to be - on average - more reliable than some of the later models.

"From my personal experience, having seen so many machines come through, the original release machines from 2005 to 2007, before they brought out the HDMI revision - they tend to be more solid a unit," says Thickbroom. "They tend not have so many recurrences on the recall rate if it doesn't fail after 6-7 weeks after it's been put through the reflow process."

As the heat damage is seemingly cumulative, Microsoft's RROD woes may have been amplified by newer machines dying at the same time as the older ones. Certainly in my case, my vintage PAL November 2005 Xbox died within weeks of the NTSC Elite I bought that was 17 months younger. This is in spite of an addition to the heatsink in the newer units that pipes air into a vacant chamber elsewhere within the unit, and also with the introduction of packaging designed to hold the GPU in place and help prevent motherboard warping. It suggests that the quality of the Xbox 360 motherboard itself declined even while cooling potential was increased.

The core problem remains however: too much power crammed into too small an area.

"I just think that with all of these machines, the power and the heat they produce have long-term effects on the units," says Thickbroom, referring to both PlayStation 3 and Xbox 360. "It's also down to the solder being used on the units: it's a lead-free based solder. The consistency and quality of the joints with lead-free isn't as good as a proper lead-based solder. It's the law now, huge companies manufacturing these machines need to use lead-free, so the long-term reliability of the connections isn't so good."

While Xbox 360 in its earlier iterations has a pretty terrible reliability record, it is swiftly becoming apparent that the same core issue is also affecting the PlayStation 3. It is perhaps not surprising. While the Sony console has a whopper cooling system, the design of the RSX, especially in its original 90nm form, is to put it frankly a beast. Under the metallic heat spreader on the motherboard, you'll find not only the GPU, but also the 256MB of GDDR3 RAM. The 90nm RSX is much larger than the 90nm Xenos - indeed, it appears to be marginally larger than the Cell CPU in the launch units. The cooling challenge, especially in the launch units, is going to be considerable. Additionally, the GPU itself has remained on the 90nm fabrication process right up until the release of the new PS3 Slim, so assuming the problem is GPU-centric, it could potentially affect all the current "fat" models.

There's also the fact that both consoles are deployed in a huge range of different environments in gamers' homes. That being the case, it is almost impossible for the platform holders to ensure that the systems stay cool enough in all scenarios, especially bearing in mind the dust and fluff build-up that can occur over time. Certainly though, if you stick your console into a closed cabinet, you won't be doing it any favours. Smokers are more likely to end up with dead machines too - the tobacco finds its way into the console, making the innards sticky and thus more attractive to dust and debris that comes in through the intakes.

In terms of the scale of the problem, and failure ratios versus the Xbox 360, it is very difficult to put a number on just how many PS3s are malfunctioning. In the case of a relatively small-scale operation like Colchester Computers, working on average, they'll get 20 dead consoles to fix each day - 12 of them will be Xbox 360s, eight of them will be PlayStation 3s. But that's an average. As Thickbroom says, "sometimes, in a hectic week, we can have entire palettes of consoles coming in."

In terms of failure rates, the 60/40 split between Xbox 360 and PS3 they experience is remarkable in that it does prove pretty conclusively that both consoles are having exactly the same issues, especially when the methodology for fixing them is effectively identical. But beyond that, the figures are too isolated to tell us much more as many additional factors need to be taken into account: the installed UK bases of both systems, the fact that the damage is cumulative over time (and Xbox 360 is a year older, remember) and also the fact that 360 has a three-year warranty, while the PS3 is limited only to one year. We can assume that machines under guarantee will not arrive at Thickbroom's establishment, which specialises in extended warranty work with the likes of Argos and catalogue companies, in addition to dealing with the public directly.

"We really do see a lot of the 60GB launch PS3s which are a couple of years old now. Generally I think the faults there are down to wear and tear," he says. "We see a slightly smaller ratio of the newer 40GB machines with the smaller motherboard, but they still suffer from the same issues."

While we can assume that the newer, smaller 65nm GPUs in both the Jasper version of the Xbox 360 and the new PS3 Slim will help to reduce the instances of console death, the fact is that there is an installed userbase worldwide in the region of 50 million units. And this presents a very real issue for a fault that is seemingly caused by cumulative damage: every day more and more Xbox 360s out there are no longer covered by the safety net of Microsoft's three-year warranty, and many of them will fail through no fault of the owner. More than that, the extremely limited one-year PS3 guarantee seems to be woefully inadequate.

Back in December 2008, around nine months after the launch of PS3 in Europe, SCEE big cheese of the era David Reeves pegged the PS3 as having a two to three per cent failure rate. But since the core issue is a cumulative one, and presumably unforeseen by the platform holder, what would it be now? I asked SCEE to comment, and await a reply.

As it is, right now, if your 360 is out of warranty, an official Microsoft repair will cost £78, which bizarrely, rises to £95 if you organise it over the phone. Sony's Careline wasn't quite so clear-cut, saying that repairs are evaluated on a case by case basis, eventually stating that a replacement refurbished unit will set you back £128. With those costs in mind, an independent repair on your own machine will typically set you back around £60 for a 360 and £70 for a PS3. You can of course go for an extended warranty but typically these tend to be under-written by insurers and the time taken to get a working machine back can be variable. An independent, like Colchester Computers, tries to get your unit turned around in 24 hours.

Clearly, the PS3 warranty situation is of concern, and there's a mammoth amount of reading to sift through in this Eurogamer thread, which in its latter pages coughs up an interesting perspective. Following a piece on the BBC's One Show, there is a powerful legal argument that suggests that the length of the manufacturer's warranty is irrelevant in face of the all-powerful Sales of Good Act. Maybe - just maybe - if enough people make a fuss about this, Sony will re-evaluate its current one-year coverage so people don't have to jump through hoops to ensure a reasonable lifespan for their premium-priced games machines.

In the meantime, it's good times for independent console-fixers like Darren Thickbroom and Colchester Computers. If his theory about the cumulative effect of heat damage is correct - and the evidence available suggests it is - his business has a very rosy future. And certainly, for the hardcore gamer, he has some pretty stark advice.

"It comes down to how much you play it," he says. "People might disagree with us, but this is the way we see it: if you handle the machines day-in, day-out with the issues they've got, I'd expect to change them every year."

Whether the advent of PS3 Slim and the Jasper revision of Xbox 360 with their smaller, cooler chips will solve the issue remains to be seen, but producing machines that pump out less heat and suck less power from the mains is clearly the way forward.

But the fact that both Microsoft and Sony have these heat-related problems suggests that the problem isn't only related to the manufacture of "shoddy machines" as Sony's David Reeves once put it. It also means that the next generation of consoles - which will inevitably see a return to larger, more power-hungry silicon - are going to require some ingenious design solutions to prevent the same thing happening all over again.

brian plank wrote: »

the person in this video makes some very interesting points about the whole thing. worth a look. :cool:

Jip wrote: »

The video you posted can hardly be described as an independent assessment whereas the Eurogamer one is.

LIGHTNING wrote: »

The problem is that our megadrives run a 7.6 mhz CPU that needed no cooling, a xbox/PS3`s GPU and CPU pump out enourmous amounts of thermal energy that needs removing. I really doubt we will have alot of working current gen consoles in 20 years time.

Niall0 wrote: »

If you buy a console that was poorly designed and stops working because of design flaws why should you have to pay to fix it even if it was 3 years after you bought it, its ridiculous.

I think the law should be changed to protect consumers angainst poorly constructed merchandise.