The Guardian: The Truth About the Xbox 360

quad_red

http://www.guardian.co.uk/technology/2008/nov/13/xbox-truth

The truth about the Xbox 360

A hurried launch, a desire to win the console war and design compromises lay behind the 'red ring of death' debacle that cost Microsoft $1.15bn

Dean Takahashi
guardian.co.uk, Wednesday November 12 2008 16.15 GMT
The Guardian, Thursday November 13 2008
Article history

But will they all work? Photograph: Elaine Thompson/AP

The infamous Xbox 360 "red ring of death" (indicating a failed unit) has caused Microsoft - and its customers - untold pain in the three years since the console's launch in 2005, and cost it $1.15bn (£738m) last year. Microsoft has never said publicly why the console was plagued with faults: it seems that poor production quality was at the heart of the failures - an all-round problem with no single cause except impatience on the company's part as it tried to become the leader in videogame consoles.

It was an ambitious attack. Microsoft's engineers started working on the Xbox 360 at least a year after Sony's engineers began work on the PlayStation 3, yet wound up shipping a year earlier. With the first Xbox, the company lost $3.7bn (£2.3bn) over four years, mostly because costs of the box - particularly its hard drive - were too high. Bill Gates didn't really care about the losses; that was simply the ante for getting into an exciting new business. But Steve Ballmer, who took over from Gates as chief executive during the first generation, really wanted the Xbox business to be profitable second time around.

Software-hearted

Even though early testing showed that production machines had flaws, Microsoft didn't delay the launch because it believed the quality problems would subside. "They got enamored with the idea of the Microsoft army rolling everything out at the same time," says one source. "Their thinking shows that they are a software company at heart," says one veteran manufacturing executive. "They put something out and figure they can fix it with the next patch or come up with a bug fix."

In the hurried design process, Microsoft decided late to add a hard disk drive, and then wireless controllers. The hard drive blocked airflow on one side of the machine; the wireless modules had to have enough space to avoid electrical interference. The console shell was poked full of holes to ensure airflow. In the end, the machine was a series of compromises.

"It turned out in the end that this was all going too far, too fast," says a source. "They were adding too many features after things were locked down. That incremental feature adding just made it fragile."

Some of the defects were latent, potentially not showing up for some time after the machine was used. Up to 50% of all defects can be latent. And production yields - the number of machines coming off the production lines that passed testing - were low. In August 2005, the machines' aggregate defect rate - from Microsoft's contract manufacturers Flextronics and Wistron, in their factories in China - was allegedly just 68%.

In a memo dated August 30 2005, the team reported overheating graphics chips, cracking heat sinks, cosmetic issues with the hard disk and the front of the box, underperforming graphics memory chips from Infineon (now Qimonda), a problem with the DVD drive - and more. At that point, the contract manufacturers were behind schedule and had only built hundreds of units when they were supposed to be making thousands every week.

"There were so many problems, you didn't know what was wrong," says a source. "The [test engineers] didn't have enough time to get up and running." Shutting down production to debug everything properly might have delayed the launch in Europe or Japan.

Microsoft responded to this story (in full at Venturebeat.com) with a statement that it has already acknowledged an "unacceptable number of repairs" to Xbox 360 consoles and had responded to the hardware failures with a free replacement program. "This topic has already been covered extensively in the media," the statement said. "This new story repeats old information, and contains rumors and innuendo from anonymous sources, attempting to create a new sensational angle, and is highly irresponsible."

I have tried to verify the facts. Microsoft has never disclosed its actual return rates. But according to data obtained by VentureBeat, the total number of returns climbed above 1.2m consoles early last year. That is a huge figure, considering Microsoft had only shipped 11.6m consoles to stores by the time of the writeoff in mid-2007.

The warning signs were present even before Microsoft shipped any machines. In August 2005, as Microsoft was gearing up production, an engineer said: "Stop. You have to shut down the line." When production results are really off-kilter, stopping a line and tracing a problem back to its roots is the answer. But in this case, the decision was made to carry on.

Nobody listened to that engineer, apparently because console launches are always hurried affairs. Yields generally start low. As the manufacturers analyse data and tighten controls on each assembly step, they learn how to drive the yields up.

On November 22 2005, the Xbox 360 went on sale. Consumers could run graphically beautiful games on high-definition TVs. But machines started failing almost immediately. At the time, the company said it had received "isolated reports" of console failures and that returns were within the normal range.

According to the Consumer Electronics Association, the average return rate for products where the consumer gets their money back is about 2%. Microsoft's internal data assumed that in the long term 6% or 7% of the consoles would be defective before shipping into the market - a yield of 94% or 93%. The rate of return was expected to be low as well. But even after 2005, Microsoft struggled to ship enough units. The yield was typically only around 70% - far short of the target - until May 2007.

Problems denied

By the end of March 2006, Microsoft said it had shipped more than 3.3m consoles to retailers. But there was a growing "bone pile" of more than 500,000 defective consoles in a warehouse at Wistron and a repair centre in Texas - either duds off the factory line or returned boxes, according to sources. Production yield was climbing, but far too slowly. Meantime, Microsoft stood by its statement that returns were within "normal rates".

The denial of widespread problems infuriated customers such as Chris Szarek, who has had five bad consoles: he felt Microsoft was stonewalling. But as production ramped, Microsoft sent replacement units out more quickly and dealt with those who were unhappy about the shortage. In September 2006 Microsoft conceded that the quality of the consoles made during 2005 wasn't as high as it expected and said that it would extend the policy of free replacement for consoles made during 2005, even though the warranties had expired. But what was actually wrong with the machines? In July 2007 Robbie Bach, president of Microsoft's entertainment and devices division, said of the launch that "we were confident the product was ready. We did a lot of testing. The problem that shows up with the three red lights on the console is a complex interaction with some very complex parts."

There was no single reason for the failures - though many could be blamed on the graphics chip, which could overheat so much it warped the motherboard. This stressed bad solder joints, causing them to fail early in the machine's life - and the console was also one of the first products that had to meet new environmental standards in Europe prohibiting the use of lead in solder. Paul Wang, a Microsoft test engineer on the Xbox 360, said in a speech in 2007 that lead-free solder created a lot of problems.

Wii takes the lead

Sometimes the heatsinks on top of the GPU were put on the wrong way, resulting in overheating. Games would sometimes crash because of sub-par memory chips from one of the two suppliers. Problems with the DVD drive lasted longer than expected. Eventually, in January last year, Microsoft shut down manufacturing of the console and didn't build any more machines until June.

Early this year, SquareTrade, which sells warranties for electronics, reported that was seeing a 16.4% failure rate for Xbox 360s. Michael Pachter, an analyst at Wedbush Morgan, estimates that the number is 3m, about 15% of the total.

In the meantime, Nintendo's Wii has outstripped rival consoles around the world, having sold some 35m machines since its launch at the end of 2006. Microsoft has sold about 22m Xbox 360s since its 2005 launch, while the Sony PS3, which came late to the party with a European launch in March 2007, has sold 16.6m consoles around the world.

But is the war really lost? Shane Kim, head of the game division's business development efforts, recently said that it was too early to call the console war in favour of Nintendo. We won't know the winner, he says, until somebody sells 100m units. Microsoft's plan is to sell 75m Xbox 360s. The only way it could ever do that is to be more aggressive on its pricing. And by doing the right thing for its customers from now on.

mika27

I bet that guy owns a Wii

Jip

I've read several stories on the whys etc but the comment below I don't really understand

quad_red wrote: »

....cosmetic issues with the hard disk ...

Batesy

I think what he's trying to say is that with the hard drive being added late on it obstructs air flow that end of the unit.

ROC1977

talk about pick out stats that make the 360 look like its failing. He never mentioned once that 360 owners buy 7 times as many games as Wii owners, and a few times more than PS3 owners. Or the fact that the Wii was only out selling the other consoles due to its low price. And now the 360 is the cheapest machine its numbers are again up ahead of sony and nintendo.
Or has he mentioned that PS3 sold so many unit just as a cheap blue ray player.
Not a good report as it bias.

quad_red

ROC1977 wrote: »

talk about pick out stats that make the 360 look like its failing. He never mentioned once that 360 owners buy 7 times as many games as Wii owners, and a few times more than PS3 owners. Or the fact that the Wii was only out selling the other consoles due to its low price. And now the 360 is the cheapest machine its numbers are again up ahead of sony and nintendo.
Or has he mentioned that PS3 sold so many unit just as a cheap blue ray player.
Not a good report as it bias.

Cut the fanboy crap.

I don't see the relevance of 360's having a higher software attachment rate than Wii owners with the issue discussed in the above article. Or that PS3 sales are being driven by it's ability to play BluRay movies.

The article is on Microsoft's appalling disregard for quality control and the catastrophic results.

Don't get me wrong - I own a 360 and from a gaming point of view, there is little to tempt me to the other consoles.

But I've been at the ****ty end of the 360 hardware 'experience' and I don't think the article is biased at all.

MayoForSam

Good article, just saying it like it is. MS have had to learn the hard way about hardware (accountants and marketing guys overruling engineers, so what's new? :rolleyes:). It will be interesting to see how the next gen machine is put together.

I'm a bit confused over the 16.4% failure rate, surely a poor design with multiple faults would mean that the vast majority of consoles (older versions especially) will fail ultimately?

Biro

Exactly. It's a poorly designed and built piece of hardware, and M$ know it.
Maybe they should sub-contract out the manufacturing of the next console to some hardware manufacturer. As was pointed out in that article they're not a hardware company. Joining forces with Dell could be sucessful!

Jip

Biro wrote: »

....M$ .....

I'm so tired of people doing this. Microsoft are a company out to make money for the benefit of themselves and their shareholders, they're not a charity.
Why not just replace an 'S' in every companies name with the dollar sign, they're all the same.

Kintarō Hattori

Batesy wrote: »

I think what he's trying to say is that with the hard drive being added late on it obstructs air flow that end of the unit.

I don't think so. The author of the piece mentioned that there were problems with air flow and then seperately mentioned that there was a cosmetic issue with the hdd.

Batesy

Oh, I thought he meant that by adding the HD late on and hence changing the cosmetic look of the unit it led to airflow/overheating issues.

ROC1977

quad_red wrote: »

Cut the fanboy crap.

I don't see the relevance of 360's having a higher software attachment rate than Wii owners with the issue discussed in the above article. Or that PS3 sales are being driven by it's ability to play BluRay movies.

The article is on Microsoft's appalling disregard for quality control and catastrophic results.

Don't get me wrong - I own a 360 and from a gaming point of view, there is little to tempt me to the other consoles.

But I've been at the ****ty end of the 360 hardware 'experience' and I don't think the article is biased at all.

Fanboy?? Don't thinks so. I own all next gen system, and had all previous gen systems. I even had a NES and master system (the first real console war! lol)
As for the hardware failures, yes the first versions of the 360 massive problems which we all know about. But that report is hardly new news. I've had two 360 with the RROD back when they came out in 2005. But they have resolved most of the teething problems.
And the relevance is clear. Its relevant because the report mentions sales figures. And Ps3 sales figures are BR driven. If they were game driven they' be selling more games.
And its bias because it highlights the 360's faults, and fails to mention that Sony and Nintendo have had problems of there own. Don't forget that the Wii had world wide shortages, and the Ps3 was delayed by 6months only to be released as half the Ps3 Japan, and US got. Less Usb ports, no backward compatibly, and a smaller HDD, all for more than US and Japan paid. And not to forget the firmare update that bricked Ps3's.
Lead free solder in 2005 had to be used in Xbox 360 consoles because the US government decided that the 360 was classed as a toy, and no toys are a loud to contain lead. I don't think it was to do with an environmental law. But I could be wrong with that.

Yes the 360 was a poorly designed machine in 2005-06 but has improved loads since then, once they added a 3rd heat sink and changed suppliers.

Xbox and Dell are already partners, but no word on Dell building the next gen Xbox as of yet. If I hear of any news on that I'll be the first to post.

mossym

come on, this bit about adding the hard drive late has to be rubbish. the original xobx had a hard drive, microsoft were already promoting their download services, why on earth would they take a step backwards and decide initially not to go with a hard drive?

i don't buy it for a minute

and anyone that does any amunt of soldering will tell you, lead free solder is a PITA to work with, i can believe it causing major problems(I work for a semiconductor company and work on PCB's all day. the switchover to PB free caused us no end of heartache either)

that article is just an anti-microsoft rant. I'll e the first to criticize microsfot for the level of problems they've had, but can't agree with that article

mossym

ROC1977 wrote: »

Lead free solder in 2005 had to be used in Xbox 360 consoles because the US government decided that the 360 was classed as a toy, and no toys are a loud to contain lead. I don't think it was to do with an environmental law. But I could be wrong with that.
.

sorry, but you are. The EU decided that nothing containing lead could be shipped to any customer, toy or not ( the semiconductor company i worked for had to go through thousands of devices to ensure none contained lead. none of these products could be called a toy. even Printed circuit boards with the IC's that we make had to be lead free, meaning each and every component, plus the solder used to assemble it) the EU was the first to bring this in, and it was an enviorenmental law, so for once we can't blame the americans.

http://www.pb-free.info/

silvine

I bought an Elite last year and it suffered from RROD so they've still failed to deal with the issue adequately.

It's obvious to most people the Xbox is a shoddy piece of hardware. It looks cheap and sounds like a hoover. The PS3 on the other hand actually feels and looks like and expensive well built kit. Even the power converter is built into it.

The Wii is a completely diffferent console to the the 360 and the PS3 which is why all this 'they won the console war' malarky doesn't wash. Wii Sports is not exactly Halo 3 now is it? There's room for more than one console.

USD 1.15Bn. Serves Micro$oft right to be honest. A lot of people were screwed over. It's just not acceptable for firms to knowingly sell defective goods. All the same roll on Xbox 720.

Jip

silvine wrote: »

.....Micro$oft .......

Grrrrrr, $ony, Wii $port$, $ega.
See, you can do it will all companies !

ROC1977

mossym wrote: »

sorry, but you are. The EU decided that nothing containing lead could be shipped to any customer, toy or not ( the semiconductor company i worked for had to go through thousands of devices to ensure none contained lead. none of these products could be called a toy. even Printed circuit boards with the IC's that we make had to be lead free, meaning each and every component, plus the solder used to assemble it) the EU was the first to bring this in, and it was an environmental law, so for once we can't blame the Americans.

http://www.pb-free.info/

Ya thought I was wrong.I remember reading about lead free solder when my first 360 for RROD. All electronic devices had to have these changes by July 2006. But the law was passed in 2002.

One thing is lead free solder needs a higher temperature to melt. So wouldn't that have made the 360 a more reliable machine? X clamps as we all know seem to be the main reason for RRoD but not all the time. I recently had a friend ask me to fix his RRoD, but when I opened the 360 I found that some of the RAM had come completely off the m/board.

5uspect

The absence of lead makes lead free solders more brittle. There are new solders currently in testing (down the hall from me infact!) that approach lead solders mechanically but they'll never be as good as lead. Melting the solder isn't the problem, when a board gets hot its complex geometry will induce various strains on the board due to thermal expansion. This will fatigue the solder joints and over time they will fail. Additionally cutting corners on a poorly designed cooling solution is a bad idea. I doubt the presence of the harddrive in any way affects cooling, the case seems pretty well vented. More than likely they're using substandart thermal paste (pads probably!).

I recently bought a 25% off 20GB xbox in tesco. I was considering a wii for a long time but, typical for nintendo, third party games haven't taken off. Its a noisy, barebones, vendor locked PC for €180 and second hand games are cheap. Can't complain.

ROC1977

Ya the thermal paste is very poor on the 360 I replaced the thermal paste with high grade arctic silver we use on Dell systems.

5uspect

Its a spectacularly stupid thing to skimp on. A good TIM like Artic Silver will transform a cooling solution.

mossym

i guess though if the board is flexing under heat, then the cooling compund used will have little effect, it's proably become unattached anyway

5uspect

Not quite, the chip would be connected via a socket, either with gold pins or a land grid array (like intel). If the board flexes the socket should mechanically allow for that and the chip and heat spreader will certainly not flex due to their thickness. Bit what will fail are packages soldered directly to the board, things like graphics memory.

Bear in mind that the flexure is small, but the solder balls holding a chip onto it are also tiny.

Here is an image of a 360 mobo. On the left it looks like a CPU sans heat spreader. Other than that its a standard enough looking die. To the right is the GPU which is surrounded by the video memory. Its the video memory and other packages soldered directly to the board that will fail.

mossym

i would have thought that with a closed design like the 360 that the cpu was soldered onto the board. the only reason for sockets on standard motherboards is so that the chips can be changed/upgraded. no reason for that in a 360. we work with bga's here every day and use a mixture of socket to allow changing and soldered parts


and the problem is that those flaming xclamps which hold the heatsinks on bend, so if they go the heatsink will move away from the package and the thermal paste won't be in contact with both. at least that is what most of the self fix guides for an xobx will tell you anyway

silvine

Grrrrrr, $ony, Wii $port$, $ega.
See, you can do it will all companies !

Hold on, hold on, I've got another! It's $tarcraft. See here for the why: http://www.boards.ie/vbulletin/showthread.php?t=2055396890&page=2

gloobag

Biro wrote: »

Joining forces with Dell could be sucessful!

Trust me, they'd only make it worse! Anyways, they're getting out of the hardware game themselves.

Fingleberries

Jip wrote: »

Grrrrrr, $ony, Wii $port$, $ega.
See, you can do it will all companies !

Damn it, Nintendo were soooo smart not to put any 'S' in their name. They show their knowledge of years of expertise in the Console field...
Hmmm. How about Nint£ndo? or Nint€ndo?? ... Nah, it'll never catch on

mossym

hmm,,just thought after posting above. cpus aren't in bga are they?

but wasn't there a special cpu made for the 360? a custom proc? easy enough to put in bga package then

humanji

silvine wrote: »

Hold on, hold on, I've got another! It's $tarcraft. See here for the why: http://www.boards.ie/vbulletin/showthread.php?t=2055396890&page=2

You're not one to point fingers $ilvin€ :P

ROC1977

gloobag wrote: »

Trust me, they'd only make it worse! Anyways, they're getting out of the hardware game themselves.

And what game would they be getting into then?

robinph

Fingleberries wrote: »

Damn it, Nintendo were soooo smart not to put any 'S' in their name. They show their knowledge of years of expertise in the Console field...
Hmmm. How about Nint£ndo? or Nint€ndo?? ... Nah, it'll never catch on

Nint¥ndo surely?

John Sheehy

Jip wrote: »

I'm so tired of people doing this. Microsoft are a company out to make money for the benefit of themselves and their shareholders, they're not a charity.
Why not just replace an 'S' in every companies name with the dollar sign, they're all the same.

Microsoft made a faulty product and sold it all the same. Their extended warranty only covers RROD and there have been cases of disc drives that eat games or won't work where Microsoft will do nothing. Their customer care is generally poor and DRM (like how I have to be online after my fourth xbox bricked to play my arcade games) is also pretty retarded.

So if he wants to satirize Microsoft by substituting an S for a $, I'm not going to disagree with him. (also, I don't think producing a faulty product and losing millions to replacements and repairs is for the benefit of Microsoft or their shareholders...)

F-Stop

Interesting article, and I've found this a really informative thread with a lot of interesting input. Ironically, I've never had a problem with the two xbox 360s I've owned, but my Wii's graphics chip overheated and the unit was replaced (admittedly Nintendo's customer care was painless). It's clear that MS knew there would be problems but want to aggressively grab as much of the market as they could while Sony was between consoles, so to speak. Though I doubt they considered exactly how big those problems would be.