Switch turned off left 75,000 Passengers Stranded

368100

https://www.rte.ie/news/2017/0602/879764-british-airways/


Ha ha, you couldn't make it up!:D

Atlantic Dawn

Your Face

Ok

seamus

Yeah, happened in a national retailer I worked at. Not quite as big and the "data centre" was in the same building as us. But a guy came in to work on a faulty air conditioning unit in the server room.

One the techs (who was supposed to be senior) let the guy into the room and then brought him to what the tech thought was the faulty air con unit and pushed a big red "stop" button on it to turn it off.

He had actually pressed the kill switch on the UPS (uninterruptible power supply) and brought everything down. Took about half an hour to just get the power supply back up after hitting that button, then a few hours it took the servers to stop freaking out about being switched off.

Lots of squeaky arses in BA at the moment I'd say trying to explain why they didn't have multi-location redundancy for their systems.

ohnonotgmail

Doesnt matter if that is what happened. It is still BA's fault for having a single point of failure.

Ubbquittious

I used to know a lad who didn't know how to clear the history of his browser.

Each time his ma tried to go onto ba.com for booking a flight the autocomplete would show up 'baldpussy.com'

greencap

Atlantic Dawn wrote: »

ChikiChiki

ohnonotgmail wrote: »

Doesnt matter if that is what happened. It is still BA's fault for having a single point of failure.

Exactly, unbelievable stuff from a major airline. No business continuity plan in place it seems. Heads should roll for sure.

gramar

seamus wrote: »

He had actually pressed the kill switch on the UPS (uninterruptible power supply) and brought everything down. Took about half an hour to just get the power supply back up after hitting that button, then a few hours it took the servers to stop freaking out about being switched off.

If the power supply is uninterruptible then why have a kill switch?

seamus

gramar wrote: »

If the power supply is uninterruptible then why have a kill switch?

You've gotta be able to switch things off. It's really there in case the UPS burst into flames or something.

All critical kit should be fed by two UPSes anyway, so it wouldn't matter if one of them was turned off.

arctictree

My local pub has a UPS connected to the till in case of a power cut. Need to get your priorities right!

Ubbquittious

arctictree wrote: »

My local pub has a UPS connected to the till in case of a power cut. Need to get your priorities right!

What about cooling the beer? Do they have a jenny for that or is the UPS sufficiently heavy?

If they start serving warm pints they won't have much use for the till either

Wossack

boardroom transcript

a: those IT monkeys are making an awful racket about only having a single datacentre and squawking about resiliency and some such
b: and how much is that gonna cost?
c: something with that many big words aint gonna be cheap
a: yea, its gonna be heaps
c: hmm, could we afford it if we fire them all, and outsource to india?
a: eh, probably

*fires them all, outsources to india*

bc: wow, we saved so much money this year, bonuses all round!
a: but weren't we gonna spend that money on resiliency!?
**inaudible champagne corks etc**

ohnonotgmail

reading a bit on this elsewhere and the problem wasnt turning it off, the problem was turning it back on in an uncontrolled manner. seemingly the racks ended up getting 480V instead of 240V.

Ciaran_B

368100 wrote: »

https://www.rte.ie/news/2017/0602/879764-british-airways/


Ha ha, you couldn't make it up!:D

A cynic might suggest that's exactly what BA just did to try to weasel out of their EC261 obligations.

Beta Ray Bill

Wossack wrote: »

boardroom transcript

a: those IT monkeys are making an awful racket about only having a single datacentre and squawking about resiliency and some such
b: and how much is that gonna cost?
c: something with that many big words aint gonna be cheap
a: yea, its gonna be heaps
c: hmm, could we afford it if we fire them all, and outsource to india?
a: eh, probably

*fires them all, outsources to india*

bc: wow, we saved so much money this year, bonuses all round!
a: but weren't we gonna spend that money on resiliency!?
**inaudible champagne corks etc**

This Multiplied by 1 Billion

There's CEO's / Senior Management people don't realise that IT is the Heart of any big business.
It has to be looked after.

Some IT day rate contractor knocking off the power and no fail over in place is outrageous!

ohnonotgmail

grahambo wrote: »

This Multiplied by 1 Billion

There's CEO's / Senior Management people don't realise that IT is the Heart of any big business.
It has to be looked after.

Some IT day rate contractor knocking off the power is outrageous!

the problems actually happened in their two data centers near heathrow.

Yamanoto

Someone at EI's AerClub could do with turning the system off & back on again.

Capt'n Midnight

gramar wrote: »

If the power supply is uninterruptible then why have a kill switch?

Hector Bellend

Ubbquittious wrote: »

I used to know a lad who didn't know how to clear the history of his browser.
ore.
Each time his ma tried to go onto ba.com for booking a flight the autocomplete would show up 'baldpussy.com'

Its been moved to xvideos by now.

He shouldn't have that problem any more