Many BA Flights cancelled after IT systems failure

la ultima guagua

Somebody tell me it isnt true :eek:

robyntmorton

Something has gone wrong with their global IT systems. What exactly, we don't know, but it will screw them up for several days, no matter how quickly they get things going again.

Irish Steve

All flights from Heathrow and Gatwick for the rest of today have been cancelled.

The problems are being blamed on " A major IT systems failure causing severe disruptions to flight operations worldwide".

Locker10a

A total nightmare for the poor passengers and ground staff trying to manage this.
I was very nearly booked on a BA flight this evening, glad I'm not now!

Foggy43

This has been going on since early this morning I'm afraid. Again it will be shorthaul flights canceled. Still trying to get 'some' longhaul flights out.

C.O.Y.B.I.B

Cyber attack, Malware??

Stheno

C.O.Y.B.I.B wrote: »

Cyber attack, Malware??

More likely a planned change which went wrong and they were unable to get back to normal

Creamy Goodness

C.O.Y.B.I.B wrote: »

Cyber attack, Malware??

Heard on RTÉ news that they ruled that out.

I was meant to be in Heathrow today on my way back from São Paulo. So glad I wasn't.

markpb

They outsourced to Tata in India last year. I wonder if any Irish airline planning to do the same thing are reevaluating their plans today.

Foggy43

I don't think it has been so quite in the skies over London since 9/11.

mickdw

Is this even possible given today's IT capability that a monster business could be caught out like that. I certainly wouldn't have thought so. First thing that entered my head when I heard that everything BA was grounded was a possible bomb plot/terror event re aircraft uncovered.

Stheno

mickdw wrote: »

Is this even possible given today's IT capability that a monster business could be caught out like that. I certainly wouldn't have thought so. First thing that entered my head when I heard that everything BA was grounded was a possible bomb plot/terror event re aircraft uncovered.

Yes it is. The BBC are now reporting that BA have said its a power supply issue that may not be fixed for 3-4 days

BA may have chosen not to pay Tata for built on fail over in their systems, or it could just be a catastrophic outage

Irish Steve

A power supply issue of that magnitude and with that length of down time makes me wonder what other corners were cut to put the contract in place, a system that has to be there 24/7/365 has to have resilience in ways that are way more complex than just having a larger fuse than the absolute minimum, Specialised power supplies, with dual modes, and battery backed systems with multiple standby generators are only the starting point, and it would seem that they have not put that level of redundancy in place, and are now going to be paying the very high price for not doing so.

The potential costs of this outage will be massive.

L1011

I work somewhere where, with the best intent in the world for power supply redundancy, we had a UPS go on fire and take an entire DC out of action.

However we were able to bring up critical services on the *second data centre* within 20 minutes. Some non critical stuff wasn't set up redundant but there was no customer impact and it allowed the flaming DC to be halon'ed and repaired offline.

markpb

mickdw wrote: »

Is this even possible given today's IT capability that a monster business could be caught out like that. I certainly wouldn't have thought so.

The opposite is true. Businesses now depend more on IT than ever before so an outage can have crippling effects. Also, as more business processes are automated, the complexity increases which makes them more susceptible to major problems that are hard to recover from.

The more IT is involved in the system, people see more potential and call for it to be more involved. Years ago, the only systems with access to an airline system were the check in desks and some travel agents. Now every mobile phone in the world can be used to book flights, amend bookings and produce boarding passes. Aviation security, shared loyalty schemes and business analytics means that number of places that data has to go to has also increased.

When you see all that happening, every change is a potential grenade and the ability to recover from even small problems can dwarf the people tasked with running it.

And then someone decides it's a good idea to get rid of those people and give it to new, cheaper people. The fact that those people are Indian and cheaper is not the idea, just that they're new.

PaulKK

mickdw wrote: »

Is this even possible given today's IT capability that a monster business could be caught out like that. I certainly wouldn't have thought so. First thing that entered my head when I heard that everything BA was grounded was a possible bomb plot/terror event re aircraft uncovered.

Today's IT capability and what these legacy companies are actually using would surprise you. Ever wonder why payments to/from banks other than yours take a day or two to show up for example? They are using systems that are up to 30 years old. I work in IT and have heard stories from people who previously worked in financial institutions that would shock you.

Even apparently small changes like updating antivirus versions etc can bring down systems like these. Hard to believe but true.

L1011

PaulKK wrote: »

Today's IT capability and what these legacy companies are actually using would surprise you. Ever wonder why payments to/from banks other than yours take a day or two to show up for example? They are using systems that are up to 30 years old. I work in IT and have heard stories from people who previously worked in financial institutions that would shock you.

Even apparently small changes like updating antivirus versions etc can bring down systems like these. Hard to believe but true.

There's a level of incompetence and refusal to do anything to alleviate problems required to leave legacy systems that bad

I have OpenVMS gear to deal with - as legacy as you can get, OpenVMS is 40 years old and some of the apps we run were ported from DEC PDP. Still use cluster disks on redundant SAN and could lose an entire site without really noticing.

I had to fix an issue this week on an app that is older than I am!

Oscar Bravo

Some flights now getting airborne, BAW21G LHR-BOS about to pass over Ireland.

PaulKK

L1011 wrote: »

There's a level of incompetence and refusal to do anything to alleviate problems required to leave legacy systems that bad

I have OpenVMS gear to deal with - as legacy as you can get, OpenVMS is 40 years old and some of the apps we run were ported from DEC PDP. Still use cluster disks on redundant SAN and could lose an entire site without really noticing.

I had to fix an issue this week on an app that is older than I am!

Management don't want to pay for something they can't see/understand, usually that simple really.

When IT come looking for money to implement the latest engineering best practices, its a hard sell to simply say reliability or security for example would be much improved.

IT is often just seen as a cost. Hence the outsourcing reference made above.

trellheim

looking at fr14 BAW airline page 318 aircraft, 30 active, whoa that is some amount of chaos for passengers and cash lost .

the mainframe must have had a serious psu problem ...!!!

Irish Steve

legacy computer issues............. A while back, I reported to one of the major banks in this country that they were using an out of date (expired) security certificate that was causing an exception warning in the latest version of Internet Explorer (Yeah I know, but at the time this bank's business package would only work on a windows based PC using Internet explorer, no other OS or browser was supported).

Their response initially was "ignore the security warning message, or change IE to turn off the warning system", which was not exactly an appropriate response, and when I pressed the matter with higher management, the "official" response was "we have it in the update cycle, but due to the time it takes to install this via dial up lines, we won't be updating it for at least 4 months". The unfortunate reality is that there are a lot of senior managers who don't have a clue about the specifics of IT, and as a result, all manner of unthinkable nasties happen with monotonous regularity, and will continue to do so until a new breed of more aware managers are in place, which will take time

I suspect that today's BA failure won't have been a mainframe (or more likely a server), as they are usually reasonably well supported, it will come down to a data switching centre somewhere that routes traffic from a major hub to the computer centre, and something like a fibre network switching centre or similar lost it's power, and there was no resilient fall back through another carrier or cable provider, which resulted in a comms link failure, rather than a computer failure as such.

Either way, you may be very sure that there are some very high level phone conversations happening every few minutes, and they will continue until this mess has not only been resolved, they will be happening until a complete backup strategy to prevent a recurrent is in place and has been proved to be resilient.

ThisRegard

PaulKK wrote: »

TThey are using systems that are up to 30 years old. I work in IT and have heard stories from people who previously worked in financial institutions that would shock you.

Even apparently small changes like updating antivirus versions etc can bring down systems like these. Hard to believe but true.

Hardware excluding, 30 year old systems are extremely reliable now, pretty much no bugs at this stage. They also tend not to run on platforms prone to too many viruses such as AS400s etc.

Senna

If you want to read about how a company can lose $500million in 45 minutes due to an IT error, google KNIGHT Capital IT glitch

Gmaximum

It might fix the aer club debacle when they restart the computer

Gamebred

PaulKK wrote: »

Ever wonder why payments to/from banks other than yours take a day or two to show up for example? They are using systems that are up to 30 years old

Sorry to go slightly off topic but I was led to believe the reason for delay in transferring of funds is to stop money laundering and fraud?

faoiarvok

Gamebred wrote: »

Sorry to go slightly off topic but I was led to believe the reason for delay in transferring of funds is to stop money laundering and fraud?

Very interesting [American] podcast episode about legacy systems affecting inter-bank transfer times:
http://www.npr.org/sections/money/2013/10/04/229224964/episode-489-the-invisible-plumbing-of-our-economy

PaulKK

Gamebred wrote: »

Sorry to go slightly off topic but I was led to believe the reason for delay in transferring of funds is to stop money laundering and fraud?

There is generally a nightly batch cycle which processes all the transactions together. That's why some banks have daily cut off times etc, you won't make that nights batch after 3pm for example.

smurfjed

[font=Arial, Helvetica, sans-serif]Chief executive Alex Cruz said the crash affected all check-in and operational systems, adding: 'We believe the root cause was a power-supply issue and we have no evidence of any cyber attack.[/font]

Wow, who designed their disaster recovery plan, or was there one?

marno21

Are BA routing many left behind passengers via EI flights via DUB or SNN?

Irish Steve

marno21 wrote: »

Are BA routing many left behind passengers via EI flights via DUB or SNN?

Their ability to do something like this will be based on what access they have to their computers at this stage, if they can see who is booked to travel, and where, they may be able to move people to alternative carriers, but if their systems are still compromised, they may not have access to the information to be able to make that decision.

duskyjoe

marno21 wrote: »

Are BA routing many left behind passengers via EI flights via DUB or SNN?

I'm sure the odd few but EIN themselves are packed to the rafters long haul these days