Another massive IT problem in the UK

mikeybrennan

http://www.bbc.com/news/uk-40069865
hot on the heels of the NHS crash ,all BA flights around the world are cancelled

Can we expect more of this?

What's surprising is how when the lot goes down and there's no way of getting these systems back running immediately

Raging_Ninja

mikeybrennan wrote: »

http://www.bbc.com/news/uk-40069865
hot on the heels of the NHS crash ,all BA flights around the world are cancelled

Can we expect more of this?

What's surprising is how when the lot goes down and there's no way of getting these systems back running immediately

More than likely the systems go down all the time and do come back up again immediately. They say this isn't an attack, so the NHS incident is not really a good comparison. Could be more like the RBS incident from a few years ago though.

mikeybrennan

mikeybrennan wrote: »

http://www.bbc.com/news/uk-40069865
hot on the heels of the NHS crash ,all BA flights around the world are cancelled

Can we expect more of this?

What's surprising is how when the lot goes down and there's no way of getting these systems back running immediately

They outsourced their IT last year..
http://www.computerweekly.com/news/4500270832/GMB-takes-concerns-over-British-Airways-IT-outsourcing-to-MPs

mikeybrennan

Raging_Ninja wrote: »

More than likely the systems go down all the time and do come back up again immediately. They say this isn't an attack, so the NHS incident is not really a good comparison. Could be more like the RBS incident from a few years ago though.

It's not an attack, but broadly speaking they're all outages of one sort or another.

Will we ever get to a stage when systems can be restored immediately regardless of cause.Surely that has to be a goal to avoid this mayhem

keith_sixteen

NoDrama wrote: »

They outsourced their IT last year..
http://www.computerweekly.com/news/4500270832/GMB-takes-concerns-over-British-Airways-IT-outsourcing-to-MPs

Delighted for them. Any cost savings are now irrelevant.

irishgeo

keith_sixteen wrote: »

Delighted for them. Any cost savings are now irrelevant.

Penalty clauses.

Raging_Ninja

NoDrama wrote: »

They outsourced their IT last year..
http://www.computerweekly.com/news/4500270832/GMB-takes-concerns-over-British-Airways-IT-outsourcing-to-MPs

Ah, so it's probably exactly like the RBS incident - pay peanuts, get monkeys. The amount of horror stories of IT support outsourced to India is unreal, it's a wonder anyone still does it.

Raging_Ninja

mikeybrennan wrote: »

It's not an attack, but broadly speaking they're all outages of one sort or another.

Will we ever get to a stage when systems can be restored immediately regardless of cause.Surely that has to be a goal to avoid this mayhem

The way to avoid these incidents is to have competent people manage your IT systems. They had a working IT systems before outsourcing it to India, didn't they?

You can have one of the most foolproof systems imaginable, but if you get rid of your experienced staff and send it to the lowest common denominator, they're going to eventually set it on fire.

mikeybrennan

Raging_Ninja wrote: »

Ah, so it's probably exactly like the RBS incident - pay peanuts, get monkeys. The amount of horror stories of IT support outsourced to India is unreal, it's a wonder anyone still does it.

What are the horror stories?

Can someone clarify the exact meaning of outsourcing IT for the less technical among us?

NIMAN

Basically stop doing it using your own staff, and subcontracting it out.

Means you lose quality control, have to rely on the 3rd party to do it properly.

Raging_Ninja

mikeybrennan wrote: »

What are the horror stories?

Can someone clarify the exact meaning of outsourcing IT for the less technical among us?

Outsourcing IT is where you fire your IT staff and hire a company in India to manage your IT systems and needs.

Horror stories would be the RBS (Ulster Bank here) outage which meant hundreds of thousands could not access their bank accounts to get cash for weeks after a contractor in India somehow managed to delete a shedload of data and no backups were kept. I suppose this is going to be another horror story. Can give more lists but would need sit down at my PC in order to do it comfortably.

Senna

Raging_Ninja wrote: »

Outsourcing IT is where you fire your IT staff and hire a company in India to manage your IT systems and needs.

Horror stories would be the RBS (Ulster Bank here) outage which meant hundreds of thousands could not access their bank accounts to get cash after a contractor in India somehow managed to delete a shedload of data and no backups were kept. I suppose this is going to be another horror story. Can give more lists but would need sit down at my PC in order to do it comfortably.

Yeah an the idiot who caused the RBS outage was on a Internet forum 30 minutes later asking how can he fix it.

Stheno

Raging_Ninja wrote: »

Outsourcing IT is where you fire your IT staff and hire a company in India to manage your IT systems and needs.

Horror stories would be the RBS (Ulster Bank here) outage which meant hundreds of thousands could not access their bank accounts to get cash after a contractor in India somehow managed to delete a shedload of data and no backups were kept. I suppose this is going to be another horror story. Can give more lists but would need sit down at my PC in order to do it comfortably.

Rbs didn't outsource to another company they transferred it operations from their English office to their Indian one

Raging_Ninja

Stheno wrote: »

Rbs didn't outsource to another company they transferred it operations from their English office to their Indian one

Same thing, they fired their competent experienced staff in the UK and hired cheap Indian staff in order to replace them.

Atlantic Dawn

Systems are only as good as the people looking after them. I recon with compensation and rescheduling of flights your looking at near £10 million worth of a mess.

mikeybrennan

Atlantic Dawn wrote: »

Systems are only as good as the people looking after them. I recon with compensation and rescheduling of flights your looking at near £10 million worth of a mess.

Ba flights worldwide disrupted

I'm guessing its a multiple of that figure

keith_sixteen

irishgeo wrote: »

Penalty clauses.

I can't imagine there is one for grounding their entire fleet....there will be eyewatering amounts of money lost here.

wes

mikeybrennan wrote: »

What's surprising is how when the lot goes down and there's no way of getting these systems back running immediately

There is a way, BA and other companies just aren't willing to pay for it. A lot of companies need to realize that IT is there company, and when it goes down they can't do business, and stop trying to cheap out on things.

Capt'n Midnight

irishgeo wrote: »

Penalty clauses.

Only if properly setup, and in most cases they aren't, because cheap.

Even then its not going to pay for the reputation damage. Remember the previous baggage fiasco ? it took 3% off the share price

And a lot of this rings true
https://hbr.org/2008/04/how-can-british-airways-recove

Planned for 20 years, at a cost of £4.3bn, the opening of BA’s new home in T5 was an astonishing catalogue of organisational blunders.

Where to start? On the operational side, there were technical errors, mechanical failures, and little system testing. On the management side, there was arrogance, complacency, poor communication, and a refusal to listen to staff and technical experts. Staff were poorly trained, morale was low, and goodwill had long evaporated.
...

BA’s shares fell 3% on T5 opening day, wiping £90m off BA’s value.

Also
https://www.theregister.co.uk/2017/05/26/major_incident_at_capita_data_centre/

A major outage at a Capita data centre has knocked out multiple services for customers – including a number of councils' online services – for the last 36 hours.

...
Capita remains the biggest provider of tech services and software to the British public sector: revenues swelled to £1.9bn in 2016, equating to a 50 per cent market share.

marno21

It's been very quiet in the skies over southern Ireland without all the British Airways flights going west from LHR.

The savings from moving the IT will be wiped out thousands of times. They are lucky it's happened at the weekend and the stock exchange isn't open but I expect a hit on Monday morning, especially if this goes on into tomorrow.

degsie

Did they try to turn it off and on again?

theteal

Having first hand experience of the BA's IT infrastructure, this is not the least bit surprising. (I was actually out in Heathrow last week - it wasn't me, I swear!)

Yes, all their IT is outsourced and the handful of IT personnel still based in Heathrow are contractors. Staff with IT issues at site need to call an Indian call centre to hopefully get a ticket logged to get anything done. It's madness. I knew companies don't want to pay for IT unless they have to but some of the penny pinching I've seen is quite shocking.....maybe things will change now....

Stheno

BBC reporting it's a power supply issue

degsie

Stheno wrote: »

BBC reporting it's a power supply issue

Have they never heard of UPSs

Damien360

Raging_Ninja wrote: »

Ah, so it's probably exactly like the RBS incident - pay peanuts, get monkeys. The amount of horror stories of IT support outsourced to India is unreal, it's a wonder anyone still does it.

The company I work for has outsourced all its global IT to India. Genuine disaster. At my level, the IT contact is low level. Everyone dreads ringing them, spelling your name a million times, they log on and reset your settings, that doesn't work and they announce it has to be escalated. That takes forever and you are stuck waiting. Someone messages you at an ungodly hour looking for you to log on. Pain in the ass.

Stheno

degsie wrote: »

Have they never heard of UPSs

Exactly

Stheno

Stheno wrote: »

BBC reporting it's a power supply issue

I heard that earlier..

i believe only yammer and email came back after the outage..

Resilience, didn't work here..

Heads will roll.

Deub

Some companies look too much at numbers. Yes, IT department, customer service, etc are departments that cost money and on papers don't seem to bring any. However if you step back you see that they can avoid you big losses and even bring you better reputation which results in more customers and therefore more money.

mikeybrennan

theteal wrote: »

Having first hand experience of the BA's IT infrastructure, this is not the least bit surprising. (I was actually out in Heathrow last week - it wasn't me, I swear!)

Yes, all their IT is outsourced and the handful of IT personnel still based in Heathrow are contractors. Staff with IT issues at site need to call an Indian call centre to hopefully get a ticket logged to get anything done. It's madness. I knew companies don't want to pay for IT unless they have to but some of the penny pinching I've seen is quite shocking.....maybe things will change now....

I'm interested but confused about this outsourcing

Who's in charge of IT once the major outage hits? Do UK based contractors take over?

Surely they do?


And what is the problem with the outsourced IT? Did they actually create the issue or cant solve it quick enough?

degsie

degsie wrote: »

Have they never heard of UPSs

They obviously dont have a proper Disaster Recovery plan either. A complete power outage in a datacentre should mean switching Operations to the backup or DR site and being operational in a matter of hours.

Outsourcing IT, even to countries like India is fine, the real issue is the level and type of service which a company is willing to pay for.

mikeybrennan

Damien360 wrote: »

The company I work for has outsourced all its global IT to India. Genuine disaster. At my level, the IT contact is low level. Everyone dreads ringing them, spelling your name a million times, they log on and reset your settings, that doesn't work and they announce it has to be escalated. That takes forever and you are stuck waiting. Someone messages you at an ungodly hour looking for you to log on. Pain in the ass.

Reminds me ,when i worked in the US i had to speak to a lot of chinese in california for tech support.

It was a great source of amusement for all the americans in the office

Capt'n Midnight

To save some of their bandwidth here is the message on their site.
the foreign language sites also have this message - in English :rolleyes:

http://www.britishairways.com/

Following the major IT system failure experienced earlier today, with regret we have had to cancel all flights leaving from Heathrow and Gatwick for the rest of today, Saturday, May 27.

We are working hard to get our customers who were due to fly today onto the next available flights over the course of the rest of the weekend. Those unable to fly will be offered a full refund.

The system outage has also affected our call centres and our website but we will update customers as soon as we are able to.

Most long-haul flights due to land in London tomorrow (Sunday, May 28) are expected to arrive as normal, and we are working to restore our services from tomorrow, although some delays and disruption may continue into Sunday.

Due to the lengths of the delays lots of people will be entitled to several hundred euro.

theteal

mikeybrennan wrote: »

I'm interested but confused about this outsourcing

Who's in charge of IT once the major outage hits? Do UK based contractors take over?

Surely they do?


And what is the problem with the outsourced IT? Did they actually create the issue or cant solve it quick enough?

I genuinely don't know who's in charge of such issues. I don't work for them. I would assume so but I'm unaware of their ability and skill level. Where there, once upon a time, would have been a team of IT staff of different skills and levels on hand at site on different shift patterns, now there might be one guy there on a Saturday, maybe another on call (not precise numbers, just a hopefully extreme example) - I hope they know what they're doing and work well under pressure!

Outsourced call centres for crap like account lock-outs (I was going to give a few examples but I'm struggling to think of anything else I'd trust them with tbh) is fine but beyond that is just frustrating for users and just adds obstacles to quick resolution of issues.

markpb

degsie wrote: »

Have they never heard of UPSs

I love these kind of glib responses on boards. Do you think it's likely that they don't have UPS on their servers? Or do you think it's more likely to be a much more nuanced and complicated problem. AWS suffered a massive outage years ago despite the fact that they had plenty of battery backup, instead of power circuit on the inside of the UPS failed.

Deleted User wrote: »

They obviously dont have a proper Disaster Recovery plan either. A complete power outage in a datacentre should mean switching Operations to the backup or DR site and being operational in a matter of hours.

DR tests are like restoring your backup tapes - it's good practice to do (like eating your greens) but it's rarely a real life solution to a problem. DR tests are normally very simple affairs like throwing the switch on the power supply or internet connection but real life problems don't happen like that. Internet connections flap or selectively lose traffic. Power supplies cut out but the fluctuating that happens at the same time fries the power circuits.

IT is complex, especially at the scale that BA operate. Glib statements from the masses ignore that complexity and ignore the good work done by the teams managing it. Pretending it's a simple mistake that the experts in boards.ie would never have made is childish.

degsie

markpb wrote: »

I love these kind of glib responses on boards. Do you think it's likely that they don't have UPS on their servers? Or do you think it's more likely to be a much more nuanced and complicated problem. AWS suffered a massive outage years ago despite the fact that they had plenty of battery backup, instead of power circuit on the inside of the UPS failed.



DR tests are like restoring your backup tapes - it's good practice to do (like eating your greens) but it's rarely a real life solution to a problem. DR tests are normally very simple affairs like throwing the switch on the power supply or internet connection but real life problems don't happen like that. Internet connections flap or selectively lose traffic. Power supplies cut out but the fluctuating that happens at the same time fries the power circuits.

IT is complex, especially at the scale that BA operate. Glib statements from the masses ignore that complexity and ignore the good work done by the teams managing it. Pretending it's a simple mistake that the experts in boards.ie would never have made is childish.

What would your proposed solution entail?

degsie

degsie wrote: »

Have they never heard of UPSs

Someone probably decided that it was unnecessary as there hadn't been any power cuts recently!

I remember working at a site in London where the IT manager told a story that back in the 1990's there was a phase of "downsizing" everything and auditors were brought in to determine what was "not required" for the core business.

All was fine until they had a "JCB induced power interruption" the UPS was rapidly draining ant the decision was made to fire up the standby generators, they failed to start. When the IT manager went to investigate, he discovered an empty space where it should have been, as part of the audit, it was deemed unnecessary and was disposed of.

Plan "B" was simply to gracefully shut everything down until power was restored, business managers went ape!

degsie

degsie wrote: »

What would your proposed solution entail?

These days, they really should look to clustering servers over the WAN in such a way that if an entire building was to go bang, the system continues to operate with minimal loss of performance. Data being constantly sync'd between the separate physical sites on a virtual RAID array.

cml387

You misunderstand the purpose of a UPS. It's not intended to keep the system going, it's only to give time to allow the system to shut down gracefully.

degsie

Deleted User wrote: »

These days, they really should look to clustering servers over the WAN in such a way that if an entire building was to go bang, the system continues to operate with minimal loss of performance. Data being constantly sync'd between the separate physical sites on a virtual RAID array.

And what about the cost for such infrastructure? How would this impact the bottom line for shareholders?

degsie

degsie wrote: »

And what about the cost for such infrastructure? How would this impact the bottom line for shareholders?

Do they gamble on the cheap solutions in current use not devaluing their shares, yes they do.

So I don't expect any decent technical solutions to get past the boardroom!

cml387

cml387 wrote: »

You misunderstand the purpose of a UPS. It's not intended to keep the system going, it's only to give time to allow the system to shut down gracefully.

Or to get an alternative supply up and running, crank up the Lister!

listermint

[Deleted User] wrote: »

Or to get an alternative supply up and running, crank up the Lister!

You called

cml387

Deleted User wrote: »

Or to get an alternative supply up and running, crank up the Lister!

An oldie from my time in DEC.

TallGlass

Raging_Ninja wrote: »

Ah, so it's probably exactly like the RBS incident - pay peanuts, get monkeys. The amount of horror stories of IT support outsourced to India is unreal, it's a wonder anyone still does it.

To be honest, the problem isn't the guys in India, I work with guys from India and they are some of the brightest workers I have come across.

The problem from speaking to one of them is there working laws, which are completely different, in fact worlds apart from ours and the UK laws.

As an example, a guy recently told me in India, he came into work thinking it was going to be a normal shift ended up working 3 days solid day and night, sleeping barely at his desk. If he tried to say no, that was him done and the company would replace him.

That is just asking for a disaster. Over worked staff make mistakes and cause massive implications for the system. Computers don't just decide to make changes, some has to tell it what to do. If that person is 3 days into a shift, something is going to go tits up, sooner rather than later.

IT doesn't change in Ireland, the UK or India, the computer follows the same protocols as it would in another part of the world. The problem is working conditions.

IT outsourcing works, it works well. Looking for the cheapest person to do it, with a cheap workforce and non existent working laws is not.

riffmongous

VinLieger

degsie wrote: »

And what about the cost for such infrastructure? How would this impact the bottom line for shareholders?

Very likely less than reschesuling and paying for all the knock on costs of a single day of all flights being canceled.

However similar to with the NHS updates the costs are only looked on as being worthwhile after the fact when the resulting problems of not spending money as a precautipn cost more than preventing the problems

TallGlass

TallGlass wrote: »

To be honest, the problem isn't the guys in India, I work with guys from India and they are some of the brightest workers I have come across.

The problem from speaking to one of them is there working laws, which are completely different, in fact worlds apart from ours and the UK laws.

As an example, a guy recently told me in India, he came into work thinking it was going to be a normal shift ended up working 3 days solid day and night, sleeping barely at his desk. If he tried to say no, that was him done and the company would replace him.

That is just asking for a disaster. Over worked staff make mistakes and cause massive implications for the system. Computers don't just decide to make changes, some has to tell it what to do. If that person is 3 days into a shift, something is going to go tits up, sooner rather than later.

IT doesn't change in Ireland, the UK or India, the computer follows the same protocols as it would in another part of the world. The problem is working conditions.

IT outsourcing works, it works well. Looking for the cheapest person to do it, with a cheap workforce and non existent working laws is not.

True, many of the Indian IT staff are the cream of the crop and in my experience they often end up as specialist in one area and know sweet FA about the other elements of the system. It's not simply a case of thinking that the other parts of the system as "black boxes" but not even knowing they exist, as in the data gets from the hard disk of one server to another "by magic", they can spend a lot if time looking at the database structure while completely ignoring the fact that source data is missing and not checking with someone else who deals with that server.

As for the silly hours, I have heard snoring on a conference call before....

Capt'n Midnight

Deleted User wrote: »

These days, they really should look to clustering servers over the WAN in such a way that if an entire building was to go bang, the system continues to operate with minimal loss of performance. Data being constantly sync'd between the separate physical sites on a virtual RAID array.

You can even do it a drive level. Fibre optics means mirrored drives can be on a different site in a different town.

cml387 wrote: »

You misunderstand the purpose of a UPS. It's not intended to keep the system going, it's only to give time to allow the system to shut down gracefully.

Only for organisations that don't need 24/7. For them UPS's only need to keep the servers up long enough for the backup generators to kick in.

markpb wrote: »

I love these kind of glib responses on boards. Do you think it's likely that they don't have UPS on their servers? Or do you think it's more likely to be a much more nuanced and complicated problem.

The simple answer is that they weren't properly prepared.
BA have a turnover of £11Bn a year, so aren't a fly by night operation.

They've had plenty of time to sort out legacy systems and there is no way in hell they should have outsourced mission critical stuff until they had resilient systems in place.

Capt'n Midnight

cml387 wrote: »

An oldie from my time in DEC.

LOL

different story, servers went down but they stayed up
http://i.imgur.com/Sb2M5Qo.jpg

Yourself isit

markpb wrote: »

I love these kind of glib responses on boards. Do you think it's likely that they don't have UPS on their servers? Or do you think it's more likely to be a much more nuanced and complicated problem. AWS suffered a massive outage years ago despite the fact that they had plenty of battery backup, instead of power circuit on the inside of the UPS failed.



DR tests are like restoring your backup tapes - it's good practice to do (like eating your greens) but it's rarely a real life solution to a problem. DR tests are normally very simple affairs like throwing the switch on the power supply or internet connection but real life problems don't happen like that. Internet connections flap or selectively lose traffic. Power supplies cut out but the fluctuating that happens at the same time fries the power circuits.

IT is complex, especially at the scale that BA operate. Glib statements from the masses ignore that complexity and ignore the good work done by the teams managing it. Pretending it's a simple mistake that the experts in boards.ie would never have made is childish.

At the scale that BA operate their experts should keep downtime to 1-2 hours a year, and there should be no data loss.