Another massive IT problem in the UK

mikeybrennan · 27-05-2017 04:32PM #1

http://www.bbc.com/news/uk-40069865
hot on the heels of the NHS crash ,all BA flights around the world are cancelled

Can we expect more of this?

What's surprising is how when the lot goes down and there's no way of getting these systems back running immediately

Raging_Ninja · 27-05-2017 04:35PM

mikeybrennan wrote: »

http://www.bbc.com/news/uk-40069865
hot on the heels of the NHS crash ,all BA flights around the world are cancelled

Can we expect more of this?

What's surprising is how when the lot goes down and there's no way of getting these systems back running immediately

More than likely the systems go down all the time and do come back up again immediately. They say this isn't an attack, so the NHS incident is not really a good comparison. Could be more like the RBS incident from a few years ago though.

27-05-2017 04:36PM

mikeybrennan wrote: »

http://www.bbc.com/news/uk-40069865
hot on the heels of the NHS crash ,all BA flights around the world are cancelled

Can we expect more of this?

What's surprising is how when the lot goes down and there's no way of getting these systems back running immediately

They outsourced their IT last year..
http://www.computerweekly.com/news/4500270832/GMB-takes-concerns-over-British-Airways-IT-outsourcing-to-MPs

mikeybrennan · 27-05-2017 04:41PM

Raging_Ninja wrote: »

More than likely the systems go down all the time and do come back up again immediately. They say this isn't an attack, so the NHS incident is not really a good comparison. Could be more like the RBS incident from a few years ago though.

It's not an attack, but broadly speaking they're all outages of one sort or another.

Will we ever get to a stage when systems can be restored immediately regardless of cause.Surely that has to be a goal to avoid this mayhem

keith_sixteen · 27-05-2017 04:45PM

NoDrama wrote: »

They outsourced their IT last year..
http://www.computerweekly.com/news/4500270832/GMB-takes-concerns-over-British-Airways-IT-outsourcing-to-MPs

Delighted for them. Any cost savings are now irrelevant.

irishgeo · 27-05-2017 04:47PM

keith_sixteen wrote: »

Delighted for them. Any cost savings are now irrelevant.

Penalty clauses.

Raging_Ninja · 27-05-2017 04:47PM

NoDrama wrote: »

They outsourced their IT last year..
http://www.computerweekly.com/news/4500270832/GMB-takes-concerns-over-British-Airways-IT-outsourcing-to-MPs

Ah, so it's probably exactly like the RBS incident - pay peanuts, get monkeys. The amount of horror stories of IT support outsourced to India is unreal, it's a wonder anyone still does it.

Raging_Ninja · 27-05-2017 04:51PM

mikeybrennan wrote: »

It's not an attack, but broadly speaking they're all outages of one sort or another.

Will we ever get to a stage when systems can be restored immediately regardless of cause.Surely that has to be a goal to avoid this mayhem

The way to avoid these incidents is to have competent people manage your IT systems. They had a working IT systems before outsourcing it to India, didn't they?

You can have one of the most foolproof systems imaginable, but if you get rid of your experienced staff and send it to the lowest common denominator, they're going to eventually set it on fire.

mikeybrennan · 27-05-2017 04:57PM

Raging_Ninja wrote: »

Ah, so it's probably exactly like the RBS incident - pay peanuts, get monkeys. The amount of horror stories of IT support outsourced to India is unreal, it's a wonder anyone still does it.

What are the horror stories?

Can someone clarify the exact meaning of outsourcing IT for the less technical among us?

NIMAN · 27-05-2017 04:59PM

Basically stop doing it using your own staff, and subcontracting it out.

Means you lose quality control, have to rely on the 3rd party to do it properly.

Raging_Ninja · 27-05-2017 05:02PM

mikeybrennan wrote: »

What are the horror stories?

Can someone clarify the exact meaning of outsourcing IT for the less technical among us?

Outsourcing IT is where you fire your IT staff and hire a company in India to manage your IT systems and needs.

Horror stories would be the RBS (Ulster Bank here) outage which meant hundreds of thousands could not access their bank accounts to get cash for weeks after a contractor in India somehow managed to delete a shedload of data and no backups were kept. I suppose this is going to be another horror story. Can give more lists but would need sit down at my PC in order to do it comfortably.

Senna · 27-05-2017 05:04PM

Raging_Ninja wrote: »

Outsourcing IT is where you fire your IT staff and hire a company in India to manage your IT systems and needs.

Horror stories would be the RBS (Ulster Bank here) outage which meant hundreds of thousands could not access their bank accounts to get cash after a contractor in India somehow managed to delete a shedload of data and no backups were kept. I suppose this is going to be another horror story. Can give more lists but would need sit down at my PC in order to do it comfortably.

Yeah an the idiot who caused the RBS outage was on a Internet forum 30 minutes later asking how can he fix it.

Stheno · 27-05-2017 05:06PM

Raging_Ninja wrote: »

Outsourcing IT is where you fire your IT staff and hire a company in India to manage your IT systems and needs.

Horror stories would be the RBS (Ulster Bank here) outage which meant hundreds of thousands could not access their bank accounts to get cash after a contractor in India somehow managed to delete a shedload of data and no backups were kept. I suppose this is going to be another horror story. Can give more lists but would need sit down at my PC in order to do it comfortably.

Rbs didn't outsource to another company they transferred it operations from their English office to their Indian one

Raging_Ninja · 27-05-2017 05:15PM

Stheno wrote: »

Rbs didn't outsource to another company they transferred it operations from their English office to their Indian one

Same thing, they fired their competent experienced staff in the UK and hired cheap Indian staff in order to replace them.

Atlantic Dawn · 27-05-2017 05:19PM

Systems are only as good as the people looking after them. I recon with compensation and rescheduling of flights your looking at near £10 million worth of a mess.

mikeybrennan · 27-05-2017 05:29PM

Atlantic Dawn wrote: »

Systems are only as good as the people looking after them. I recon with compensation and rescheduling of flights your looking at near £10 million worth of a mess.

Ba flights worldwide disrupted

I'm guessing its a multiple of that figure

keith_sixteen · 27-05-2017 07:00PM

irishgeo wrote: »

Penalty clauses.

I can't imagine there is one for grounding their entire fleet....there will be eyewatering amounts of money lost here.

wes · 27-05-2017 07:16PM

mikeybrennan wrote: »

What's surprising is how when the lot goes down and there's no way of getting these systems back running immediately

There is a way, BA and other companies just aren't willing to pay for it. A lot of companies need to realize that IT is there company, and when it goes down they can't do business, and stop trying to cheap out on things.

Capt'n Midnight · 27-05-2017 08:24PM

irishgeo wrote: »

Penalty clauses.

Only if properly setup, and in most cases they aren't, because cheap.

Even then its not going to pay for the reputation damage. Remember the previous baggage fiasco ? it took 3% off the share price

And a lot of this rings true
https://hbr.org/2008/04/how-can-british-airways-recove

Planned for 20 years, at a cost of £4.3bn, the opening of BA’s new home in T5 was an astonishing catalogue of organisational blunders.

Where to start? On the operational side, there were technical errors, mechanical failures, and little system testing. On the management side, there was arrogance, complacency, poor communication, and a refusal to listen to staff and technical experts. Staff were poorly trained, morale was low, and goodwill had long evaporated.
...

BA’s shares fell 3% on T5 opening day, wiping £90m off BA’s value.

Also
https://www.theregister.co.uk/2017/05/26/major_incident_at_capita_data_centre/

A major outage at a Capita data centre has knocked out multiple services for customers – including a number of councils' online services – for the last 36 hours.

...
Capita remains the biggest provider of tech services and software to the British public sector: revenues swelled to £1.9bn in 2016, equating to a 50 per cent market share.

marno21 · 27-05-2017 08:26PM

It's been very quiet in the skies over southern Ireland without all the British Airways flights going west from LHR.

The savings from moving the IT will be wiped out thousands of times. They are lucky it's happened at the weekend and the stock exchange isn't open but I expect a hit on Monday morning, especially if this goes on into tomorrow.

degsie · 27-05-2017 08:28PM

Did they try to turn it off and on again?

theteal · 27-05-2017 08:38PM

Having first hand experience of the BA's IT infrastructure, this is not the least bit surprising. (I was actually out in Heathrow last week - it wasn't me, I swear!)

Yes, all their IT is outsourced and the handful of IT personnel still based in Heathrow are contractors. Staff with IT issues at site need to call an Indian call centre to hopefully get a ticket logged to get anything done. It's madness. I knew companies don't want to pay for IT unless they have to but some of the penny pinching I've seen is quite shocking.....maybe things will change now....

Stheno · 27-05-2017 08:40PM

BBC reporting it's a power supply issue

degsie · 27-05-2017 08:44PM

Stheno wrote: »

BBC reporting it's a power supply issue

Have they never heard of UPSs

Damien360 · 27-05-2017 08:44PM

Raging_Ninja wrote: »

Ah, so it's probably exactly like the RBS incident - pay peanuts, get monkeys. The amount of horror stories of IT support outsourced to India is unreal, it's a wonder anyone still does it.

The company I work for has outsourced all its global IT to India. Genuine disaster. At my level, the IT contact is low level. Everyone dreads ringing them, spelling your name a million times, they log on and reset your settings, that doesn't work and they announce it has to be escalated. That takes forever and you are stuck waiting. Someone messages you at an ungodly hour looking for you to log on. Pain in the ass.

Stheno · 27-05-2017 08:45PM

degsie wrote: »

Have they never heard of UPSs

Exactly

27-05-2017 08:48PM

Stheno wrote: »

BBC reporting it's a power supply issue

I heard that earlier..

i believe only yammer and email came back after the outage..

Resilience, didn't work here..

Heads will roll.

Deub · 27-05-2017 08:48PM

Some companies look too much at numbers. Yes, IT department, customer service, etc are departments that cost money and on papers don't seem to bring any. However if you step back you see that they can avoid you big losses and even bring you better reputation which results in more customers and therefore more money.

mikeybrennan · 27-05-2017 08:50PM

theteal wrote: »

Having first hand experience of the BA's IT infrastructure, this is not the least bit surprising. (I was actually out in Heathrow last week - it wasn't me, I swear!)

Yes, all their IT is outsourced and the handful of IT personnel still based in Heathrow are contractors. Staff with IT issues at site need to call an Indian call centre to hopefully get a ticket logged to get anything done. It's madness. I knew companies don't want to pay for IT unless they have to but some of the penny pinching I've seen is quite shocking.....maybe things will change now....

I'm interested but confused about this outsourcing

Who's in charge of IT once the major outage hits? Do UK based contractors take over?

Surely they do?

And what is the problem with the outsourced IT? Did they actually create the issue or cant solve it quick enough?

27-05-2017 08:58PM

degsie wrote: »

Have they never heard of UPSs

They obviously dont have a proper Disaster Recovery plan either. A complete power outage in a datacentre should mean switching Operations to the backup or DR site and being operational in a matter of hours.

Outsourcing IT, even to countries like India is fine, the real issue is the level and type of service which a company is willing to pay for.

mikeybrennan · 27-05-2017 09:00PM

Damien360 wrote: »

The company I work for has outsourced all its global IT to India. Genuine disaster. At my level, the IT contact is low level. Everyone dreads ringing them, spelling your name a million times, they log on and reset your settings, that doesn't work and they announce it has to be escalated. That takes forever and you are stuck waiting. Someone messages you at an ungodly hour looking for you to log on. Pain in the ass.

Reminds me ,when i worked in the US i had to speak to a lot of chinese in california for tech support.

It was a great source of amusement for all the americans in the office

Another massive IT problem in the UK

Comments