Ransomware & HSE

tobefrank321

Despite many years of HSE screwups, its unlikely any heads will roll.

If someone from the HSE burnt down the new National Childrens Hospital, the worst that would happen is they get moved sideways or even promoted.

When there is no accountability, you will continue to get costly clusterf*cks.

Xander10

Isn't the first lesson for all office workers, don't open attachments from unknown sources?

tobefrank321

https://www.boards.ie/discussion/comment/118224257#Comment_118224257

This was coming for years, they had plenty of notice and plenty of time to prepare. The HSE and similar organisations which deal with highly sensitive and critical personal information are prime targets for ransom attacks unfortunately.

A large number of PCs in the health service are Windows 7 or older, with very little security updates support.

I'd bet a large number of people working in the HSE do not have cyber security training or do at least one course a year on it which is the bare minimum.

If the HSE was a very cyber secure organisation and still fell foul, fair enough. But it sounds like they had and have an extremely lax approach to cyber security and sooner or later it was going to bite them.

bokale

Just out if interest why do they release these reports? Is it not helpful to hackers wanting to attack again?

BluePlanet

"During the recovery process in the days following the ransomware attack it became apparent that disaster recovery (DR) arrangements for IT systems were ad hoc and inconsistent. With the Attacker able to corrupt some primary data stores for disaster recovery, there was a requirement to identify secondary stores and attempt to recover from them. A workstream was initiated to attempt to locate them and test the viability of recovery. Were systems to have been recovered using this method, they would have been recovered to different points in time that backups were available for, and there was no confidence in the completeness (or in some cases tested viability) of recovery solutions. As a result, when the decryption key became available from the Attacker, the decision was made to abandon work to recover from backups, and instead recover systems from their production environment, using the decryption capability provided by the Attacker. It cannot be confidently asserted that all health services would have been able to recover in a timely manner (or even at all) without the provision of the decryption key by the Attacker."

So basically the HSE didn't have a Disaster Recovery process in place, and were unable to recover from backups. Any IT Director should fall on their sword over that.

Hotblack Desiato

A large number of PCs in the health service are Windows 7 or older, with very little security updates support.

I keep on hearing this from people who really should know better.

Windows 7 is still fully supported. That support isn't free - but the HSE did pay for extended support. No point having patches available if you don't deploy them.

So basically the HSE didn't have a Disaster Recovery process in place, and were unable to recover from backups. Any IT Director should fall on their sword over that.

A major part of the problem here is that it's not really "the HSE" we're talking about as some sort of monolith. It's the HSE and literally dozens of organisations big and small in the health sector which operate with a greater or lesser degree of autonomy. This is all a legacy not only of the old health board system, but the provision of a huge proportion of health services by "voluntary" organisations which go back to the early days of this state or even long before then.

72sheep

This report was always going to be a joke: 

• Written by an accountancy firm.
• Long list of PWC names but no MD-level practice lead sign-off. See bottom page 5 for how many people it took PWC to conclude to the HSE: “You are idiots for running Win 7; this is not a surprise as you’re being run by the likes of Dublin City councillors."
• Exec Summary | Conclusions and by para 3 it’s already gibberish: "The relative disadvantage in this Incident for organisations who have greater dependency on technology services, illustrates the critical need for resiliency to be built into the IT architecture and systems, to foster the confidence required to enable future migration to more digital provision of health services.”
• Can’t blame PWC for the evasive language around the decryption key - “sourced”, “became available”, LOL - as they could not relate how our state had to run around desperately to make arrangements for the ransom to be paid to the attacker.

Hotblack Desiato

Many of the usual IT outsourcing / services companies would have already been involved with the HSE or one of the related organisations in some capacity, so could not be used.

There was a lot more to this than "you're an idiot for running Windows 7" (which isn't even true as Windows 7 is not out of support)

What is your evidence that a ransom was paid?

AndrewJRenko

https://www.boards.ie/discussion/comment/118301907#Comment_118301907

Why would you assume it came from an unknown source? It could well have come from (or appeared to have come from) a known contact.

https://www.boards.ie/discussion/comment/118302715#Comment_118302715

PWC's consulting business is very substantial, as it is for Deloitte and KPMG. They're really not just 'an accountancy firm'. Signing on behalf of the practice is standard practice for partnership businesses like this.

What role do Dublin City Councillors have in running the HSE?

If you're claiming that a ransom was paid, you should really produce some evidence of this. Who paid it and when and how?

salonfire

Ah good old Andrew. Old one-sided Andrew. Lept to the defence of the HSE and staff prior to the report, but not a whimper now the facts become clear.

Instead he goes off in a tangent about PWC, Dublin City Council, evidence about ransomware.

No comment to make why the staff of the HSE left the door open to attack?

BrianD3

Jesus fcuking wept. Looks like our health service is about as competent at securing its (our) data as it is at providing healthcare.

What will the consequences be for:

The donkey who clicked on the link that started the whole thing?

The idiots who didn't have effective disaster recovery systems in place in 2021?

The liars/incompetents who spun this as being a zero day exploit?

This debacle should be borne in mind next time the HSE is crying about resources or deflecting blame when they are criticised over waiting lists A&E chaos or the lives they have ruined (necessitating 8 figure court settlements in many cases) It also needs to be borne in mind anytime anything is said about Covid, the hospitals are under pressure because <insert scapegoat here> aren't behaving themselves.

Leaving aside fundamental issues such as inadequate backups there are several other pathetic aspects to this story. E.g. M. Martin and his crying over this "cruel" attack. Weak as water.

AndrewJRenko

https://www.boards.ie/discussion/comment/118303394#Comment_118303394

They're not my tangents. I didn't bring up any of the tangents. The other poster did.

There's oodles of evidence of how badly the HSE screwed up here. I'm not sure why anyone would need to make up fairy stories about what happened.

https://www.boards.ie/discussion/comment/118304508#Comment_118304508

I would t call the person who clicked on the link a donkey. I would call the person in charge of totally failing to implement security a donkey. But that would be to insult donkeys. What people forget is the human factor. Most people in tech are sitting at a computer doing one thing at a time. Users with many core duties in front line healthcare, like resuscitating patients, have multiple things going on the minutes before and after opening emails.

If an email arrives apparently from what looks precisely like usual supplier X (the cyber attackers having previously intercepted suppliers etc and replicated/simulated their sites) and a busy staff member clicks into the excel sheet with credentials to order another month’s supply of cannulae then that is the beginning of the issue. I’m sure suspicions begin to arise when said cannulae don’t arrive and the legit company is contacted, and when the attack happens the two incidents get linked and proved to be the causation upon investigation.

Most air accidents in history have happened because of the “donkey” (human) factor, and enormous leaks have been made to make aviation by far the safest means of distance transport. Senior cybersecurity experts are still to get to grips with the psychology factor and still seem to rely a bit too much on the tech aspect. It needs to be 50/50, as humans are the ones this far engineering the attacks and they seem to be ahead of the curve in understanding how people actually respond to and use tech. Of course they have the leisure of time to do this as the payouts are enormous for months of thought, whereby busy end users and cyber security people are just trying to keep up with the tasks put in front of them.

AndrewJRenko

https://www.boards.ie/discussion/comment/118305386#Comment_118305386

Yes, absolutely right, I meant to come back on the 'donkey' point myself.

Jim Browning from NI, who has made a career, and a living, and a highly entertaining YouTube channel out of scamming the scammers, was sucked in over the summer, resulting in the temporary loss of his YouTube channel.

He's no donkey. If he can be scammed, anyone can be scammed.

There is indeed an important point about end user training. This is particularly challenging in the health service environment with high turnover of 'permanent' and daily turnover of agency staff. But it is possible - they did an online certification for hand washing in the early days of covid, so they should have an online certification for being able to use a computer safely.

https://www.boards.ie/discussion/comment/118305560#Comment_118305560

I saw that!

I worked for a smallish (but not that small!) of the public service, running Windows 7 for internal admin work and a Unix system for our dedicated core operations for the public. Ordinary staff like myself, with other core duties, were trained to be admins alongside the other stuff we did. It would kind of happen by default, as someone from the wider big parallel organisation would ask “who is the Unix admin your side?” and somebody would be “volunteered” by the local manager because “you are good at this kind of stuff”. There were absolutely no dedicated employees in-house. It has been quite a public service thing to send people on various short courses to train them up un various areas to save on employing people specific to tasks in organisations like mine, consisting around 150 people.

We would run patches frequently enough, probably a lot better than in the HSE by the sounds of it. In fact we had a lot of down-timing our systems to keep them updated; most would be run at night, though some into the work shifts.

Regarding backups, the organisation I worked for once had its data in a certain building in town which suffered a catastrophic small fire which destroyed the entire databases, in the 90s. There simply were no backups. It meant a massive amount of overtime, evenings, weekends, over several weeks, to put up the entire data again. Staff made a small fortune out of it, bless that little fire that thankfully harmed nobody. There had been nobody responsible for data maintenance in the organisation even though our services were entirely grounded on databases. Needless to say, an expensive lesson was learned and a fairly decent system of backups was instated.

https://www.boards.ie/discussion/comment/118301907#Comment_118301907

Yes of course, we all pretty well know this. Like we know not to try and sneak through orange at the lights, etc. Yes, we should always look carefully at the sender address and hopefully be actually familiar with all legitimate sender addresses.

But… and I will use an analogy from my own former workplace, where staff would be tasked to select books, magazines and other items online. We would get an email from HQ asking us to complete orders by the end of the week from X Supplier, who will be sending an email. Let’s call them Dublin Book Suppliers. We have been using several book suppliers, and occasionally some have been changing the appearance of their sites. So let’s pretend a hacker (goodness only knows why any hackers would be arsed over books😂 but I’m simply extrapolating own workplace to a much more pressured HSE scenario) clones part of Dublin Book Suppliers site, enough to make it seem legit when the staff member duly responds to the expected email. The staff member is dealing with a queue at the desk at the same time as ordering, there is no luxury of doing this in some back office where I worked. Eyes on the email/website, a moment later eyes on the site the customer in front of you needs accessing. It is easy to slip up when under a lot of pressure. Staff member gets a gap in the queue, quickly clicks on link of what seems entirely legit email, and with time pressure to complete the rest, the trap door has opened.

In the case of a medical front line worker, the customers in the queue are sick patients. That worker is required to do a heck of a lot of admin these days in addition to core duties. Distraction & defocussing can cause errors, and a medical worker will tend to mentally prioritise the immediate needs of patients. It needs to be factored into devising systems.

plodder

https://www.boards.ie/discussion/comment/118304508#Comment_118304508

"The liars/incompetents who spun this as being a zero day exploit?"

Right. Some of us were very sceptical about that claim, and there's no mention of it in the report. In fact, the report makes the point that it was a mitigating factor of the attack that "it would appear that the Attacker used relatively well-known techniques and software to execute their attack."

That story was concocted as a way of deflecting blame.

Remember also the palaver about the decryption key not being usable initially. Here's what the report says about that:

"It is unclear how much data would have been unrecoverable if a decryption key had not become available as the HSE’s backup infrastructure was only periodically backed up to offline tape."

They would still be out of action today, if they hadn't got the key.

Northernlily

Between our banks and civil service, clearly people operating at strategic levels who don't have a clue and fail to keep themselves up to speed in the face of ever hanging threats.

The ah sure it will be grand attitude doesn't cut it.

Wombatman

Went through the report. PWC obviously trying to paint their paymasters in the best possible light. A difficult task giving the LOL bad state of the HSE's cyber-security management and infrastructure. 75% of the report is boilerplate and the key detailed technical timeline has been redacted.

Some beauties from the report:

“On 18 March 2021, a HSE staff member interacted with a malicious Microsoft Office Excel file attached to a phishing email. This resulted in a Malware infection of the Patient Zero Workstation.”

Why didn’t the mail filter pick up on this email? Why didn’t anti-malware stop the infection on the client? No EDR system in place?

“On 31 March 2021, the HSE’s antivirus software detected the execution of two software tools commonly used by ransomware groups: Cobalt Strike and Mimikatz, on the Patient Zero Workstation. The antivirus software was set to monitor mode so it did not block the malicious commands.”

WTF??

“On 8 May 2021, first identified evidence of the Attacker compromising systems within Hospital K and Hospital D.”

What was the response to this?????

“Timeline prior to the Incident and the response at Hospital A, Hospital C and the DoH

Two voluntary hospitals, Hospital A and Hospital C, identified suspicious activity prior to the Incident. In addition, the DoH, a third party to the HSE’s environment, successfully acted on a detection of the Attacker which prevented the execution of the Conti ransomware across the vast majority of the DoH.

The following timeline describes the key activities at Hospital A, Hospital C and the DoH prior to the Incident.

On 10 May 2021, Hospital C asked Hospital C’s cybersecurity solutions provider whether they should be concerned about Cobalt Strike alerts. They were advised by Hospital C’s cybersecurity solutions provider that since the threat had been remediated by their antivirus software, their risk was low. Hospital C did not initiate a cyber incident response investigation.

On 12 May 2021, Hospital A engaged Hospital A’s Incident Response provider to investigate alerts of malicious activity. They reset passwords for 4,500 accounts and made firewall configuration changes56 to contain the activity, and made contact with the HSE to request information on two IP addresses. To further contain the activity, Hospital A utilised their existing security tooling across their environment. On 13 May 2021, the HSE identified the IP addresses reported by Hospital A related to two servers within the HSE’s domain. The HSE conducted an investigation into the activity identified by Hospital A and incorrectly concluded in an email between the HSE teams58 that the suspicious activity originated from Hospital A, rather than the other way round.

On 13 May 2021, DoH’s cybersecurity solutions provider59 alerted the DoH to a potential attack on their network. DoH contacted the NCSC and engaged DoH’s IR Provider who installed endpoint detection and response (“EDR”) security tooling on the majority of their systems. These actions blocked the execution of the Conti ransomware across the vast majority of the DoH’s infrastructure, including critical and data servers.”

Just after the event the line from the HSE and some on here was “Ah sure if these Russian lads want to target you there is fook all you can do. The boys inside are killing themselves trying to contain the attach. They are heroes.

This is obviously a crock of sh1t. The HSE could have largely prevented the worst of the attack, like the DoH, did if they hadn’t their heads up their holes.

The anti virus was set to monitor only? 😂

Wombatman

https://www.boards.ie/discussion/comment/118310381#Comment_118310381

Not the end of the world if somebody was notified and took action. Did that even happen?

cafflingwunts

its a joke that this methodology, the tools and the whole attack plan itself is public knowledge nowadays to anybody with even a slight interest in this sort of thing.

Wombatman

How is Paul Reed still in a job? This, covid madness, the children's hospital, Sláintecare. Guy has been caught lying and misleading on numerous occasions. Don't get me started on that dolt Donnelly. Us laughing a Bojo across the water and this sh1te is going on here.

Igotadose

https://www.boards.ie/discussion/comment/118310458#Comment_118310458

That report's a doozy alright. Some rando workstation running some janky antivirus and still the compromise was detected but nothing happened. Attackers probably couldn't believe their luck.

The report likes to repeat itself. Kind of ridiculous to repeatedly show HSE orgcharts - at least, not under a section entitled "Bloated management structure that led to the systemic infrastructure problems causing this attack." Months of notice, no backups, and their solution is to decrypt all the servers that were encrypted - I wonder if anything was left behind by the encryption process, too.

About the only thing I will say in defense of the HSE is that getting this stuff right is very expensive and time consuming and requires really smart people to do it. Even then, companies get hacked all the time. Not having reliable offline backups is inexcusable. Thank goodness the Covid vaccine rollout was on Cloud servers, presumably administered by owners that were not the HSE and protect them.

AndrewJRenko

https://www.boards.ie/discussion/comment/118310064#Comment_118310064

In fairness, all kinds of organisations, public and private, all round the world, including those who earn a living advising others about how to prevent these attacks (Accenture) have been hit badly in recent months. It's not specific to banks or civil service, and it's not specific to Ireland.

irishgeo

the anti viurs on computer 0 hadnt been updated in a year.

Wombatman

https://www.boards.ie/discussion/comment/118311806#Comment_118311806

Where did you get that from?

Wombatman

https://www.boards.ie/discussion/comment/118311031#Comment_118311031

"Ah sure there is nothing you can do against these boyos". Tell that to the DoH.

The vulnerable are targeted and exploited because of ignorance, negligence or incompetence. I think we have all three in the case of the HSE, laced with some hubris from senior management.