Ransomware & HSE

magicbastarder

Pandiculation wrote: »

I worked in one place where you weren’t allowed enter certain areas with any kind of device with memory, smartphones, USB sticks etc. Doing so would have resulted in being fired, as there was a risk of someone accessing systems.

you should see the machine symantec have to trash devices leaving their virus research lab in ballycoolin. basically turns computers/phones/tablets to tinsel, i have been told (alas, i did not see it in action).

Pandiculation

The HSE could put them on a waiting list to speak to a cyber security consultant. By the time they get the appointment in 2031 they’ll give up and apologise.

leahyl

Pandiculation wrote: »

Teams, Zoom etc .. a lot of corporate networks are heavily locked down and were never intended to use conferencing facilities. The pandemic caused a organisations to be catapulted into using things that they never would have normally.

Healthcare in particular is hugely problematic, as you’d have lots of personal data.

Financial services and infrastructure providers are even tighter. I worked in one place where you weren’t allowed enter certain areas with any kind of device with memory, smartphones, USB sticks etc. Doing so would have resulted in being fired, as there was a risk of someone accessing systems.

Fair enough, don't like being misquoted though, as I didn't mention Zoom - it's not the same thing and Zoom would have been a bit of an issue in the University for a while also. You'd imagine a microsoft application would be somewhat more "trustworthy". I was accused of being a moaner also simply from stating my experience of things - jaysus! Can't say anything now and you're a complainer and a moaner (I know you weren't the original poster!)

Pandiculation

You’d imagine, but there’s nothing more trustworthy about them. Some of the most common vectors for viruses and malware are Microsoft applications like Outlook because they’re ubiquitous.

Secure IT environments don’t necessarily allow users to install anything.

Teams, for example, includes collaboration services that would potentially grant access to screen sharing, which is a BIG *no* in some contexts and health is likely one of those.

Prisoner6409

Are their IT people living in the land of Nod, who did not see this coming so much so they would have an off site backup that is maintained off the Internet, one live system and at least 2 backups, 1 not connected, its hardly rocket science.

biko

.42. wrote: »

Are the HSE still using redundant OS like Windows XP?

It's not "redundant", it's outdated.

But yes, I think quite a few of their systems are outdated. Too expensive to stay updated I assume.
They should just go full Linux.

BailMeOut

When they figure out what happened it will on all likelihood come down to an individual with higher-level access being tricked into doing something stupid to allow access. We all think these hacks are very sophisticated but usually are very low tech and the weak link is a human who clicks something, installs software, or gives out information over the phone to the bad folks which then allowed access. From what I am reading this hack is about data loss/theft so the hackers are simply copying data from the HSE to their systems and probably just using the permission of the user or admin who had access.

It's very tough to prevent humans from doing stupid things and the HSE will have layers of systems, processes, and training in place to stop people from doing stupid things but the bad folks will always find a way around this.

ixoy

Prisoner6409 wrote: »

Are their IT people living in the land of Nod, who did not see this coming so much so they would have an off site backup that is maintained off the Internet, one live system and at least 2 backups, 1 not connected, its hardly rocket science.

Even if they did - Do you know where the trigger for the ransomware is? Do you know when it is installed? Attackers could have left it dormant for months, embedded in the system, so a backup could just restore the same security hole.

madcabbage

Social engineering is the by far the most common cause of these attacks. But saying that, it could be a number of factors.

heroics

BailMeOut wrote: »

When they figure out what happened it will on all likelihood come down to an individual with higher-level access being tricked into doing something stupid to allow access. We all think these hacks are very sophisticated but usually are very low tech and the weak link is a human who clicks something, installs software, or gives out information over the phone to the bad folks which then allowed access. From what I am reading this hack is about data loss/theft so the hackers are simply copying data from the HSE to their systems and probably just using the permission of the user or admin who had access.

It's very tough to prevent humans from doing stupid things and the HSE will have layers of systems, processes, and training in place to stop people from doing stupid things but the bad folks will always find a way around this.

If you look at the info about conti earlier in this thread it both extracts data and also encrypts the data that is left behind. Uses a number of tools to gain elevated privelages and laterally move through an environment.

Another link with different initial attack to previous link
https://news.sophos.com/en-us/2021/02/16/conti-ransomware-attack-day-by-day/

Pandiculation

The reality is we don’t know. The HSE systems probably are complex, involving a large number of instituons and legacy systems and have multiple risks, but I would also doubt they are as bad as some people seem to imagine.

Most healthcare organisations have faced this kind of issue. There were major cyberattacks on Spanish hospitals at the peak of the pandemic impact in Spain. The NHS has been hit multiple times and so on.

The shut down of systems is a standard way of quarantining them until you know what you’re dealing with and how much exposed they are.

It’s likely the attack is isolated to one system, so they’ll be able to safely restart systems bit by bit.

magicbastarder

one thing to bear in mind about the HSE, and i'm comparing it to the large multinational i work for - we've the luxury of being large, but contained and relatively homogeneous.
the HSE have to maintain systems which are not only massively diverse in terms of technology, but also massively diverse in terms of geography. you'd probably find that on plenty of sites, IT help is half an hour away at the very best. i would hate to work for them, it'd be a nightmare.

leahyl

leahyl wrote: »

Try reading my post - where did I mention Zoom???

Apologies, I meant to say "zoom, teams, etc".

I did not mean to misrepresent your post and I'll edit it now. My point still stands. Blocking the installation of non-standard applications is a good thing and indicative of a good ICT policy and infrastructure, and not a bad one (which you implied).

BailMeOut

heroics wrote: »

If you look at the info about conti earlier in this thread it both extracts data and also encrypts the data that is left behind. Uses a number of tools to gain elevated privelages and laterally move through an environment.

Another link with different initial attack to previous link
https://news.sophos.com/en-us/2021/02/16/conti-ransomware-attack-day-by-day/

Thank you. Did not know there were encrypting the data left behind. This looks very similar to the US pipeline hack.

Prisoner6409

Prisoner6409 wrote: »

Are their IT people living in the land of Nod, who did not see this coming so much so they would have an off site backup that is maintained off the Internet, one live system and at least 2 backups, 1 not connected, its hardly rocket science.

How do you know they don't?

Just for clarity, I have been working in IT Security for approximately 15 years and currently work with some nationally critical systems, though not the HSE. We have reached out to them and offered our services though.

The discussion about out of date software is something of a red herring because even if they were running Windows 10 across the board, the threat actors could still leverage a zero day exploit to get access. A zero day exploit is an exploit for which there is no patch. So they could use a sophisticated exploit, or it could have been as straight forward as an Excel file with a macro.

Also, I'd like to point out that shutting down the network was the right thing to do and would commonly be part of an Incident Response Plan.

penny piper

Badly fukt wrote: »

No mostly windows 7 though which is also end of life

I think some in the hse are using windows 10.

what?

slightly tangential, this is a proof of concept
https://www.zdnet.com/article/academics-steal-data-from-air-gapped-systems-using-pc-fan-vibrations/

air-gapped are now just networks with extreme latency :-)
literally a game of whack a mole

Prisoner6409

Prisoner6409 wrote: »

Are their IT people living in the land of Nod, who did not see this coming so much so they would have an off site backup that is maintained off the Internet, one live system and at least 2 backups, 1 not connected, its hardly rocket science.

Kind of the way things used to be done back in the day with non-internet networks. I know you're suggesting an internet based system be backed up on a disconnected server from which the data is transported by a human to an offsite location to mitigate against fire risk. Then you add the risk of the human being having the portable device stolen off them the way we used to often hear about laptops going missing with precious data compromised.

JDxtra

Times have changed since, but I remember around 10 years ago a company I was working with had a standby disaster recovery site which was online and ready in a recovery centre. We could flip between live and standby as needed.

The folks from HSE IT were using the same recovery centre. They were testing their recovery processes one day, which involved wheeling in servers on a trolley with a load of recovery tapes. It takes an incredible amount of time to recover systems and data in this manner.

More recently on the RTE News (2 years ago?) they showed the inside of the HSE IT dept. as part of a report. In the background there were Windows XP machines still running - and this was within IT (i.e. not connected to some legacy radiology machine).

ShayNanigan

I don't understand why they are still using Windows! From what I've seen, in many European countries they use Linux for several reasons.

Also this bit in the Journal article sort of made me smirk: "A contingency plan has been put in place to revert back to the “old-fashioned” paper-based system". Isn't that what they normally use most of the time anyway...? Time for an upgrade in the systems in any case I'd say. There's a good chance of mistakes daily just because either some papers go missing or they give a patient the wrong medication (unless the patient is very observant) because they have no system where to check. And of course then there's these types of cyber attacks because the system they do have in place is running on Windows.

KildareP

Prisoner6409 wrote: »

Are their IT people living in the land of Nod, who did not see this coming so much so they would have an off site backup that is maintained off the Internet, one live system and at least 2 backups, 1 not connected, its hardly rocket science.

If the source of the ransomware is some generic back office PC, then restoring from backup and/or a hot standby system will get you absolutely nowhere, the ransomware will start making instant mincemeat of that too.

You have to find the source of the compromised PC and that's no easy feat. Imagine someone clicks a dodgy link five minutes before heading home and leaves their PC turned on - that PC could progress through terabytes of data overnight. In the meantime, the user of that PC is none the wiser, most other staff who might notice are also gone home, you finally spot something is wrong but now have tens of thousands of PCs to blindly scan through (which PCs do you prioritise first?) and by the time the problem user gets in the next day to notice something is far worse on only their PC they will be one of thousands choking the helpdesk to say they can't get onto their systems.

Any PC from the last 5 years and a good office network could easily encrypt 100GB+ of data in about half an hour, not only on the compromised PC itself but on fileshares, network drives and exploits in other PCs running an unpatched OS. That can do serious damage to a network in just a few hours.

sphinxicus

JDxtra wrote: »

Times have changed since, but I remember around 10 years ago a company I was working with had a standby disaster recovery site which was online and ready in a recovery centre. We could flip between live and standby as needed.

The folks from HSE IT were using the same recovery centre. They were testing their recovery processes one day, which involved wheeling in servers on a trolley with a load of recovery tapes. It takes an incredible amount of time to recover systems and data in this manner.

Ahh memories. Takes me back to when we used to do our annual DR test. back in the early 2000's. We had a Hot DR site like yourselves. Of course, its no help when file systems are being encrypted and this is being replicated to the DR site in near real time.


Read-only file and block snapshots a big lifesaver here.

Hurrache

ShayNanigan wrote: »

I don't understand why they are still using Windows! From what I've seen, in many European countries they use Linux for several reasons.

Because it works.

BailMeOut

JDxtra wrote: »

Times have changed since, but I remember around 10 years ago a company I was working with had a standby disaster recovery site which was online and ready in a recovery centre. We could flip between live and standby as needed.

This would not help in this situation as the DR site is a replica of the production site so if the one is encrypted the other would be as well. The only way to get your data back is to decrypt it (pay the ransom) or recover from a backup.

AndrewJRenko

Prisoner6409 wrote: »

Are their IT people living in the land of Nod, who did not see this coming so much so they would have an off site backup that is maintained off the Internet, one live system and at least 2 backups, 1 not connected, its hardly rocket science.

Why would you assume that they don't have these backups?

dubrov

JDxtra wrote:

Times have changed since, but I remember around 10 years ago a company I was working with had a standby disaster recovery site which was online and ready in a recovery centre. We could flip between live and standby as needed.

Was the database replicated across both sites in real-time as well? If so then it also would have been encrypted and inaccessible.

The only option is to restore from a backup but the data would be old and could be reencrypted if the source of the malware is not removed first

AndrewJRenko

biko wrote: »

It's not "redundant", it's outdated.

But yes, I think quite a few of their systems are outdated. Too expensive to stay updated I assume.
They should just go full Linux.

What's the cost of retraining 100k users to Linux?

PatrickSmithUS

ShayNanigan wrote: »

I don't understand why they are still using Windows! From what I've seen, in many European countries they use Linux for several reasons.

Also this bit in the Journal article sort of made me smirk: "A contingency plan has been put in place to revert back to the “old-fashioned” paper-based system". Isn't that what they normally use most of the time anyway...? Time for an upgrade in the systems in any case I'd say. There's a good chance of mistakes daily just because either some papers go missing or they give a patient the wrong medication (unless the patient is very observant) because they have no system where to check. And of course then there's these types of cyber attacks because the system they do have in place is running on Windows.

There's no issue with using Windows if you have the correct system in place. All it would take is a decent email spam filter (explainer here), proper off site back ups (which should be standard for every Govt body) and some other AI solutions and this would have been either avoided or mitigated pretty quickly.


It's nigh on impossible to stay ahead of cybercriminals and the Babuk strain that is after rearing its head recently might be to blame here. https://www.theguardian.com/technology/2021/may/11/washington-police-hack-russian-speaking-babuk-gang

Hurrache

AndrewJRenko wrote: »

What's the cost of retraining 100k users to Linux?

As well as contracting professionals to port over or create APIs to allow all the systems communicate, and then have the various vendors supply applications running across Unix and Linux.

It's a stupid idea.

LillySV

Hope the bastards who did this die roaring

BailMeOut

Hurrache wrote: »

As well as contracting professionals to port over or create APIs to allow all the systems communicate, and then have the various vendors supply applications running across Unix and Linux.

It's a stupid idea.

+ this hack used a Linux command to steal the data! (rclone)

Stewball

The first page of this thread is probably the most idiotic collection of posts I've ever read on boards.

leahyl

Deleted User wrote: »

Apologies, I meant to say "zoom, teams, etc".

I did not mean to misrepresent your post and I'll edit it now. My point still stands. Blocking the installation of non-standard applications is a good thing and indicative of a good ICT policy and infrastructure, and not a bad one (which you implied).

Fair enough, thank you for the clarification

AndrewJRenko

AndrewJRenko wrote: »

What's the cost of retraining 100k users to Linux?

To the average end user a modern desktop Linux system wouldn't be a massive leap. Then I remember my days training end users on new updates and applications and it was like dealing with big kids, the resistance to changes even for minor things like going to 365 from an older version of Office was unreal.

Supporting Linux could be an issue. Of the people that work with me only 3 of us would have any experience with Linux or the need to regularly use it. Many would have done some stuff at college and never used it again so would need some refreshing.

JP Liz V1

Cancer appointments cancelled those poor patients

magicbastarder

Deleted User wrote: »

The discussion about out of date software is something of a red herring because even if they were running Windows 10 across the board, the threat actors could still leverage a zero day exploit to get access.

this is true - but if a target is running XP say, your options for compromising the system are far more open.

circadian

I wouldn't doubt for a second that upgrades to the IT system has been recommended for a long time. We seen recently some third level institutions suffering from these attacks. It was obvious institutions in Ireland were being targeted.

Let's face it, anyone in an IT management role should be treating ransomware prevention as a regular maintenance task. It's not difficult to mitigate against although I suspect a lack of funding or willingness to fund upgrades being a problem here. More often than not IT departments are seen as a cost that needs to be kept down, I've worked in enough companies to see how non tech companies often underfund their IT services, including one that got hit with ransomware and got wrecked in 2015. I promptly handed in my notice as I was the one thrown under the bus despite raising the flag constantly.

I doubt those responsible in the civil service will ever get questioned, even worse, they'll probably move to another department and continue to make the same mistakes.

madcabbage

Most of the systems are either Windows 7 or Windows 10

magicbastarder

did the HSE purchase extended support for Win7?

JP Liz V1

JP Liz V1 wrote: »

Cancer appointments cancelled those poor patients

Appalling. We have such a post Covid backlog and now this.

As for the Windows v Linux argument. Windows is perfectly fine if you keep on top of the latest cybersecurity updates and the organisation is informed and trained on latest security risks. I wonder did someone click onto a phishing email which allowed them in? Anyone know?

Corruptedmorals

This is insane. And naturally enough it's the hospitals and departments that have gone to electronic records that are affected. My department is still on charts so we are fine but so many appointments cancelled it's a disgrace. I pity anyone going into labour or having emergency surgery with no notes available. Dangerous isn't the word.

whippet

Malik Clean Stairwell wrote: »

Appalling. We have such a post Covid backlog and now this.

As for the Windows v Linux argument. Windows is perfectly fine if you keep on top of the latest cybersecurity updates and the organisation is informed and trained on latest security risks. I wonder did someone click onto a phishing email which allowed them in? Anyone know?

I'd say the recovery team in the HSE don't know this yet ... they may have an idea but all efforts will be on locking down and then developing a plan to bring systems back online

twinytwo

magicbastarder wrote: »

did the HSE purchase extended support for Win7?

They did.

Head of IT Ops from the HSE on the radio now saying they believe it was a zero day exploit.

BrianD3

whippet wrote: »

I'd say the recovery team in the HSE don't know this yet ... they may have an idea but all efforts will be on locking down and then developing a plan to bring systems back online

Absurd if all of this is because somebody clicked on a link or attachment in an unsolicited email. Any consequence for them?

IME these emails are very easy to spot even if they are more clever than the "click here to win money!!" type emailed links that we used to see.

If someone uses a computer as part of their job, it's also part of their job not to make these errors. Just as if someone works in a warehouse full of valuable stock, it's part of their job not to leave the door open and alarm off.

AndrewJRenko

circadian wrote: »

I wouldn't doubt for a second that upgrades to the IT system has been recommended for a long time. We seen recently some third level institutions suffering from these attacks. It was obvious institutions in Ireland were being targeted.

Let's face it, anyone in an IT management role should be treating ransomware prevention as a regular maintenance task. It's not difficult to mitigate against although I suspect a lack of funding or willingness to fund upgrades being a problem here. More often than not IT departments are seen as a cost that needs to be kept down, I've worked in enough companies to see how non tech companies often underfund their IT services, including one that got hit with ransomware and got wrecked in 2015. I promptly handed in my notice as I was the one thrown under the bus despite raising the flag constantly.

I doubt those responsible in the civil service will ever get questioned, even worse, they'll probably move to another department and continue to make the same mistakes.

What mistakes did the HSE IT folks make?

topdecko

This is a huge issue. Healthlink down for us in GP land and we don't seem to have a back up in place. We cannot refer for COVID tests now and there will be backlog with other referrals, hope they get sorted soon however the haphazard nature of irish Health IT infrastructure is very concerning. Absolutely no plan in place to respond to this - these attacks are part of modern life, must have contingency ready to go - not merely cancelling appointments etc.

Hamsterchops

Thankfully all the Anti Virus software is up to date & the latest version, so that's good news. The HSE also acted very quickly to shut down all systems . . .

So how much is the ransom? and who's behind it?

Pandiculation

Unknown - apparently they’ve made no demands yet. It’s a human directed attack though, not a purely automated thing.

BrianD3

BrianD3 wrote: »

Absurd if all of this is because somebody clicked on a link or attachment in an unsolicited email. Any consequence for them?

IME these emails are very easy to spot even if they are more clever than the "click here to win money!!" type emailed links that we used to see.

If someone uses a computer as part of their job, it's also part of their job not to make these errors. Just as if someone works in a warehouse full of valuable stock, it's part of their job not to leave the door open and alarm off.

I was just in a meeting and one of my colleagues had been involved in phishing training. After the training, they sent a phishing mail to the attendees. Literally, the next email they received, was a phish. 20% of the attendees clicked the link. 20%! Having literally just received the training.

Some people: