Blacknight Hacked?

bluemachaveli

Just after reading an email from yourselves about user data being comprimised!

Any idea what details other than email address are out in the open?

Morlar

Got the email too. I thnk customers have a right to know where this came from, who it was directed at and how long it lasted.

Telling customers 'Change passwords' is woefully insufficient in terms of customer service.

Boards.ie: Neil

Guys give them a chance.

We've been there as well, and the scars are still here in this office and it's been 2 years. I can still remember the day and the weeks following and the stress that it forced upon in here.

There are procedures they have to follow with regards to a data breach and I'm sure they are doing them. It's such a time sensitive issue that they probably can't release any more information than they have at the moment.

The guys at Blacknight are decent chaps (use them for my personal hosting and domain names) and are a pleasure to deal with. I have no doubts that they'll handle this to the best of ability.

sf80

They'll have a lot to deal with, and they'll need to do a lot of analysis and talks with the Gardai before they can fully inform their customers.

Morlar

sf80 wrote: »

They'll have a lot to deal with, and they'll need to do a lot of analysis and talks with the Gardai before they can fully inform their customers.

So long as they do actually inform paying customers.

johnmurph01

*facepalm*

I got the email this morning.

It never ceases to amaze me how arrogant they can sound when something happens. Same with the time their email system went south a few months back for 48 hours.

Yeah, it takes time to investigate, but whats with the crappy email and tweets. Time for some humility on their part. Stick your hands up and stop dodging.

cian1500ww

I just got a fake paypal email sent to the email address I had registered with Blacknight: http://twitpic.com/8ep4pq I mentioned it on twitter as well in case anyone falls for it.

johnmurph01

Actually, thinking about it...im done with them. Im going to move my VPS and client's to vps.net or dediserver

- Mail up and down like a yoyo for months

- Mail down for 48 hours solid in 2011

- VPS server with no backups for 1 client last year = Data loss. (because of their hardware failing)

- Blacknight on tax DEFAULTERS list for VAT (721,291.08 quid - http://www.carlowpeople.ie/news/carlow-tax-defaulters-appear-on-revenue-commissioners-list-2881702.html). Software error my eye. I was worried at first, checked their accounts this year on the CRO and they're nowhere near having it paid off.

- Servers up and down like a yoyo aswell (check their status page - says it all). Even spotted one of their engineers saying 'Since I start covering on call for @blacknight over 2 years ago, 16,382 SMS alerts were sent to my phone.' - work that out per day!!! Alot of issues there.

https://twitter.com/#!/bkenny/status/162525581244764160

Once they get through their backlog of support mails, they'll find my cancellation. One step too far at this point.

mneylon

bluemachaveli wrote: »

Just after reading an email from yourselves about user data being comprimised!

Any idea what details other than email address are out in the open?

At this juncture customer contact details.
We know for sure that email addresses were affected

We're still investigating and are in contact with various 3rd parties who are assisting us.

Pacing Mule

Hi there Blacknight,

First off I'd like to say as a cutomer of yours that I'm very sorry to hear of this hack for everybody concerned - including yourselves who are in the front face of the fallout. I hope things work out for you.

I have a major concern though that needs to be addressed sooner rather than later. The email advises changing customer passwords to the control panel. Have you any indication at all that these have been compromised or is this a simple precaution ?

As you're no doubt well aware control panel access can be used to get to everything else on the server including databases which would in turn would compromise all of your customers own customers / registrant details for forums etc. This is worrying and on far greater scale to boards.ie's own experiences and I would appreciate if it could be confirmed as soon as possible if these passwords were compromised.

Quick Edit - Would also in fairrness like to point out that my own control panel logs show no access being made on my account other than myself.

mneylon

Just to confirm, no passwords that we can see have actually been compromised and all passwords and sensitive financial information (such as credit cards) are encrypted.

From what we can see here the only information compromised is your first name, last name, and email address.

However as a precaution we are advising customers to change their passwords just in case.

Morlar

Blacknight wrote: »

Just to confirm, no passwords that we can see have actually been compromised and all passwords and sensitive financial information (such as credit cards) are encrypted.

From what we can see here the only information compromised is your first name, last name, and email address.

However as a precaution we are advising customers to change their passwords just in case.

Can you confirm - do you plan on informing customers where this breach came from, how long it lasted and if it was directed against specific customers ? What level of detail are you intending to provide ?

Pacing Mule

Blacknight wrote: »

Just to confirm, no passwords that we can see have actually been compromised and all passwords and sensitive financial information (such as credit cards) are encrypted.

From what we can see here the only information compromised is your first name, last name, and email address.

However as a precaution we are advising customers to change their passwords just in case.

Thanks for the reassurance.

mneylon

Morlar wrote: »

Can you confirm - do you plan on informing customers where this breach came from, how long it lasted and if it was directed against specific customers ? What level of detail are you intending to provide ?

We are liaising with IRISS, Data Protection and I believe one of our managers is in contact with the Gardai

We cannot provide any further details at this time as it could compromise an ongoing investigation.

col.in.Cr

Im a customer too and I just got that paypal spoof email as well

Kohhal

I got it too and am a customer. Also having client site issues which may be related - investigating now...

col.in.Cr

I think you need to send out another email warning your customers of that paypal spoof email.

Creamy Goodness

col.in.Cr wrote: »

I think you need to send out another email warning your customers of that paypal spoof email.

No they do not, if you fall for that glaringly obvious paypal phishing attempt you deserve to lose all your money.

Oh and they aren't related I get these every two to three days.

Morlar

Creamy Goodness wrote: »

No they do not, if you fall for that glaringly obvious paypal phishing attempt you deserve to lose all your money.

No you don't.

Creamy Goodness wrote: »

Oh and they aren't related I get these every two to three days.

If there is suspicion of a connection Blacknight need to update the vast bulk of customers who do not read this thread.

MacGyver

I don't think the paypal one is anything to do with blacknight, I've been getting those about once a week since before christmas

Kohhal

I rarely if ever get spam making it to my inbox in gmail - I did get this paypal one today, coincidence... who knows

col.in.Cr

Kohhal wrote: »

I rarely if ever get spam making it to my inbox in gmail - I did get this paypal one today, coincidence... who knows

same here

Morlar

I did not get the pp mail.

Phibsboro

Just a +1 on the paypal scam email, and ive never seen it before

TouchingVirus

What encryption was used on the passwords Blacknight? Did you md5 them ? md5 + salt them? were the salts unique per user? sha1? sha256? sha512? A little more detail on this would go a long way

I can confirm that the email address from my last job which I used for Blacknight has been sent a Paypal scam email and has never received anything like it before (either in Spam filter, or right to the inbox like this).

Scam Paypal Email wrote:

Subject: PayPal: Your account has been temporarily limited. CASE ID: 19Xu32
From accounts <notice@limitedpaypal.com>
Date Thursday, February 2, 2012 11:17
To: <redacted>
Message:

Hello,

Your account has been temporarily limited.
To remove the limitation from your account please
proceed right away and update your account details.

For confirmation, please click the link below:

Log on to your PayPal account

Pure phishing stuff and the link throws a Reported Web Forgery error for me in Firefox/Chrome - if you have the brains to buy a domain/hosting then you shouldn't fall for this.

Wendolene

TouchingVirus wrote: »

What encryption was used on the passwords Blacknight? Did you md5 them ? md5 + salt them? were the salts unique per user? sha1? sha256? sha512? A little more detail on this would go a long way

Hi Blacknight,

Firstly - you all have my sincere sympathies on the sh!tstorm that is your workplace atm.

[ReallyReallyNotWantingToSoundCondescendingBut...]

I've been there, and as a customer, I want you to come out of this. Most customers are mature and experienced enough to accept that breaches happen when it can be demonstrated that reasonable precautions were in place.

You have had a number of requests (both here and on Twitter) to confirm that passwords were hashed. These requests have not been answered directly by you.

People need a simple, clear and concise answer to these requests to allay their current fears about the integrity of their passwords. Denying, delaying or obfuscating such an answer only heightens peoples fears for their services and further damages their confidence in you as their service provider.

Do yourselves (and your customers) a massive favour and give a simple answer to a simple question, please ... and we can all rest a little easier and let you get back to the job of clearing up the mess.

We don't need the nitty gritty details of the hashing / salting - divulging that to us would also potentially divulge it to the neerdowells, and we don't want to make their lives easier - but we do need a little more on this from you.

Oh, and BTW, (before anyone suggests it might) such a concise reply will not adversely affect ongoing investigations. You have an opportunity to quell some grave worries - please take it.

Yours most sincerely,
Wendolene.

jh385

I've already emailed BK support this. But I'll +1 the request to know how the passwords are stored; plaintext or hashed?

It's a simple question, really.

mneylon

Critical user data is encrypted using 2 factor encryption.

However we don't know for sure if the control panel passwords were decrypted or not.

We are asking people to change passwords only as a precaution.

jh385

Blacknight wrote: »

Critical user data is encrypted using 2 factor encryption.

Thanks, but that really doesn't answer the question.

Sure, the database file or 'user data' might be encrypted. But were the user control panel passwords stored in the table in plaintext or hashed form?

If they were stored in plaintext, then it doesn't make much difference how the file was encrypted, if the encryption was compromised then the passwords are out there.

ETA: I'm not asking this to hang anyone - I genuinely want to know so I can assess the risk myself. I have a ton of databases, sites with other hosts etc. that might be using the same password. It'll take me hours to go through changing passwords everywhere. I need to know if I have to go to that effort, or if I'm willing to take the risk.

Amalgam

What about back ups?

Any sign of duplication?

If I'm running a PHPBB Forum on your site, I presume the Admin passwords should get a change, along with maybe the Moderators?

This is worrisome.

jh385

Amalgam wrote: »

If I'm running a PHPBB Forum on your site, I presume the Admin passwords should get a change, along with maybe the Moderators?

phpBB stores hashed passwords: http://www.phpbb.com/kb/article/difference-between-encryption-and-hashing/ Which means even if someone got hold of your phpBB database, they won't be able to tell what the passwords are.

You'll only have to worry about changing your phpBB admin passwords if you were using the same passwords as your blacknight control panel (and if the blacknight passwords were stored in plaintext, which we're waiting for an answer on)

fenris

Just got the same paypal phishing email. can send it in if you want.

Amalgam

Cheers jh385. I use, different, long and nasty (lovely big chunks of ASCII) passwords for everything, even the mundane stuff..

Control panel changed. Still worried about back up access.

bobbytables

If you have a web app E.g. phpBB hosted, the user accounts of the app are completely different to the user accounts associated with Blacknight (control panels, etc). Many popular web app platforms tend to encrypt user account passwords using various mechanisms, etc. However some dont.

At the end of the day, if unauthorised 3rd parties have control panel access to your hosting account & from there could get copies of DBs associated with hosted web apps then yes you have a bigger problem on your hands. Changing your control panel passwords will not ensure the integrity of your web apps security will be upheld going forward. There are many layers here to consider.

Unfortunately these things happen & its never a matter of IF but WHEN. There is no such thing as 100% security. So it's what happens after the event that makes the difference. Although I am not a customer of Blacknight, but I am confident that they would be diligent enough to apply a reasonable level of security. This does not mean that they can't be hacked or this won't happen again with them or whoever else you consider moving to. The only reason why we're not reading other hosting company press releases is because they were not targeted to the same extent on this occasion. I would not trust competing hosting providers to vary significantly in their security provisions.

Also requesting that a provider make available how exactly they manage security isn't the best idea. If I had malicious intent that information would save me a lot of time. At the end of the day it's a matter of confidence, but none of us are 100% secure and we all play out part in maintaining security.

A crap user password, although encrypted will be exposed pretty quick through rainbow tables, etc.

jh385

bobbytables wrote: »

Also requesting that a provider make available how exactly they manage security isn't the best idea.

They've already been compromised. The attacker may already have the user table in their possession. And if they have, the attacker already knows if the user passwords are hashed or not.

So don't you think we have a right to that information as well? In fact, I think it's critical. This isn't like the boards.ie hack. A lot of us are hosting multiple sites with multiple databases (phpBB etc). How do we know the attacker isn't already in the process of swiping the user tables from our own sites?

So this hack could have a domino effect on us, and a reply from blacknight at this point on the hash/plaintext question is time critical. At least let us make our own minds up on the risk, instead of giving us a vague "change your passwords" email.

bobbytables wrote: »

A crap user password, although encrypted will be exposed pretty quick through rainbow tables, etc.

But the attacker is less likely to spend time brute-forcing them. If he's got his hands on plaintext he's might already have been and gone.

TouchingVirus

Blacknight - what is two factor encryption? And how are you encrypting? Do you mean you're hashing passwords with md5 and then salting them all with a "control password"/secret key/universal app salt?

Again, some more technical detail would be nice so we can make up our own minds on whether to run the risk in changing all our domain passwords etc

Reamer Fanny

TouchingVirus wrote: »

Blacknight - what is two factor encryption? And how are you encrypting? Do you mean you're hashing passwords with md5 and then salting them all with a "control password"/secret key/universal app salt?

Again, some more technical detail would be nice so we can make up our own minds on whether to run the risk in changing all our domain passwords etc

I dont think they could give that information out it might be advantageous to a hacker watching the forum.

bobbytables

jh385 wrote: »

They've already been compromised. The attacker may already have the user table in their possession. And if they have, the attacker already knows if the user passwords are hashed or not.

So don't you think we have a right to that information as well? In fact, I think it's critical. This isn't like the boards.ie hack. A lot of us are hosting multiple sites with multiple databases (phpBB etc). How do we know the attacker isn't already in the process of swiping the user tables from our own sites?

So this hack could have a domino effect on us, and a reply from blacknight at this point on the hash/plaintext question is time critical. At least let us make our own minds up on the risk, instead of giving us a vague "change your passwords" email.


But the attacker is less likely to spend time brute-forcing them. If he's got his hands on plaintext he's might already have been and gone.

That's all legitimate concerns, but I still stand by what I said for the following reasons...

I am aware that the attack is past tense, and existing/recent security provisions in place are known to the attackers, but they are not known to the world of others potential attackers out there that could carry out a separate attack. I am sure you will agree that every one of us will have a notion in our heads of what constitutes satisfactory levels of security. This will indeed vary from person to person and will certainly be influenced by their respective acknowledgement of risk and risk reduction measures.

Asking a home owner whether or not they lock all doors and windows at night and arm the alarm may satisfy one inquiring group at a particular point in time, because they feel confident when weighing up the risk as it's deemed sufficient. Security rarely becomes a concern for people until they fall victim to a successful attack, and then their perception and acknowledgement of risk goes through the roof in an array of disbelief & panic. In reality all they are looking for is that confidence back that they've lost. Most people believe, if they're not a victim, then they're currently immune from attack, and instead it could just be that they're not currently a target.

Every company you use will have vulnerabilities. I am positive if Blacknight came back and said we did X,Y,Z, some people would think "OK", while others would rant and roar saying you eejit, why didn't you do A,B,C. What would that accomplish? Do you think A,B,C would make you completely immune from future attacks. Yes it may have helped in this particular case, but the attack could have taken another form where X,Y,Z may have been a better choice. If attackers want to get in, they'll get in, given enough time and effort.

In situations like this there is a lot more to risk assessment that the technical measures that have been put in place by a single provider at a particular point in time.

Best thing any of us can do is know how we'll respond WHEN stuff like this happens is all I'm saying.

Spacedog

Not impressed by this, no point closing the stable door after the horse has bolted with your customers information on it's back.

Lucky for you the Data Protection Commissioner here is a limp dick form stamper who'll let you off with a slap on the wrist at the worst.

Not so lucky for us customers who have to worry if you are telling the truth about the extent of the breech, or are pulling a damage control exercise at the behest of an overpriced PR crisis consultant.

I blame myself though, for keeping your service after the breech on the Ragnell server a couple of years back.

Thanks for nothing.

SD.

prospect

I too received the Paypal e-mail, and never normally receive spam on that account.


I have changed my password on my control panel as per the advice. I hope you guys get this resolved soon.


Everyone who received the Paypal e-mail, be sure to report it, on GMail there is a "Report Phishing" option.

blatantrereg

johnmurph01 wrote: »

*facepalm*

I got the email this morning.

It never ceases to amaze me how arrogant they can sound when something happens. Same with the time their email system went south a few months back for 48 hours.

Yeah, it takes time to investigate, but whats with the crappy email and tweets. Time for some humility on their part. Stick your hands up and stop dodging.

Where are they arrogant? I only saw the email but it doesn't strike me as arrogant.
They aren't dodging that I can see. I only know about the breach because they publicised it.
Knowing details of how the breach happened would be good wrt assessing how secure they actually are. I would like to know that too. But it would be a bad idea to make it public until they are certain the security hole is completely fixed.

jh385

bobbytables wrote: »

Best thing any of us can do is know how we'll respond WHEN stuff like this happens is all I'm saying.

I'd love to know how to respond, but I still can't make that decision because I don't know the answer to the basic question: Were the passwords in plaintext?

I'm not asking so I can start bitching, I don't have the time or energy for that. And I'm not really interested in the details of the hash (if it is hashed) - But knowing that small piece of information would help me considerably in assessing the impact to me and how I should respond.

BK can respond to me on email if they're worried about potential attackers finding out - I promise I won't tell anyone! :rolleyes:

Random

so i have 3 domains with blacknight. i dont have any hosting.
what risks do i face? i have changed my password.
any other info they would have got is in my whois anyway?

thanks

bobbytables

jh385 wrote: »

I'd love to know how to respond, but I still can't make that decision because I don't know the answer to the basic question: Were the passwords in plaintext?

...

BK can respond to me on email if they're worried about potential attackers finding out - I promise I won't tell anyone! :rolleyes:

I thought BK already said they weren't in plain text, but as I said that doesn't mean much if your password was of poor quality. If I was managing an online forum (which I have several times before) I'd assume the worst, not just in vase, but in general.

Changing passwords to something strong doesn't require a day off work & regardless of acknowledged threat should be done regularly anyway. The regulator will demand certain things are put in place by a provider. If done, & still compromised, slap on the wrist isn't going to happen.

To quote Bruce Schnier, "Security is a process, not a solution". You can't expect anyone to be responsible for completely safe guarding you from threats. To do so would be to undermine the dynamics associated with acquring & maintaing control even in an abstract sense.

Personally I would trust that BK do care about security, are aware of common threats, & do take pre-emotive measures. If you want more security then move away from shared infrastructures & hire a team to work 24x7 on threat assessment, monitoring & response.

With regards the poster with just domains & not hosting. Check all your DNS settings & change your control panel password to something strong. Get on with your life, this will happen you again.

jh385

bobbytables wrote: »

If I was managing an online forum (which I have several times before) I'd assume the worst, not just in vase, but in general.

A lot of BK clients are hosting multiple sites with a multitude of back-end databases, and not just forums.

Okay, let's assume the worst (not my scenario, but quite possible for others) : The password was in plaintext and used across all systems by the admin. The attacker saw the value of this and logged into control panel, and swiped all the back-end databases from the host.

This now means the attacker has not only the BK users table, but all the customers users tables, sales order tables, etc..etc... (This was the domino effect I'm talking about)

So worst case... do I now start emailing *my* clients telling them their data and their customers data may have been compromised? - do we form a queue at the Data Protection Commissioner's office?

bobbytables

jh385 wrote: »

A lot of BK clients are hosting multiple sites with a multitude of back-end databases, and not just forms

Of course, that's why I first started talking about web apps in a generic sense & only referenced forums as an example because it had been previously mentioned.

Also do you think I don't appreciate the potental magnitude of the problem, knock on effects etc?. The point of all my posts is that this stuff happens even when excellent measures are in place.

If I was you I wouldn't even be waiting for an email or PM to take action. Every password changed. Check emaill address references. Lock everything down to the best of your ability. Check code bases, DB content., etc. Lock the doors first, then check & test the integrity of what's inside.

You will not find a provider that will host stuff for you that would not be immune from
all threats. I once shared your mindset, & would have freaked out, but it's not worth it.

MOH

Presumably checking your login history in the CP will show whether there's been any logins to your CP from an unusual IP address (My Account, then Login History at the bottom of the left menu).
If there have been, you might want to WHOIS the IP address and see if it's potentially an issue.

That's assuming they didn't have some alternate method of accessing the CP details, and had no way of deleting the access logs.

bobbytables

There are many ways to approach a break in. If somebody parachutes in an upstairs window, CCTV on the front door will probably give you nothing useful other than the fact that it shows nothing.

When we're talking about web apps probably running way up a stack incl. virtualization, that's a lot of layers of abstraction ultimately beyond your control.

A false sense of security is possibly worse than plain text passwords in a database.

blatantrereg

MOH wrote: »

Presumably checking your login history in the CP will show whether there's been any logins to your CP from an unusual IP address (My Account, then Login History at the bottom of the left menu).
If there have been, you might want to WHOIS the IP address and see if it's potentially an issue.

That's assuming they didn't have some alternate method of accessing the CP details, and had no way of deleting the access logs.

Never noticed that. Nothing unusual in recent history for me. There are a couple of logins from Blacknight offices under my username which coincide with times when I was in touch with support. Didn't realise they did that.

There is one very odd login from Russia listed in November. Any ideas what that might be? I used the automatic setup for Wordpress around that time in case that involves a program logging in?

mneylon

If you notice any strange IP addresses that you did not recognise please contact our support desk directly

A Russian IP address *could* be Parallels technical support staff - if there had been an issue with your account that our team could not resolve immediately we may have escalated it to them.

I don't personally know which IP ranges they use, but I'll ask someone else to check the IP you've posted and confirm

Cathaoirleach

I got that PayPal email twice and another from RadioShack which is odd.