Major Outage at Hosting365

the_batman

There was a major outage at hosting 365 today. We lost access to our deidcated servers at a little before 13:15. For the next 30 minutes, all site using hosting 365, including boards.ie and www.hosting365.ie, were unavailable.

The sites came back at about 13:45, but there is no comment on the hosting 365 site as yet.

When the servers came back, we noticed that our dedicated servers had been rebooted, so it looks to me like there was a power outage.

Santa Claus

Surely it couldn't have been a power outage, they have redundant systems and generators on site don't they ?

eoge

Yeah. It got me bad too.

Apparently there was a fire at their data centre.

eoge

Anyone know of an SMTP server that'll let me send mail through it? I still can't use mail.hosting365.ie.

the_batman

I tried to log on to their support forum, but as you can see from the attached image, it is not available.

There is still no update on their site. I can understand them having a fire and this causing them an outage, but surely they should put something up on their site to let their 50,000 + customers know.

mneylon

eoge wrote:

Anyone know of an SMTP server that'll let me send mail through it? I still can't use mail.hosting365.ie.

Your ISP should allow you to relay

Sponge Bob

and did the sprinklers come on <cough>

the_batman

Sponge Bob wrote:

and did the sprinklers come on <cough>

I think they have a halon system there.

Sponge Bob

the_batman wrote:

I think they have a halon system there.

How many were choked to death ???

flamegrill

apparently an esb substation blew and took out power in park west ... 365 should have been unaffected by that.

Paul

DOLEMAN

In fairness they are probably still trying to fix everything and will make an announcement later.

steve-hosting36

An official announcement will be posted to our forums in the next few minutes. Apologies to those customers affected, we suffered from an unfortunate confluence of events. All systems are 100%.

the_batman

Sponge Bob wrote:

How many were choked to death ???

I used to work in a large datacentre with the halon system. I was always terrified that it would go off when I was in the room.

I was told that there was a 20 second delay between the alarm going off and the system initiating. I still never felt comfortable in that room!

smarty

steve-hosting36 wrote:

An official announcement will be posted to our forums in the next few minutes. Apologies to those customers affected, we suffered from an unfortunate confluence of events. All systems are 100%.

I have to question this 100% comment, because I can see a couple of shared hosting sites that have not come back since the outage. PM me if you want details.

clarkee

Any further updates from anybody?

smarty

clarkee wrote:

Any further updates from anybody?

I have logged onto the Hosting 365 forums, but there doesn't appear to be any udpate. The hosting365 blog hasn't been updated either.

I can understand that they are trying to get the service back to normal, but it is a pretty poor effort that the only update I have seen from hosting 365 was a post on boards.ie promising an official announcement in a few minutes which didn't arrive.

Cloud

I can confirm our machines were rebooted, sorry for the temporary lack of service.

DJB

steve-hosting36 wrote:

An official announcement will be posted to our forums in the next few minutes. Apologies to those customers affected, we suffered from an unfortunate confluence of events. All systems are 100%.

1 years free hosting for everyone, Steve?????

smarty

Those sites I mentioned about came back online in the last five minutes.

smarty

DJB wrote:

1 years free hosting for everyone, Steve?????

*edit from mod
Keep your friend's problems with H365 between him and H365

eoge

smarty wrote:

...it is a pretty poor effort that the only update I have seen from hosting 365 was a post on boards.ie promising an official announcement in a few minutes which didn't arrive.

Give them a chance. AFAIK this has never happened before (in the many years they've been going) so I think they deserve the benefit of the doubt here. Anyway, we're back up and running and nobody got hurt. It is funny how much the Irish web scene relies on them though!

Ph3n0m

smarty wrote:

...it is a pretty poor effort that the only update I have seen from hosting 365 was a post on boards.ie promising an official announcement in a few minutes which didn't arrive.

Why is it a poor effort - this is not the h365 support forums nor is it their helpline.

If people want to start bitching about h365 do it on their forums, not here. Atleast that way you are bitching to the people who care

anthonymcg

Ph3n0m wrote:

Why is it a poor effort - this is not the h365 support forums nor is it their helpline.

If people want to start bitching about h365 do it on their forums, not here. Atleast that way you are bitching to the people who care

I'm presuming he thinks it's a poor effort because there was no similiar announcement made on their forums afaik.

cgarvey

perhaps it's a poor effort because customers should have been emailed / updated, before boards.ie users?

Stephen posted here before I got a response to my support ticket. Still no sign of the promised official announcement on their forums either.

.cg

DOLEMAN

Is everything defo fixed? ALL my sites are still down.

Anyone else having this problem?

smarty

DOLEMAN wrote:

Is everything defo fixed? A few of my sites are still down.

Anyone else having this problem?

I am aware of a few sites that were offline until about 16:45. They were on shared hosting. Maybe they required some manual intervention after the first problem.

DOLEMAN

I am not shared hosting. I have a dedicated server... Try www.dole.ie

Cake Fiend

cgarvey wrote:

perhaps it's a poor effort because customers should have been emailed / updated, before boards.ie users?

A significant number of the readers of this forum are H365 customers. Pretty quick way to reach the masses IMO.

steve-hosting36

http://forums.hosting365.ie/phpBB2/viewtopic.php?t=952

Apologies for the delay, the priority has been on restoring service to any effected customers.

smarty

DOLEMAN wrote:

I am not shared hosting. I have a dedicated server... Try www.dole.ie

Your issue is a DNS issue. I tried to ping www.dole.ie and it would not resolve.

smarty

Sico wrote:

A significant number of the readers of this forum are H365 customers. Pretty quick way to reach the masses IMO.

The quickest way to reach the masses would be to put a message on the Hosting365 site or hosting365 forum.

That is the first place that I went. Then I came to boards.ie.

DOLEMAN

smarty wrote:

Your issue is a DNS issue. I tried to ping www.dole.ie and it would not resolve.

Well, it's all via hosting365... Hopefully it's fixed soon. I'm sure they're trying their best.

Sparky

Yeah my site went offline at lunch time.
Im amazed I only got one database error email.

smarty

DOLEMAN wrote:

Well, it's all via hosting365... Hopefully it's fixed soon. I'm sure they're trying their best.

Your name servers are configured as ns1.hostago.com. This server is hosted by hosting365. This server is not processing DNS request. It might be an idea to send an email to support@hostago.com.

DOLEMAN

smarty wrote:

Your name servers are configured as ns1.hostago.com. This server is hosted by hosting365. This server is not processing DNS request. It might be an idea to send an email to support@hostago.com.

Yes, as I said earlier, my server is down.

I have contacted H365 already...

smarty

DOLEMAN wrote:

Yes, as I said earlier, my server is down.

I have contacted H365 already...

Your server may not be down. The domain name www.dole.ie is not being resolved to an IP address. Take a look at the attached image. When I ping www.hosting365.ie it resolves the name to the IP Address 82.195.144.16.

When I attempt to Ping www.dole.ie, it says that it could not find the host www.dole.ie. This does not mean your server is down. It means that there is a DNS issue.

This doesn't tell us whether your server is up or down, but it does tell us that you have another issue.

DOLEMAN

Well, my server is also my DNS server, so my server is down... or they've broken my DNS thingy. Either way, that pesky fire did something!

Cake Fiend

smarty wrote:

The quickest way to reach the masses would be to put a message on the Hosting365 site or hosting365 forum.

And a quick way to reach customers who mightn't know that their site is even down yet would be to post on one of Ireland's most popular discussion forums.

DOLEMAN

Sico wrote:

And a quick way to reach customers who mightn't know that their site is even down yet would be to post on one of Ireland's most popular discussion forums.

Most people in Ireland have not heard of Boards.ie. Only a small group of nerds know about it. Don't believe me? Ask random people do they use boards.ie. Pretty much all will say, "What is boards.ie?"

Normal customer service would be to let your customers know there is a problem (post on H365 website) rather than answering questions on a discussion forum.

However, I believe boards.ie is owned by someone who works for H365, and I understand their image is important, so posting here is not a terrible thing to do. They should have announced on their website first though.

Sparky

You must be talking about Regi.

Does he own boards?

mneylon

Sparky-s wrote:

You must be talking about Regi.

Regi no longer works for 365. He moved to Digiweb a couple of months ago

DOLEMAN

Anyone elses server still down?

DOLEMAN

Ah this is ridiculous. The explanation in their forum is suggesting they just lost power and everything was sorted in less than an hour, but that's obviously not the case.

My server is still down, and they're not answering my support calls (3 calls, 2 voice messages) or e-mails (2 e-mails)...

I can't access their support forum without an administrator first manually verifying my account...

Come on H365, at least let me know what's going on!

DOLEMAN

Just got an update from H365.

I love boards.ie!

misterq

Doleman,

Would I be right in thinking your issue related to Bind (your DNS server) not starting up again after the server came back online?

That was the only major issue I had with the outage, a server I look after came back up but MySQL was still down.

If so be sure that bind is now added to programs that start automatically on boot up so it doesn't crop up again.

DOLEMAN

Well, the H365 guy did say my server was down until he rang me (9.20ish), so I'm assuming it was totally dead. However I will check to see if bind is on autostart, just in case...

Cheers!

aphex™

It should be Hosting364 now then I guess

Doleman: should you not try and use some other DNS service too, just in case? Even 364's one? Better yet someone elses?

DOLEMAN

Yeah, I know. The reason for my original DNS setup is complicated (was required at the time), but another fallback would be the clever thing to do! Indeed...

dahamsta

aphex&#8482 wrote:

It should be Hosting364 now then I guess

Cruel but funny.

eoge

aphex&#8482 wrote:

It should be Hosting364 now then I guess

Love it! Haha.

cgarvey

Sico wrote:

A significant number of the readers of this forum are H365 customers. Pretty quick way to reach the masses IMO.

Their own forums, their own website, their own ticket support system, their customer email database? I can understand a delay, while they wipe the brows stuff off the fan blades. I can't understand why they wouldn't let their customers know (not just a tiny subset) first.

.cg