Hosting365 Problem ?

NIBBS

Hi, looks like Hosting365 is down, and there Phone lines appear to be down to - anyone know anything about this ???

cheers

liverpool

Same prob here and my server is down!!
Have sent e-mail

NIBBS

cheers liverpool,

at least I'm not going mad.......hopefully it's just an outage, though I thought they'd foolproofed the system.....

onechewy

I've managed to get through to one of the lines and I'm holding - will update you as soon as I know what the story is!

dil999

whats their phone numbers

onechewy

Power outage - they expect to be back online in five or ten minutes.. soo, it'll probably be a while yet before your back online.

onechewy

1800 365 965
4673600
4673640 - NOC (Got through to this one, but pressed redial about 50 times)

dil999

cheers

NIBBS

thanks for the update,

Steve

dereko

agh!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

monkeyfudge

http://www.hosting365status.com/

Power has been restored.. .so hopefully things will be back up and running again soon.

onechewy

Cheers monkeyfudge, I'd been trying to remember what that status link was!

liverpool

H365 website still down as are phones and my server

Lemo

Power is back now for around 30 minutes but I'm still not able to get further than one of the edge routers (border2.hosting365.ie). Anyone else having more success?

zzap64

Still dead as a doornail for me and all our client webs sites on our reseller account :-o

the main hosting365.ie site is still down too.

You'd think they'd have some backup power systems....

Davy

they are meant to able for outages. they are feed from an esb main sub and they have their own gene outside. thought this was in the name, '365' always on everyday

Hal1

Will this be sorted by monday?

Davy

zzap64 wrote:

Still dead as a doornail for me and all our client webs sites on our reseller account :-o

the main hosting365.ie site is still down too.

You'd think they'd have some backup power systems....

they do. there prob not used to using them enough tho clearly

steve-hosting36

Full details are published at www.hosting365status.com

Of course we have redundancy, the problem in this case was a critical failure in one of the redundant systems.

misterq

for the second time in what, 6 months Steve? That means you don't have redundant systems in my book.

Random

This effected all H365 customers Steve?

steve-hosting36

misterq wrote:

for the second time in what, 6 months Steve? That means you don't have redundant systems in my book.

Our last outage was on 6th July 2006 - nearly a year ago - and was caused by a failure in the generator circuit, which we entirely replaced with a new MV board and substation.

Sparks

That's two major outages in one year steve. Saying there's no problem isn't really realistic - nor does it tally with what the rest of your company's saying.

Frankly, the 50 minutes downtime caused by a whoopsie during maintenance is understandable - not acceptable since it was presumably preplanned maintenance, but understandable as human error does occasionally just happen to even the best of us - but the way that the other problems suddenly cropped up like mushrooms is what gets me. Your DNS has no off-site backup, so that's down for 50 minutes, then it doesn't reboot and takes hours to get back up and working again. Email servers are clogged up for a day or so. And there are all manner of other fun issues caused by a restart. Those are all things that shouldn't have happened. Frankly, it looks like you bought in generators after the last incident and figured that having thrown that much money at the problem that it could never happen again and therefore there was no point looking at the power loss part of the DR plan again. When was the last time you rand a power outage recovery drill before this outage? And when's the next one scheduled for?

steve-hosting36

Hi Sparks,

You'll find full details at www.hosting365status.com

The Generator, UPS and systems were all in place from when we moved in two years ago. We've spent a lot of money since - but systems do fail and usually in new and wierd ways. I would invite you to personally come in for a tour and see the infrastructure and team who look after it with your own eyes.

ntlbell

steve-hosting36 wrote:

Hi Sparks,

You'll find full details at www.hosting365status.com

The Generator, UPS and systems were all in place from when we moved in two years ago. We've spent a lot of money since - but systems do fail and usually in new and wierd ways. I would invite you to personally come in for a tour and see the infrastructure and team who look after it with your own eyes.

Steve,

You're like a broken record every second post your inviting people to look at your "Data Centre" which is a pointless exercise you've been using this phrase for a long time now.

If I walked around and took the tour on Thursday 14th June, Would I have spotted that your DR plans were going to fail miserably the following morning? unlikley.

It's a pointless exercise, show me years of expirience in running a DC show me year after year how on the rare ocassions you have had problems your DR plan has worked effectivley and no money was lost by your clients.

Show me a DC that's open two or three years and in the last two years have had problems with power the most recent losing the vast majority of your clients a days work and I laugh and walk away.

IT descision makers who choose to stay with h365 after these events seriously need to ask themselves a few questions.

You get what you pay for I suppose.

Sparks

Steve, there's no details on the status blog about the next planned power outage drill or the last one. Or any other DR plan drills. And thank you for the offer, but I've been in the data centre recently and I don't see how seeing it again could help. I know you have backup generators and batteries for the UPS system; that wasn't the problem. The problem - according to the status blog - was that you had a single point of failure in the output route from the UPS, it had little to do with the inputs to the UPS systems.

steve-hosting36

Show me any host that has DR drills on their status blogs? All hosts have outages, of varying levels, and , from our customer feedback we think we handled recovery well. Could it have been better - absolutely. Can any host deliver 100% - absolutely not.

Our operations and facilities team have extensive datacentre experience, our Ops Manager ran 7 datacentres and 120 engineers for the UK's largest facilities provider.

Fact of the matter is all systems fail at some point. I'm happy to meet personally with anyone who would like that tour and discussion about our plans and redundancy, but boards is hardly the place for that chat.

Sparks

steve-hosting36 wrote:

Show me any host that has DR drills on their status blogs?

From Tom Raferty's blog on this point, in response to a comment from Ed Byrne:

Ed, an outage like this couldn’t have happened in the DEG. They have had 100% power uptime since they opened their doors in early 2001. They run monthly power outtage tests to ensure that (for instance) no feed breakers trip! The tests are logged and available to clients. Hence my saying it looks like it was a bad move.

And that's just the most recent, most relevant example from the top of my head.

All hosts have outages, of varying levels, and , from our customer feedback we think we handled recovery well.

Are you reading the same posts I am?
I mean, seriously, should I cut and paste some of the really rather annoyed posts from there or Raferty's blog or any of the three or four other places on the net where H365 is now being rather loudly savaged?

Could it have been better - absolutely. Can any host deliver 100% - absolutely not.

Well, let me put it this way - you're mathematically correct, since at some point the sun will swell to a red giant and engulf the planet, vapourising all the data centres.

In practical terms though, five nines is a pretty standard contract requirement (H365 is basicly at what now, 3 nines? If that?) and higher levels are available; but those are minimum levels of downtime. With guarantees and hefty penalty clauses. Many US and EU data centres are running at 100% and have been for some years now. And that's your competition.

Our operations and facilities team have extensive datacentre experience, our Ops Manager ran 7 datacentres and 120 engineers for the UK's largest facilities provider.

So how come he didn't have a backup DNS server off-site and how come it took so long for the DNS servers to be rebooted correctly? I'm uninterested in his CV, I'm interested in why critical servers hadn't been tested under power failure conditions.

Fact of the matter is all systems fail at some point. I'm happy to meet personally with anyone who would like that tour and discussion about our plans and redundancy, but boards is hardly the place for that chat.

Indeed. Well, I've seen the data centre, I've read the incident reports, and frankly - especially given your attitude of "stuff happens" as a reason why a hiccup on the ESB line can kill your entire centre's power for 50 minutes and your services for days - I believe I'll just recommend other hosting providers in future when asked professionally to do so.

steve-hosting36

With all due respect Sparks, I think you're misrepresenting us a little. We've delivered 99.99984% power uptime in the last 12 months. Compared to other small to medium sized hosting companies running their own datacentres and infrastructure, that's a pretty good record.

All customers bar basic shared hosting have enforceable contracts with SLA's and compensation clauses, as industry standard.

Boards is hardly a place to 'justify' ourselves, and while there is understandable frustration, if you would like to talk about your experience in more detail, I would invite you to come in in person and do so.

Random

Do all these DNS issues effect people who have domains with r365 which point to nameservers on another host?

TheDriver

Due to the unique multi-homed nature of our network infrastructure, we can provide 100% uptime service guarantees by meshing multiple carriers and peering partners. This provides a level of redundant connectivity unparalleled in the Irish market.

Maybe this should be changed to 99.9% uptime............

The important thing for me, and I have a number of shared hosting a/c with 365 is that the problem was indeed unforeseen and they have put in place tangible changes to ensure it doesn't occur again.

steve-hosting36

ciaranfo wrote:

Do all these DNS issues effect people who have domains with r365 which point to nameservers on another host?

No, such customers would not have seen an impact at all.

steve-hosting36

TheDriver wrote:

The important thing for me, and I have a number of shared hosting a/c with 365 is that the problem was indeed unforeseen and they have put in place tangible changes to ensure it doesn't occur again.

And we will be publishing that full plan to customers asap.

Sparks

steve-hosting36 wrote:

With all due respect Sparks, I think you're misrepresenting us a little.

That's not my intention Steve, but you have to understand that what you're saying here is not in agreement with either what we saw on Friday or what your company has since stated on its status blog.

We've delivered 99.99984% power uptime in the last 12 months.

Which is utterly irrelevant. As I said above, you lost power for 50 minutes, but you lost service for far longer because your centre didn't recover from a power failure scenario correctly. Ensuring that the centre recovers cleanly from a disaster like that is why you pay a hosting provider rather than just keeping your servers in the office.

steve-hosting36

Hi Sparks,

Broadly speaking I agree with you but we have been consistent (and extremely open) with what we have been saying. Any minor inconsistencies are down to 'point in time' information gaps.


Our recovery this time wasn't as clean as we would have liked, and has highlighted some weaknesses in our shared hosting cluster and how it is architected, which we are working to rectify.

There are no excuses for the failures - we did some things well, we did some things badly and we learned a bit more. We are fairly unique in this country in that we are the only hosting provider (ISP's excepted) that run our own facilities and infrastructure, so our problems are always going to be 'bigger' than anyone elses.

Sponge Bob

steve-hosting36 wrote:

We are fairly unique in this country in that we are the only hosting provider (ISP's excepted) that run our own facilities and infrastructure, so our problems are always going to be 'bigger' than anyone elses.

There is Blacknight and there is the CIX too. Of course you are still "fairly unique" as you put it but you are not the only one all the same

I find that claims of 100% uptime by any hosting provider..bar the desperate... do not survive the SLA guarantee negotiations. Once you mention "compo" then the .999% figures get a mention

365 have good uptimes by industry standards but none of them have 100% uptimes as weird **** does indeed happen .

The aircon in anhy hosting facility is a single point of failure as well, is it not Steve. In exceptionally hot weather it can cascade out unless 100s of Kw of extra cooling are added.

steve-hosting36

HI Sponge Bob,

Well, Blacknight dont run their own datacentre and CIX (while not open for business yet) is a datacentre selling colo (as opposed to a hosting provider).

No one claims 100% uptime - our SLA's cover a 100% guarantee, which we back up with cash compensation if we don't hit it.

Aircon isnt really a single point of failure as long as you have enough units and sufficient 'head room' for really hot weather spikes.