Lots of The Irish Internet Gets Clattered For An Hour

Sponge Bob

The cause was a fire in Parkwest or Citywest apparently . Only the legendary beecher.net in Cork survived the carnage .

smarty

Sponge Bob wrote:

The cause was a fire in Parkwest or Citywest apparently . Only the legendary beecher.net in Cork survived the carnage .

From what I could see it only affected sites hosted on hosting365. We have dedicated servers there and we lost them for just over 30 minutes. Site hosted elsewhere appeared to be unaffected.

Blowfish

Sponge Bob wrote:

The cause was a fire in Parkwest or Citywest apparently . Only the legendary beecher.net in Cork survived the carnage .

heanet was grand aswell, I was wondering why I couldn't get on any Irish sites though.

smarty

Sponge Bob wrote:

Only the legendary beecher.net in Cork survived the carnage .

Beecher.net is hosted in the states, so it would not have been affected.

smarty

Blowfish wrote:

heanet was grand aswell, I was wondering why I couldn't get on any Irish sites though.

I was able to get onto sites hosted by blacknight, and the RTE Site was grand too.

Blaster99

What happened to the Hosting365 redundant tier 1 connections...

Blowfish

http://www.boards.ie/vbulletin/showthread.php?t=2054956771

mike65

beecher.net would survive the appocalypse, cockroach-like!

Mike.

the_batman

Blaster99 wrote:

What happened to the Hosting365 redundant tier 1 connections...

It looks like they turned off the power to the whole building.

aidan_walsh

Ah.

In light of this news, Cloud I take it all back. Sorry.

Sponge Bob

mike65 wrote:

beecher.net would survive the appocalypse, cockroach-like!

feral survival skills seeing as half the country is waiting to give that fella the <snip> .

dahamsta

smarty wrote:

Beecher.net is hosted in the states, so it would not have been affected.

Ah but he keeps his customers closer to home.

<snip snip>

adam

Sponge Bob

That IS actually in Cork is it not so it will take 2 x <snip> to Behead the beast.

Blaster99

the_batman wrote:

It looks like they turned off the power to the whole building.

I guess that's the end of the 100% power SLA then.

dahamsta

To be fair, an SLA isn't an uptime guarantee, it's an agreement that if downtime is experienced, the customer will be compensated. 100% uptime just doesn't happen without massive investment -- multiple machines, multiple ups', multiple generators, multiple geographic locations, etc, etc.

I'd be curious why the redundancy systems (appear to have) failed in this case though, it seems odd.

adam

TimTim

Might explain the little drop in Inex traffic today

https://www.inex.ie/technical/stats

Sponge Bob

Must get onto the beecher.net NOC and yerra shure see what the latest Boilerplate BS answer is down in Cork if you ask this kinda question here

http://forums.hosting365.ie/phpBB2/viewtopic.php?t=869

flamegrill

TimTim wrote:

Might explain the little drop in Inex traffic today

https://www.inex.ie/technical/stats

I would attest that to ftp.heanet.ie being offline for quite a bit of the day as it is being moved to its new home in servecentric

Paul

dahamsta

Sponge Bob wrote:

Must get onto the beecher.net NOC and yerra shure see what the latest Boilerplate BS answer is down in Cork if you ask this kinda question here

I've already answered that question on Open Bob:

me wrote:

My machines run on air, and thus are invincible to power cuts. Green too!

TimTim

flamegrill wrote:

I would attest that to ftp.heanet.ie being offline for quite a bit of the day as it is being moved to its new home in servecentric

Paul

I got a ticket about server moves for a totally unreleated reason, didn't put 2 and 2 together.

mneylon

smarty wrote:

I was able to get onto sites hosted by blacknight, and the RTE Site was grand too.

Of course you were

Our network wouldn't be affected by an ESB power outage

dahamsta

Update from H365 here. A bit late in coming and the outage really should haven't happened in the first place, but sometimes the gods work against you. No doubt the lads in 365 will learn from it.

adam

jwt

Things like making sure your dual PSU servers are fed from seperate PDUs which in turn are fed from seperate rings/phases.

Sounds blindingly obvious but you'd be suprised how easy it is to forget which PDU is which when racking a server.

I suspect most techies will look at this as a case of "There but for the grace of God go I" and promptly check their own cabling very discreetly.


John

dahamsta

Backups are the same. As soon as everything is back up and running, you walk away to catch your breath. Then a few days later you think to yourself: Arse, did I ever restart the backup processes? So easy to do.

adam

damien

blacknight wrote:

Of course you were

Our network wouldn't be affected by an ESB power outage

Then what causes the outages you have?

shamage

jwt wrote:

are fed from seperate rings/phases.

John

Just goes to show that a little knowledge can be a dangerous thing.

You should be looking for the same phase from different supplys. If you have a dual PSU server connected to two different phases EG L1 and L3 you would have up to a 400V potential difference between live on both of the PSU’s. All you need is to loose the netural on one of the phases and BANG there goes your server. You will also loose neutral easier that you think, lets assume you are removing ome of the supply leads and rock it from left to right when removing there is a 50% change the neutral will disconnect before the live.

jwt

Just goes to show that a little knowledge can be a dangerous thing

Yes you are right, and thank you for proving it. :rolleyes:



John

EDIT I was going to leave it at this but seeing as I'm feeling peeved

http://www.cisco.com/en/US/products/hw/switches/ps708/products_installation_guide_chapter09186a008020e0d3.html#wp1026596

All AC power supply inputs are fully isolated.
-Source AC can be out of phase between multiple power supplies in the
same chassis, which means that PS1 can be operating from phase A and
PS2 can be operating from phase B.
-Source AC can be out of phase between AC inputs on power supplies
that are equipped with multiple AC inputs, which means that power cord
1 can be plugged into phase A and power cord 2 can be plugged into
phase B.

HP, Dell etc all do the same and isolate their supply inputs. I can dig out the technical docs for you if you want :eek:

Try googling switch mode power supplies, subtle difference from transformer based power supplies. It'll get you pointed in the right direction, you might even figure out why they are isolated.

cgarvey

I'd be more worried that it took 27 minutes to find the circuit breaker, let alone the clearly flawed planning of the electrical systems!

.cg

mneylon

damien.m wrote:

Then what causes the outages you have?

We've never had a power outage and as long as we use proper data centres we never will

Sponge Bob

blacknight wrote:

We've never had a power outage and as long as we use proper data centres we never will

Which means you have considered using IMPROPER Data Centres at some stage

You better hope Blacknight NEVER has a power outage because that comment there will come back to haunt you ....I assure you

oscarBravo

shamage wrote:

Just goes to show that a little knowledge can be a dangerous thing.

You should be looking for the same phase from different supplys. If you have a dual PSU server connected to two different phases EG L1 and L3 you would have up to a 400V potential difference between live on both of the PSU’s. All you need is to loose the netural on one of the phases and BANG there goes your server. You will also loose neutral easier that you think, lets assume you are removing ome of the supply leads and rock it from left to right when removing there is a 50% change the neutral will disconnect before the live.

Um. Is this from experience, or hypothetical? How, exactly, does 400V potential difference between the live feeds (to two completely separate power supplies) translate to BANG?

mneylon

Sponge Bob wrote:

Which means you have considered using IMPROPER Data Centres at some stage

Hehe - of course. I consider a lot of things..

Sponge Bob wrote:

You better hope Blacknight NEVER has a power outage because that comment there will come back to haunt you ....I assure you

That's fine. Have a look at:

http://www.dataelectronics.ie/power.html

Sponge Bob

blacknight wrote:

That's fine. Have a look at:
http://www.dataelectronics.ie/power.html

Nope.It will come back to Haunt YOU not Dataelectronics , they have not asserted anything in this thread ! Believe me I have seen many outages in 24/7 buildings with power stations attached and for the strangest of reasons. One spider nesting in the switchgear and p00f .

You never can tell. Therefore you should be happy it was not Blacknight this time.

And yes, I am trying to haunt beecher.net as well but the fecker hired an exorcist an put him on my ass.

Sparky

LOL @ blacknight and Sponge.

dahamsta

Sponge Bob wrote:

And yes, I am trying to haunt beecher.net as well but the fecker hired an exorcist an put him on my ass.

Blacknight likes his carrots, I like my stick.

Sponge Bob

Ken Shabby wrote:

I like my stick.

Call him off please, he wants to shave my back . I'll be nice to everyone for the rest of the week

MiCr0

and thats enough - locked