What is wrong with the site? (Please read updated post #1)

Dav

Hello everyone.

I'm back form 2 week's holidays today, only to arrive in and find that an ESB substation has popped somewhere around the South William St area and so we've been sitting here with no internet connection in the office for nearly 4 hours. When it rains, it pours, eh?

OK, so the site has been, to use the technical term, bolloxed. I don't fully know why, but we are obviously investigating as I type. Preliminary investigations suggest that one of our database servers went kaput (we have 3) and then a DDOS attack took a second one out of sync which is why you see notification errors, 404's when you post and basically, most of the problems that've been reported over the last 5 days.

It is obviously our number 1 priority to get all 3 database servers back up and running at full speed and in full synchronisation this afternoon so that the vast bulk of these problems disappear.

I will keep you all posted via this thread as the day goes on - I have a couple of weeks worth of stuff to catch up on, so please bear with me.

Finally, I'm really sorry the site's been such a pain in the arse to use this last week or so, I've been just as frustrated as the rest of you because I've been trying to use it too on my time off. We've dropped the ball and that's not good enough and not how we do things, so again, I'm sorry, but we're trying to put it right as we speak.

---

*Update - November 2016*

(Boards.ie: Mark) Been talking with the tech guys about how errors are reported, and it would help everyone involved (yourselves included as it could help us get to the root of issues) if you could do the following.

Before submitting a bug report, please take the following steps to attempt to resolve it:

• Clear cache and cookies
• Restart your browser
• Try a different browser
• Try incognito mode
• Restart your machine
• Try a different computer
• Helpful link: https://support.mozilla.org/en-US/kb/websites-dont-load-troubleshoot-and-fix-errors

Should that not work, please include as much information as possible, such as:

• The URL of the page you were on when the error occurred
• Steps to reproduce the process
• Screenshots
• What you expected to happen
• What actually happened
• The results of the Diagnostics page - please delete the IP address line
• Whether you are using the Legacy or Responsive site
• Whether you are based in Ireland or overseas
• Whether you are in an office / operating behind a proxy/firewall

Thanks all!


NOTICE (02/03/17):
Hi everyone,

Some work is going to be carried out today in the datacentre, which MAY cause some disruption (though hopefully not). Sorry for any inconvenience caused.

Darkglasses

Thanks Dav, good luck with the fix.

martyos121

Won't somebody please think of the hamsters! They must be knackered, they need a holiday!

Orion

What I take from this thread ... Dav gets holidays?

libelula

I can only imagine the head on Dav coming back from holidays to this. He'll be all over the R&R forum later

jonski

I was coming on to have a whinge about there being no central thread with a notice in the first post from admin telling us what the problem was . Now I realise there is but I couldn't see it due to the problems with the site . Now I'm whinging at myself for not thinking about that ....but none of this matters because this might not even post and even if it does no one will probably see it, at least until the site is fixed at which stage no one will care about this thread any more ........... waffle waffle waffle .

my3cents

Just wondering if anyone has officially said anything to the companies and reps that reply in the Talk to forums? They are afaik paying for a service and might be more in the dark than those of us that bother to post about site issues?

MJohnston

I'm surprised nothing has been posted to the Boards.ie Twitter or Facebook accounts - even Facebook uses Twitter for downtime updates!

Synode

MJohnston wrote: »

I'm surprised nothing has been posted to the Boards.ie Twitter or Facebook accounts - even Facebook uses Twitter for downtime updates!

I contacted Boards on Twitter and they never responded. They're normally quite quick so not sure why they didn't this time

libelula

I can only imagine the head on Dav coming back from holidays to this. He'll be all over the R&R forum later

Is there anything here?

Just to add - I followed the link from Dav's post, but there's absolutely nothing here. No posts. Nothing.

Myrddin

Refresh the page

jimgoose

Relational databases are like two-stroke engines. Mighty yokes while they keep zinging along, but now and then the Gods of Ballsup notice you and send Hell and Perdition. It is at this point that the administrator realises what a perversion of physics the thing is and how amazing it is that it works at all. The divil is stuck in 'em - the divil, I tells ye!!

Skylinehead

jimgoose wrote: »

Relational databases are like two-stroke engines. Mighty yokes while they keep zinging along, but now and then the Gods of Ballsup notice you and send Hell and Perdition. It is at this point that the administrator realises what a perversion of physics the thing is and how amazing it is that it works at all. The divil is stuck in 'em - the divil, I tells ye!!

Never change, Jim :cool:

CatInABox

Myrddin wrote: »

Refresh the page

You're assuming that he can see your post?

I've actually start to find the whole situation hilarious, I'm starting to feel like I've got Alzheimers. I go into a forum, see that there's no new posts and think to myself "Wait, why did I come in here?"

Ruu

Thanks for the update.

Myrddin

CatInABox wrote: »

You're assuming that he can see your post?

It's happening me too, I click the thread link from the forum & get taken to an empty boards page with no post/details in it. A refresh fixes it, mostly

I've actually start to find the whole situation hilarious, I'm starting to feel like I've got Alzheimers. I go into a forum, see that there's no new posts and think to myself "Wait, why did I come in here?"

Lol, I know the feeling

Myrddin

Seems a lot better now...

Tompatrick

martyos121 wrote: »

Won't somebody please think of the hamsters! They must be knackered, they need a holiday!

Oh Lollers ! What an original joke....

martyos121

Much improved now, still one or two problems but nowhere near as bad as it was an hour or two ago or last night.

SEPT 23 1989

working well now thanks

Oryx

Most people go on holidays from work and know, regardless of how critical their role, the place won't fall down without them.

But not in Dav's case. Hes not allowed to leave again until they clone him.

Dav

OK, so I'm told that the databases have been brought back into synchronisation and that this should have resolved a lot of the problems we've been having.

We're obviously monitoring carefully to see that everything is stable, but with a bit of luck, we're past the worst of it.

Oryx

Just a fyi, still getting logged out on touch.

Irish Steve

Now comes the inquest into Who, how, why and identifying the innocent and guilty parties, which could be "interesting".

It will be nice to be able to rely on the page showing what's actually happening again, if I'd read one particular message once more, I think there would have been an imprint of a laptop on the wall the other side of the room.

thesandeman

Seems ok now.
Thanks for the update but one single announcement on Twitter from whoever was minding the shop would have stopped a lot of confusion.

Hollister11

It's working a lot better now. I still get the odd '404' when trying to access treads.

Gordon

thesandeman wrote: »

Seems ok now.
Thanks for the update but one single announcement on Twitter from whoever was minding the shop would have stopped a lot of confusion.

This wouldn't have helped I imagine:

Dav wrote: »

we've been sitting here with no internet connection in the office for nearly 4 hours.

jimgoose

Skylinehead wrote: »

Never change, Jim :cool:

Remember this: It ain't over until the bastard thing eats another set of pistons. And even then it ain't over - the only end to this living Hell is divorce, poverty and rocking back-and-forth in a foetal position gibbering semi-coherently about port heights and premix ratios.

GLaDOS

My avatar isn't loading in the floating menu, tear the whole thing down.

jimgoose

ScienceNerd wrote: »

My avatar isn't loading in the floating menu, tear the whole thing down.

Yes, my hovercraft is also full of eels. We should, like, set fire to shít an' stuff.

foggy_lad

I've sent a tazer to boards HQ to provide a little encouragement to the hamsters but it may not arrive due to the post being bollixed too.

jack presley

Dav,

http://touch.boards.ie/thread/2057501964/1/#post97232953

This issue is still happening.

thesandeman

Gordon wrote: »

This wouldn't have helped I imagine:

They could have walked outside and tweeted it from their phone.

Insect Overlord

thesandeman wrote: »

They could have walked outside and tweeted it from their phone.

But then they wouldn't have been working on fixing the problem. :P

naughto

Oryx wrote: »

Just a fyi, still getting logged out on touch.

Ya same here and on the phone

Gordon

thesandeman wrote: »

They could have walked outside and tweeted it from their phone.

They could have been trying to fix the site, fix their internet, not been able to log in to company twitter account due to passwords/policy etc.

circadian

Dav wrote: »

Hello everyone.

I'm back form 2 week's holidays today, only to arrive in and find that an ESB substation has popped somewhere around the South William St area and so we've been sitting here with no internet connection in the office for nearly 4 hours. When it rains, it pours, eh?

OK, so the site has been, to use the technical term, bolloxed. I don't fully know why, but we are obviously investigating as I type. Preliminary investigations suggest that one of our database servers went kaput (we have 3) and then a DDOS attack took a second one out of sync which is why you see notification errors, 404's when you post and basically, most of the problems that've been reported over the last 5 days.

It is obviously our number 1 priority to get all 3 database servers back up and running at full speed and in full synchronisation this afternoon so that the vast bulk of these problems disappear.

I will keep you all posted via this thread as the day goes on - I have a couple of weeks worth of stuff to catch up on, so please bear with me.

Finally, I'm really sorry the site's been such a pain in the arse to use this last week or so, I've been just as frustrated as the rest of you because I've been trying to use it too on my time off. We've dropped the ball and that's not good enough and not how we do things, so again, I'm sorry, but we're trying to put it right as we speak.

Welcome back.

This is a similar s**tstorm to what I and probably every other sys admin/engineer returns to from holidays.

Good luck with the SQL resync!

thesandeman

Gordon wrote: »

They could have been trying to fix the site, fix their internet, not been able to log in to company twitter account due to passwords/policy etc.

It never stopped them the other times.
I've no problem with the site being down. These things happen. But the fact they didn't even acknowledge that there was a problem is annoying.

GLaDOS

thesandeman wrote: »

It never stopped them the other times.
I've no problem with the site being down. These things happen. But the fact they didn't even acknowledge that there was a problem is annoying.

Well tbf, Dav has acknowledged that they should have acknowledged it.

Gordon

thesandeman wrote: »

It never stopped them the other times.
I've no problem with the site being down. These things happen. But the fact they didn't even acknowledge that there was a problem is annoying.

These things happen.

MugMugs

thesandeman wrote: »

It never stopped them the other times.
I've no problem with the site being down. These things happen. But the fact they didn't even acknowledge that there was a problem is annoying.

I'm sure the follow on begging and pleading for forgivness thread is inevitable. They may be busy sacrificing a small cow at the moment before they need to resort to the pleading part.

Mrs OBumble

Orion wrote: »

What I take from this thread ... Dav gets holidays?

Not any more he doesn't. Look what happens when he goes away.

Dav

I wouldn't mind, but those were my first holidays (barring the odd day off here and there) since January.

martyos121

Dav wrote: »

I wouldn't mind, but those were my first holidays (barring the odd day off here and there) since January.

I imagine your return to Boards HQ went something like this:

Into The Blue

Orion wrote: »

What I take from this thread ... Dav gets holidays?

Dav gets holidays
Site runs on hamsters
Site goes down while Dav is on hols


You calling Dav a hamster????

Tom Dunne

Dav goes on holidays, site goes wobbly.

Dav returns from holidays, site stabilises.

Just sayin' like.....

+1 for DavClone

TherapyBoy

Dav wrote: »

..since January.

Jaysus, 2 holidays a year!?

Mary! Break out the MasterCard, the Celtic Tiger's back in town! :P

_Tombstone_

Dav wrote: »

I wouldn't mind, but those were my first holidays (barring the odd day off here and there) since January.

How many you normally have?

padd b1975

I don't know what has been more annoying these past few days....

The site acting up, or the "hilarious" one-liners about hamsters:rolleyes::rolleyes:

ThisRegard

The hamster analogy is pretty old alright.