What is wrong with the site? (Please read updated post #1)

Dav · 05-10-2015 12:15pm #1

Hello everyone.

I'm back form 2 week's holidays today, only to arrive in and find that an ESB substation has popped somewhere around the South William St area and so we've been sitting here with no internet connection in the office for nearly 4 hours. When it rains, it pours, eh?

OK, so the site has been, to use the technical term, bolloxed. I don't fully know why, but we are obviously investigating as I type. Preliminary investigations suggest that one of our database servers went kaput (we have 3) and then a DDOS attack took a second one out of sync which is why you see notification errors, 404's when you post and basically, most of the problems that've been reported over the last 5 days.

It is obviously our number 1 priority to get all 3 database servers back up and running at full speed and in full synchronisation this afternoon so that the vast bulk of these problems disappear.

I will keep you all posted via this thread as the day goes on - I have a couple of weeks worth of stuff to catch up on, so please bear with me.

Finally, I'm really sorry the site's been such a pain in the arse to use this last week or so, I've been just as frustrated as the rest of you because I've been trying to use it too on my time off. We've dropped the ball and that's not good enough and not how we do things, so again, I'm sorry, but we're trying to put it right as we speak.

*Update - November 2016*

(Boards.ie: Mark) Been talking with the tech guys about how errors are reported, and it would help everyone involved (yourselves included as it could help us get to the root of issues) if you could do the following.

Before submitting a bug report, please take the following steps to attempt to resolve it:

Clear cache and cookies
Restart your browser
Try a different browser
Try incognito mode
Restart your machine
Try a different computer
Helpful link: https://support.mozilla.org/en-US/kb/websites-dont-load-troubleshoot-and-fix-errors

Should that not work, please include as much information as possible, such as:

The URL of the page you were on when the error occurred
Steps to reproduce the process
Screenshots
What you expected to happen
What actually happened
The results of the Diagnostics page - please delete the IP address line
Whether you are using the Legacy or Responsive site
Whether you are based in Ireland or overseas
Whether you are in an office / operating behind a proxy/firewall

Thanks all!

NOTICE (02/03/17):
Hi everyone,

Some work is going to be carried out today in the datacentre, which MAY cause some disruption (though hopefully not). Sorry for any inconvenience caused.

Darkglasses · 05-10-2015 12:25pm

Thanks Dav, good luck with the fix.

martyos121 · 05-10-2015 12:29pm

Won't somebody please think of the hamsters! They must be knackered, they need a holiday!

Orion · 05-10-2015 12:48pm

What I take from this thread ... Dav gets holidays?

libelula · 05-10-2015 12:56pm

I can only imagine the head on Dav coming back from holidays to this. He'll be all over the R&R forum later

jonski · 05-10-2015 1:16pm

I was coming on to have a whinge about there being no central thread with a notice in the first post from admin telling us what the problem was . Now I realise there is but I couldn't see it due to the problems with the site . Now I'm whinging at myself for not thinking about that ....but none of this matters because this might not even post and even if it does no one will probably see it, at least until the site is fixed at which stage no one will care about this thread any more ........... waffle waffle waffle .

my3cents · 05-10-2015 1:27pm

Just wondering if anyone has officially said anything to the companies and reps that reply in the Talk to forums? They are afaik paying for a service and might be more in the dark than those of us that bother to post about site issues?

MJohnston · 05-10-2015 1:30pm

I'm surprised nothing has been posted to the Boards.ie Twitter or Facebook accounts - even Facebook uses Twitter for downtime updates!

Synode · 05-10-2015 2:09pm

MJohnston wrote: »

I'm surprised nothing has been posted to the Boards.ie Twitter or Facebook accounts - even Facebook uses Twitter for downtime updates!

I contacted Boards on Twitter and they never responded. They're normally quite quick so not sure why they didn't this time

libelula · 05-10-2015 2:09pm

I can only imagine the head on Dav coming back from holidays to this. He'll be all over the R&R forum later

05-10-2015 2:12pm

Is there anything here?

Just to add - I followed the link from Dav's post, but there's absolutely nothing here. No posts. Nothing.

Myrddin · 05-10-2015 2:15pm

Refresh the page

jimgoose · 05-10-2015 2:24pm

Relational databases are like two-stroke engines. Mighty yokes while they keep zinging along, but now and then the Gods of Ballsup notice you and send Hell and Perdition. It is at this point that the administrator realises what a perversion of physics the thing is and how amazing it is that it works at all. The divil is stuck in 'em - the divil, I tells ye!!

Skylinehead · 05-10-2015 2:27pm

jimgoose wrote: »

Relational databases are like two-stroke engines. Mighty yokes while they keep zinging along, but now and then the Gods of Ballsup notice you and send Hell and Perdition. It is at this point that the administrator realises what a perversion of physics the thing is and how amazing it is that it works at all. The divil is stuck in 'em - the divil, I tells ye!!

Never change, Jim :cool:

CatInABox · 05-10-2015 2:27pm

Myrddin wrote: »

Refresh the page

You're assuming that he can see your post?

I've actually start to find the whole situation hilarious, I'm starting to feel like I've got Alzheimers. I go into a forum, see that there's no new posts and think to myself "Wait, why did I come in here?"

Ruu · 05-10-2015 2:29pm

Thanks for the update.

Myrddin · 05-10-2015 2:31pm

CatInABox wrote: »

You're assuming that he can see your post?

It's happening me too, I click the thread link from the forum & get taken to an empty boards page with no post/details in it. A refresh fixes it, mostly

I've actually start to find the whole situation hilarious, I'm starting to feel like I've got Alzheimers. I go into a forum, see that there's no new posts and think to myself "Wait, why did I come in here?"

Lol, I know the feeling

Myrddin · 05-10-2015 2:32pm

Seems a lot better now...

Tompatrick · 05-10-2015 2:35pm

martyos121 wrote: »

Won't somebody please think of the hamsters! They must be knackered, they need a holiday!

Oh Lollers ! What an original joke....

martyos121 · 05-10-2015 2:37pm

Much improved now, still one or two problems but nowhere near as bad as it was an hour or two ago or last night.

SEPT 23 1989 · 05-10-2015 2:42pm

working well now thanks

Oryx · 05-10-2015 2:44pm

Most people go on holidays from work and know, regardless of how critical their role, the place won't fall down without them.

But not in Dav's case. Hes not allowed to leave again until they clone him.

Dav · 05-10-2015 2:57pm

OK, so I'm told that the databases have been brought back into synchronisation and that this should have resolved a lot of the problems we've been having.

We're obviously monitoring carefully to see that everything is stable, but with a bit of luck, we're past the worst of it.

Oryx · 05-10-2015 3:15pm

Just a fyi, still getting logged out on touch.

Irish Steve · 05-10-2015 3:20pm

Now comes the inquest into Who, how, why and identifying the innocent and guilty parties, which could be "interesting".

It will be nice to be able to rely on the page showing what's actually happening again, if I'd read one particular message once more, I think there would have been an imprint of a laptop on the wall the other side of the room.

thesandeman · 05-10-2015 3:31pm

Seems ok now.
Thanks for the update but one single announcement on Twitter from whoever was minding the shop would have stopped a lot of confusion.

Hollister11 · 05-10-2015 3:36pm

It's working a lot better now. I still get the odd '404' when trying to access treads.

Gordon · 05-10-2015 3:37pm

thesandeman wrote: »

Seems ok now.
Thanks for the update but one single announcement on Twitter from whoever was minding the shop would have stopped a lot of confusion.

This wouldn't have helped I imagine:

Dav wrote: »

we've been sitting here with no internet connection in the office for nearly 4 hours.

jimgoose · 05-10-2015 3:46pm

Skylinehead wrote: »

Never change, Jim :cool:

Remember this: It ain't over until the bastard thing eats another set of pistons. And even then it ain't over - the only end to this living Hell is divorce, poverty and rocking back-and-forth in a foetal position gibbering semi-coherently about port heights and premix ratios.

GLaDOS · 05-10-2015 4:03pm

My avatar isn't loading in the floating menu, tear the whole thing down.

jimgoose · 05-10-2015 4:04pm

ScienceNerd wrote: »

My avatar isn't loading in the floating menu, tear the whole thing down.

Yes, my hovercraft is also full of eels. We should, like, set fire to shít an' stuff.

foggy_lad · 05-10-2015 4:07pm

I've sent a tazer to boards HQ to provide a little encouragement to the hamsters but it may not arrive due to the post being bollixed too.

jack presley · 05-10-2015 4:21pm

Dav,

http://touch.boards.ie/thread/2057501964/1/#post97232953

This issue is still happening.

thesandeman · 05-10-2015 5:14pm

Gordon wrote: »

This wouldn't have helped I imagine:

They could have walked outside and tweeted it from their phone.

Insect Overlord · 05-10-2015 5:18pm

thesandeman wrote: »

They could have walked outside and tweeted it from their phone.

But then they wouldn't have been working on fixing the problem. :P

naughto · 05-10-2015 5:19pm

Oryx wrote: »

Just a fyi, still getting logged out on touch.

Ya same here and on the phone

Gordon · 05-10-2015 5:19pm

thesandeman wrote: »

They could have walked outside and tweeted it from their phone.

They could have been trying to fix the site, fix their internet, not been able to log in to company twitter account due to passwords/policy etc.

circadian · 05-10-2015 5:23pm

Dav wrote: »

Hello everyone.

I'm back form 2 week's holidays today, only to arrive in and find that an ESB substation has popped somewhere around the South William St area and so we've been sitting here with no internet connection in the office for nearly 4 hours. When it rains, it pours, eh?

OK, so the site has been, to use the technical term, bolloxed. I don't fully know why, but we are obviously investigating as I type. Preliminary investigations suggest that one of our database servers went kaput (we have 3) and then a DDOS attack took a second one out of sync which is why you see notification errors, 404's when you post and basically, most of the problems that've been reported over the last 5 days.

It is obviously our number 1 priority to get all 3 database servers back up and running at full speed and in full synchronisation this afternoon so that the vast bulk of these problems disappear.

I will keep you all posted via this thread as the day goes on - I have a couple of weeks worth of stuff to catch up on, so please bear with me.

Finally, I'm really sorry the site's been such a pain in the arse to use this last week or so, I've been just as frustrated as the rest of you because I've been trying to use it too on my time off. We've dropped the ball and that's not good enough and not how we do things, so again, I'm sorry, but we're trying to put it right as we speak.

Welcome back.

This is a similar s**tstorm to what I and probably every other sys admin/engineer returns to from holidays.

Good luck with the SQL resync!

thesandeman · 05-10-2015 5:38pm

Gordon wrote: »

They could have been trying to fix the site, fix their internet, not been able to log in to company twitter account due to passwords/policy etc.

It never stopped them the other times.
I've no problem with the site being down. These things happen. But the fact they didn't even acknowledge that there was a problem is annoying.

GLaDOS · 05-10-2015 5:42pm

thesandeman wrote: »

It never stopped them the other times.
I've no problem with the site being down. These things happen. But the fact they didn't even acknowledge that there was a problem is annoying.

Well tbf, Dav has acknowledged that they should have acknowledged it.

Gordon · 05-10-2015 5:43pm

thesandeman wrote: »

It never stopped them the other times.
I've no problem with the site being down. These things happen. But the fact they didn't even acknowledge that there was a problem is annoying.

These things happen.

MugMugs · 05-10-2015 5:47pm

thesandeman wrote: »

It never stopped them the other times.
I've no problem with the site being down. These things happen. But the fact they didn't even acknowledge that there was a problem is annoying.

I'm sure the follow on begging and pleading for forgivness thread is inevitable. They may be busy sacrificing a small cow at the moment before they need to resort to the pleading part.

Mrs OBumble · 05-10-2015 5:53pm

Orion wrote: »

What I take from this thread ... Dav gets holidays?

Not any more he doesn't. Look what happens when he goes away.

Dav · 05-10-2015 6:02pm

I wouldn't mind, but those were my first holidays (barring the odd day off here and there) since January.

martyos121 · 05-10-2015 6:16pm

Dav wrote: »

I wouldn't mind, but those were my first holidays (barring the odd day off here and there) since January.

I imagine your return to Boards HQ went something like this:

coming%20back_zpsjal6q9k4.gif

Into The Blue · 05-10-2015 6:22pm

Orion wrote: »

What I take from this thread ... Dav gets holidays?

Dav gets holidays
Site runs on hamsters
Site goes down while Dav is on hols

You calling Dav a hamster????

Tom Dunne · 05-10-2015 6:36pm

Dav goes on holidays, site goes wobbly.

Dav returns from holidays, site stabilises.

Just sayin' like.....

+1 for DavClone

TherapyBoy · 05-10-2015 6:36pm

Dav wrote: »

..since January.

Jaysus, 2 holidays a year!?

Mary! Break out the MasterCard, the Celtic Tiger's back in town! :P

_Tombstone_ · 05-10-2015 7:11pm

Dav wrote: »

I wouldn't mind, but those were my first holidays (barring the odd day off here and there) since January.

How many you normally have?

padd b1975 · 05-10-2015 7:58pm

I don't know what has been more annoying these past few days....

The site acting up, or the "hilarious" one-liners about hamsters:rolleyes::rolleyes:

ThisRegard · 05-10-2015 8:00pm

The hamster analogy is pretty old alright.

What is wrong with the site? (Please read updated post #1)

Comments