Advertisement
If you have a new account but are having problems posting or verifying your account, please email us on hello@boards.ie for help. Thanks :)
Hello all! Please ensure that you are posting a new thread or question in the appropriate forum. The Feedback forum is overwhelmed with questions that are having to be moved elsewhere. If you need help to verify your account contact hello@boards.ie

So what happened this time?

Options
  • 14-01-2014 6:03pm
    #1
    Dav
    Closed Accounts Posts: 8,840 ✭✭✭


    You go all year with no outages and then 2 come along within 10 days.

    So what happened today?

    First of all, it wasn't me! :D

    One of the database slaves had a major failure with it's hard disks. Before anyone asks, yes they were in RAID (1 to be exact), but both disks failed. It's rare that your redundancy fails at the same time as the main device, but not unheard of.

    We all probably noticed the slow down around lunch time as the knock on effects of the disks dying. At 14:30 one of our sys admins was in his car on the way out to Digiweb to see what was wrong. At 15:00, we took the site offline as it was clear that we were risking data loss and corruption if we continued to let the database machines get out of sync.

    After a quick investigation, we realised that if we didn't allow posting, people could still browse Boards, so we got busy making the site Read Only and set that live whilst Chris went about rebuilding the server with new hard disks.

    So here we are at 5pm and work is ongoing. You can read this, but can't reply yet, but we wanted to let you know what's going on with as much detail as possible.
    Post edited by Shield on


«13

Comments

  • Options
    #2
    The Hill Billy
    Registered Users Posts: 22,756 ✭✭✭✭The Hill Billy


    Is a Thank not a write?


  • Options
    #3
    Rovi
    Closed Accounts Posts: 8,570 ✭✭✭Rovi


    I've found that non-mods can still send PM's :D


  • Options
    #4
    Mrs OBumble
    Registered Users Posts: 25,766 ✭✭✭✭Mrs OBumble


    Do threads all look locked to ordinary users?

    I've had a reported post saying "Why is this thread locked???can you re open it please this applies to me yet again"

    Due to the no-text-only-emails "feature" (grr), I cannot go directly to the thread that's referred to. But am wondering if that's what really happened ... and if so, que lots of reported posts :-)


  • Options
    #5
    Dav
    Closed Accounts Posts: 8,840 ✭✭✭Dav


    They're not locked, the person will get a message telling them they can't post because of the problem if they try.

    I've had to close some forums manually because they have weird permissions, so that will say "forum is closed for posting" but that'll all be switched back when we're live again.


  • Options
    #6
    Myrddin
    Closed Accounts Posts: 33,733 ✭✭✭✭Myrddin


    Dav wrote: »
    One of the database slaves had a major failure

    This is why slavery has largely been abolished throughout the world, it's just not damned reliable enough. I recommend getting in some paid workers, yes it costs in the long run, but it balances out with reliability...


  • Advertisement
  • Options
    #7
    Gran Hermano
    Closed Accounts Posts: 9,330 ✭✭✭Gran Hermano


    How old was the database server/drives?

    Could one say they were twelve years a slave?


  • Options
    #8
    TouchingVirus
    Registered Users Posts: 8,584 ✭✭✭TouchingVirus


    Gran Hermano wrote: »
    How old was the database server/drives?

    Could one say they were twelve years a slave?

    Boooooo :pac:
    The Hill Billy wrote: »
    Is a Thank not a write?

    My guess on this is a thank is a write but writes aren't really a problem...on a small scale. The problem would be with a slave out of action you'd massively load the other slave(s?) leading to slowdowns, strange "missing" posts when you post in a thread but when it's displayed your post isn't there. Cue the posts about the "missing" posts, notifications without there being new posts etc etc.

    mySQL's caching is surprisingly efficient but it does get nuked every time you write and when Boards is down a slave you really can't be affording to nuke the caches for posts/threads that often.

    All hail the new and improved hamster wheels, whenever they arrive. And fair play for implementing the Read-only site on such short notice. Commiserations to Chris, it has been a rough 10 days for him :p


  • Options
    #9
    Posy
    Moderators, Recreation & Hobbies Moderators Posts: 27,539 Mod ✭✭✭✭Posy


    TouchingVirus wrote: »
    All hail the new and improved hamster wheels, whenever they arrive. And fair play for implementing the Read-only site on such short notice. Commiserations to Chris, it has been a rough 10 days for him :p
    Whatever. This sort of thing never happened back when Danny was here. :rolleyes: :pac:


  • Options
    #10
    El Guapo!
    Registered Users Posts: 12,431 ✭✭✭✭El Guapo!


    Is it safe to come out now?

    <_<

    >_>


  • Options
    #11
    Dav
    Closed Accounts Posts: 8,840 ✭✭✭Dav


    Not quite, but it's getting safer...


  • Advertisement
  • Options
    #12
    Micky Dolenz
    Registered Users Posts: 44,080 ✭✭✭✭Micky Dolenz


    El Guapo! wrote: »
    Is it safe to come out now?

    <_<

    >_>


    We are all open minded here.

    Good on you.


  • Options
    #13
    knucklehead6
    Registered Users Posts: 6,752 ✭✭✭knucklehead6


    Are mods the only ones allowed post?


  • Options
    #14
    AlmightyCushion
    Moderators, Education Moderators, Technology & Internet Moderators Posts: 35,059 Mod ✭✭✭✭AlmightyCushion


    I think we all know what really happened here. Dav messed something up again and didn't want to get in trouble so he poured a big jug of coffee all over the server and blamed it on a hardware malfunction. For shame, Dav, for shame.


  • Options
    #15
    TouchingVirus
    Registered Users Posts: 8,584 ✭✭✭TouchingVirus


    knucklehead6 wrote: »
    Are mods the only ones allowed post?

    For now :) Enjoy it while it lasts :pac:


  • Options
    #16
    knucklehead6
    Registered Users Posts: 6,752 ✭✭✭knucklehead6


    TouchingVirus wrote: »
    For now :) Enjoy it while it lasts :pac:

    Weird, cos I've been demoted to reg user! And my grumpy baby avatar is AWOL.

    Ah well, I'm only a hosted mod, not like all you real mods. :-(


  • Options
    #17
    TouchingVirus
    Registered Users Posts: 8,584 ✭✭✭TouchingVirus


    Your avatar is present and accounted for. I also see your HMOD status


  • Options
    #18
    knucklehead6
    Registered Users Posts: 6,752 ✭✭✭knucklehead6


    TouchingVirus wrote: »
    Your avatar is present and accounted for. I also see your HMOD status

    Must be my phone so. Feckin iPhones!!


  • Options
    #19
    Micky Dolenz
    Registered Users Posts: 44,080 ✭✭✭✭Micky Dolenz


    TouchingVirus wrote: »
    Your avatar is present and accounted for. I also see your HMOD status


    Stop messing with his head.


  • Options
    #20
    TouchingVirus
    Registered Users Posts: 8,584 ✭✭✭TouchingVirus


    Micky Dolenz wrote: »
    Stop messing with his head.

    I'm expecting you to steal his avatar any moment now


  • Options
    #21
    knucklehead6
    Registered Users Posts: 6,752 ✭✭✭knucklehead6


    TouchingVirus wrote: »
    I'm expecting you to steal his avatar any moment now

    I'm getting onto the mods to report you all for bullying! ;-)


  • Advertisement
  • Options
    #22
    johnnyskeleton
    Moderators, Entertainment Moderators, Politics Moderators Posts: 14,479 Mod ✭✭✭✭johnnyskeleton


    Any chance that important announcements could be made available on the mobile site as well as the full site?


  • Options
    #23
    MugMugs
    Registered Users Posts: 17,875 ✭✭✭✭MugMugs


    Micky Dolenz wrote: »
    Stop messing with his head.

    Heheh...... Messing with his head.....



    I'll get me coat.


  • Options
    #24
    El Guapo!
    Registered Users Posts: 12,431 ✭✭✭✭El Guapo!


    Micky Dolenz wrote: »
    We are all open minded here.

    Good on you.

    I'm fabulous!!!


  • Options
    #25
    robindch
    Moderators, Society & Culture Moderators Posts: 24,401 Mod ✭✭✭✭robindch


    knucklehead6 wrote: »
    Are mods the only ones allowed post?
    Suits me.


  • Options
    #26
    Mr. G
    Moderators, Technology & Internet Moderators Posts: 4,621 Mod ✭✭✭✭Mr. G


    knucklehead6 wrote: »
    Are mods the only ones allowed post?

    Yep, the announcement said mods and verified reps.


  • Options
    #27
    CrowdedHouse
    Registered Users Posts: 6,814 ✭✭✭CrowdedHouse


    Back ?

    Seven Worlds will Collide



  • Options
    #28
    TouchingVirus
    Registered Users Posts: 8,584 ✭✭✭TouchingVirus


    CrowdedHouse wrote: »
    Back ?

    Yes. Pats on the back all round for the sysadmins & developers - some effort :)


  • Options
    #29
    GLaDOS
    Moderators, Regional East Moderators Posts: 23,212 Mod ✭✭✭✭GLaDOS


    Oh look, Dav's getting all the thanks again. What a coincidence.

    Cake, and grief counseling, will be available at the conclusion of the test



  • Options
    #30
    NicsM
    Subscribers Posts: 342 ✭✭NicsM


    Fair play Chris and Colm, that was the last thing you needed on a Tuesday :)


  • Advertisement
  • Options
    #31
    5starpool
    Subscribers Posts: 32,850 ✭✭✭✭5starpool


    A private forum I post in still isn't available for posting, yet another private one I'm a member of is. Are there still glitches in the matrix?

    Edit: Private forum working ok now - Thanks for getting it all back up and running!


Advertisement