The story with the slowdown and its fix.

DeVore

Lo all,

I´m writing this from Spain because I wanted to give you all an update on the slowdown and whats happening.

Our best explanation of this problem is that on occasion, particularly at peak times, our servers are getting bottlenecked and then wait-state various threads (unix threads, not boards threads) pending release of machine resources (CPU and Memory).

We cant find any rhyme nor reason for the bottlenecks but its probably a combination of things including big searchs, big threads and a few of the options we have allowed.

Ok, so we knew this would happen though I must admit I am concerned that it happened so soon after we moved to the 3 server system. All the same we have prepared for this and the system is configured for us to slide in another server fairly easily.

So thats what we´re going to do, we´re going to get another server to spread the load and hopefully stop these top outs and subsequent bottlenecks and slowdowns.

Now, this is going to take time. Cloud´s missus is expecting their first child soon (yay!) and Regi has two weeks off to study for his exams. We´re going to requisition the server in the next few days, allow 5 days for staging, another day or two for setup, a day to install and a day to debug and contingency. Thats the best case scenario. So, we can expect probably 2 weeks from Tuesday to be realistic.

Sorry about the slowdowns, yet more growing pains but we´ll get past them.

DeV

aidan_walsh

Thanks for the update.

Out of interest, are the likes of adverts.ie, boards.us etc running on the same servers, and if so do you know how much of a strain they are putting on the performance of boards?

hullaballoo

Wow, another server! You guys are really spoiling us!

I was expecting searches to go again. Thanks admins!

flogen

aye, another one... if this keeps up you'll be renting out entire apartment blocks to house them by 2010...

so is the income of boards.ie sufficient to afford a new server? Maybe that idea of biggest pledge gets to name the server could be implimented to ease the cost!

CuLT

flogen wrote:

aye, another one... if this keeps up you'll be renting out entire apartment blocks to house them by 2010...

Bucephalus!

Edit: I've just realized that this reference is incredibly vague; Bucephalus is the name of a supercomputer which took over an entire apartment building as it was expanded by its owner, "Alexander", in a short story by Isaac Asimov.

The_Minister

Does a new server cost much?

Ruu_Old

A bajillion dollars like! Id be happy to donate a few squid if needed

Beruthiel

DeVore wrote:

Cloud´s missus is expecting their first child soon

Daddy Cloud will really be a Daddy!
Congrats John!

and Regi has two weeks off to study for his exams.

good luck with them Dan

jeez ye are all busy boys, hope the new server does the trick

kaimera

flogen wrote:

so is the income of boards.ie sufficient to afford a new server?

Yea, and a trip to Spain for one lucky admin :v:

Victor

What if you put AH on one sever and spread everything else around the rest?

RuggieBear

Victor wrote:

What if you put AH on one sever and spread everything else around the rest?

I'd say banning you would lighten the server load:D

aidan_walsh

Victor wrote:

What if you put AH on one sever and spread everything else around the rest?

aidan_walsh, you do not have permission to access this page. This could be due to one of several reasons:

1. Your user account may not have sufficient privileges to access this page. Are you trying to edit someone else's post, access administrative features or some other privileged system?
2. If you are trying to post, the administrator may have disabled your account, or it may be awaiting activation.
3. afterhours.boards.ie may have crashed under the weight of white noise and bebo threads. If this is the case, the administrators have been notified by email that all is well.

mike65

I just like the way Dev made a point of saying he was writing to us from Spain.

'tis nice to know you're thinking of us as we struggle on heroically.

Mike.

Talliesin

The description of the original problem says "thread deadlock" to me. Any luck investigating that?

Mimikyu

This post has been deleted.

GoneShootin

DeVore wrote:

I´m writing this from Spain

Good to see my subscription money at work!

amp

I hear that some people do work in other countries, even Spain.

Victor

amp wrote:

I hear that some people do work in other countries, even Spain.

What? During the afternoon? On Sunday?

BuffyBot

Victor wrote:

What? During the afternoon? On Sunday?

Sangria testers?

Beruthiel

BuffyBot wrote:

Sangria testers?

where do I sign up!?

John

Beruthiel wrote:

where do I sign up!?

I think it's part of a time share weekend. You all presume Dev is having fun but he's actually trying to flog a load of apartments that will be built ~2017.

CrowdedHouse

Priority Access for Subscribers - let the rest wait

DeVore

Ruu wrote:

A bajillion dollars like! Id be happy to donate a few squid if needed

They cost about 3.5k each. We are good for cash, thats because not everyone blocks our ads. See, ad impressions = money = servers = capacity = growth = users = ad impressions = etc etc
Then people wonder why I get annoyed when people talk about blocking our ads.

We don´t need money, we are self sufficent at the moment. We plan for this sort of thing. Its just another one of those useless things I do around here. <-- token smiley.

DeV.

Victor

Beruthiel wrote:

where do I sign up!?

Realise there is no vodka in sangria.

Talliesin

There was in the sangria that was, according to the broken English of the Spanish woman I was drinking it with "made better".

This was at a party we somehow ended up in when my dad was taking me for a couple of halves to teach me how to drink responsibly.

Giblet

Have you thought about cutting back on some of the features. Like mouse over thread previews, stuff that doesn't really need to be there. Also large threads do cause problems. Maybe they can be locked after a certain point and a new thread started with the same subject.

Some of the little helpers might be causing problems too, like when you type a user name into search, it will bring up a list dynamically of all the user names. Maybe have stuff like sigs off as default and they have to be turned on by the user on each post (haha). Crap like birthday notifications can surely be dropped aswell ...


All the little things might add up :S:S

Sparky

DeVore wrote:

They cost about 3.5k each. We are good for cash, thats because not everyone blocks our ads. See, ad impressions = money = servers = capacity = growth = users = ad impressions = etc etc
Then people wonder why I get annoyed when people talk about blocking our ads.

We don´t need money, we are self sufficent at the moment. We plan for this sort of thing. Its just another one of those useless things I do around here. <-- token smiley.

DeV.

I could of swore it was your poker wins that was keeping this place running

अधिनायक

Maybe ask for some help from users in your programming forum. Database & web server optimisation is often misunderstood and you likely already have members who could help you. Could save you some cash and give a better solution.

Hobbes

अधिनाय&#2325 wrote:

Maybe ask for some help from users in your programming forum. Database & web server optimisation is often misunderstood and you likely already have members who could help you. Could save you some cash and give a better solution.

We don't answer Homework questions!

Actually if I recall there is already a forum for dev-boards related stuff.
http://www.boards.ie/vbulletin/forumdisplay.php?f=228

Sparky

अधिनाय&#2325 wrote:

Maybe ask for some help from users in your programming forum. Database & web server optimisation is often misunderstood and you likely already have members who could help you. Could save you some cash and give a better solution.

Well Regi and Cloud know their stuff well, so I'd assume they went that road aswell.

अधिनायक

No disrespect but a few times I've seen this kind of conversation:

Boss: how is the optimisation going?
Technical manager: Excellent. We have optimised our internal counters related to waits latches and locks, we are getting over 99% cache hit ratio and we haven't had a deadlock in a week.
Boss: <clicks on Buy Now button - page takes a minute to load> Doesn't look so good from my end.

I don't see much participation in the dev forum. It looks more like a place where people report bugs once in a while.

Sparky

Just so you know, one of the admins is the Tech manager at H365, where boards is hosted, so I doubt they over looked tweaking etc.

अधिनायक

Sparky-s wrote:

Just so you know, one of the admins is the Tech manager at H365, where boards is hosted, so I doubt they over looked tweaking etc.

I didn't mean to suggest that the site technicians were unskilled just that a participatory approach might help, particularly when the OP states that the people involved in the optimisation effort now have other commitments. My reading of the first post is that the optimisation efforts have failed and that the next step is the reluctant purchase of another server (a web server I guess). This may or may not improve performance.

Posting server logs and discussing optimisation approach is free.

WizZard

One of the admins has previously asked for help in the Programming forum regarding optimisation of MySQL statements.
I'm sure they know that they can avail of the forum if they want.

WhiteWashMan

i bet scaramanga never had this problem when he was building his empire....

jimmycrackcorm

WhiteWashMan wrote:

i bet scaramanga never had this problem when he was building his empire....

But Scaramanga stuck to Ansi SQL 92.

Chong

Is the removal of the new posts function part of the update?

Esel

Willstev wrote:

Is the removal of the new posts function part of the update?

Yeah, noticed 'New Posts and 'Search' have just been disabled - presumably due to the speed problems?

Dave!

Giblet wrote:

Have you thought about cutting back on some of the features. Like mouse over thread previews, stuff that doesn't really need to be there. Also large threads do cause problems. Maybe they can be locked after a certain point and a new thread started with the same subject.

Some of the little helpers might be causing problems too, like when you type a user name into search, it will bring up a list dynamically of all the user names. Maybe have stuff like sigs off as default and they have to be turned on by the user on each post (haha). Crap like birthday notifications can surely be dropped aswell ...


All the little things might add up :S:S

I agree!

Danke for the update anywho, it was getting terribly frustrating recently :mad:

Victor

Things seem significantly faster, with no hung pages.

mike65

Yep its about 100% quicker.

Mike.

Mimikyu

This post has been deleted.

Zaph

Good news - new server. WooHoo!!!
Bad news - no search.
Better news - Cloud junior. Congratulations!
Crappy news - Regi has exams. Best of luck with them.

Sometimes it's not always about us - even the Admins have lives that they have to fit the rest of us into somehow. Who'd have guessed that?

Esel

I think if a choice had to be made between losing 'New Posts' and 'Search', then 'New Posts' should be the one to go. The main users are bored geeks and nerds who have nothing better to do than click on a link - apart from me of course. They should use wikipedia.org's 'Random Page' instead - they might learn something!

DublinWriter

esel wrote:

I think if a choice had to be made between losing 'New Posts' and 'Search', then 'New Posts' should be the one to go.

No way! It's my main excuse not to do any work, plus gives you a great cross section of what people are talking about in general.

I can't see why it had to go to improve performance. I don't know what DBMS boards uses, but how hard can it be to implement a reverse datetime index?

GoneShootin

Tell you what though since Search was disabled the site is a hell of alot faster

zAbbo

Disable all the crap, sigs, images, most of what giblet said. Maybe limit the max posts per page.

Cloud

Okay, I'm currently working with BoardTracker.com to replace our search with their engine. Things have improved as you can see already...

CrowdedHouse

It's flying now - even on dial-up

Cloud

Okay, to figure out this problem, I've disabled search for now.

Tonight at 9PM, we're gonna turn search back on for the night...

I expect to see a big jump in server loads.

So what I want you guys to do is, between 9 PM and 9 AM tomorrow morning, search, search, search - we'll see what the effect is and figure out what is slowing us down.

Many thanks to Board Tracker for all their help today.

tbh

done and done.

and I mean done!