Tomcat Application Slow & Timing Out (High Volume)

t1mm · 18-09-2014 1:00pm #1

Hi there,

I have a site that consistently has 500 users tinkering about. It's written in Java Spring, sitting on Tomcat7, proxied with apache httpd mod_proxy, all running on a dedicated server with more than enough power I would have thought (quad core 3ghz, 8gb ram). The site is however running very slowly, and even timing out sometimes.

- Application logic is fine
- Tomcat is configured to 1gb of heap - way more than enough according to psi probe.

Does anyone have any tips? I have tried mod_disk_cache within apache and that seems to have at least eliminated timeouts, but the wait time is still pretty awful.

amen · 18-09-2014 1:55pm

what is the app doing ? Calculations, database access, rendering screen elements?

Is the entire application slow or just some parts of it or at just some times during the day (loads of users logged in)

t1mm · 18-09-2014 2:03pm

Database access. MySQL stuff - nothing too out of the ordinary. It seems to function fine when there are about 350 users on it, but 450ish and it starts to struggle.

The whole application is slow at that point - even pages that don't even make any database calls.

ChRoMe · 18-09-2014 2:19pm

t1mm wrote: »

Database access. MySQL stuff - nothing too out of the ordinary. It seems to function fine when there are about 350 users on it, but 450ish and it starts to struggle.

The whole application is slow at that point - even pages that don't even make any database calls.

Got any indexes on your database?

t1mm · 18-09-2014 2:45pm

Yep. I don't think MySQL is really the problem though - considering pages without any MySQL connections load just as slowly as others, under times of stress. Is it possible though that the users who ARE viewing a mysql-loading page are causing some backlog on tomcat/application time whereby even standard flat pages aren't being served?

test2014 · 18-09-2014 2:54pm

Had a similar problem that turned out to be the host. I had an Octa-Core Dedicated with 24gb and it died with less traffic.

They promised me blind it wasnt them but i chanced migrating to a different Quad-Core and it was smooth as silk and using barely any resources.

Also when you say page without MYSQL load just as slowly. That doesnt mean anything. It could still be MYSQL killing the server. It wont just slow down sites with queries.

Process of elimination. Switch off the MYSQL and see what happens to the pages with no MYSQL and the resource usage.

t1mm · 18-09-2014 2:56pm

If I turn off the mysql, one of two things will happen: 1. The static pages will load nicely. 2. Nothing will change. Would you have any recommendations for steps following on from those scenarios?

Thanks for your help guys, much appreciated

test2014 · 18-09-2014 3:03pm

Im assuming you dont have another Server to chance a migrate and see what happens. Correct?

t1mm · 18-09-2014 3:04pm

I do actually - the development server. Only issue is that I don't directly control the DNS records so its a pain. I might try give that a shot if it can't be otherwise fixed, but would like to exhaust a few other options first if possible!

test2014 · 18-09-2014 3:11pm

Server faults are a pain in the arse alright. Anywhere you ask for help they just say check the logs.

I have this nasty feeling youll go around in circles like i did. You think its something. Then it just dies again after a few days when you think all is good.

Itll be plain as day if its the server if it runs smoothly after the migrate. If its the same at least you still have files on old server and you just need to change DNS back.

Any chance of getting the DNS to point to Cloudflare to make it easier for yourself in future?

h57xiucj2z946q · 18-09-2014 3:14pm

Could you try something more lightwight before you make code changes?

Like jetty for the servelt container (using asynchronous connections) and nginx for static content.

Giblet · 18-09-2014 3:34pm

Profile. Seriously, it doesn't matter how powerful your machine is if some rogue function is taking up all the cpu time / disk / network resources.

Aswerty · 18-09-2014 3:49pm

Just a few things:

Have you checked resource consumption during these slow periods? (e.g. check RAM and CPU usage) If either are sturated it shouldn't be too hard to work your way backwards as to what is causing the slowdown.

Its not obvious from your description but is MySQL and Tomcat on the same server?

Can you profile your DB queries to check how long they are taking to hit the DB? Ideally try profiling on the production server and not just on the development server.

Use some browser tools to check if there are any specific HTTP requests causing the delay.

t1mm · 18-09-2014 5:12pm

Thanks again for all of your suggestions guys. I'm going to try profiling it on my local machine and see if I can dig out any rogue functions. If nothing turns up there I'll try the DNS switch.

@Aswerty: Yep, resource usage looks fine according to "top". Yep, Tomcat and MySQL are on the same server. Sorry to ask such a perhaps silly question but how would I go about profiling the queries?

Do you guys know of any profiler I could run on the server alongside my app?

Aswerty · 18-09-2014 6:10pm

I don't use MySQL so am not sure what the best options are for profiling plain old SQL. I expect the common management clients (e.g. MySQL Workbench) have some profiling features.

If you're interested in going down this route you might let us know how you access the DB from your application (e.g. SQL baked into the application/stored procedures/an ORM). Though I would expect if the DB qureies are your issue that you would see high resource usage. Since you don't it might not be whats causing you issue.

If the servers are hosted then it might be worth getting in touch worth your host to check there is no networking issue on their end.

robbiezero · 18-09-2014 6:22pm

In some Linux installations of Apache, HTTP Keepalive is turned off by default in httpd.conf. Might be something to check and turn it on if its and see if it improves performance.
Check your OS ulimits on file handles and memory, stack also. Max Number of threads in tomcat too.

magooly · 19-09-2014 8:08pm

1. Are you using a dbconnection pool(c3) and if so what are your min/max connections

2. Are there any remote 3rd party JS/images on your pages that are timing out(google analytics).

3. Are there any hints in the tomcat or /var/log/messages logs during this time

4. What is your MaxThreads setting in Tomcat(default is 200)

5. Are you logging excessively?

GreeBo · 19-09-2014 8:30pm

t1mm wrote: »

Database access. MySQL stuff - nothing too out of the ordinary. It seems to function fine when there are about 350 users on it, but 450ish and it starts to struggle.

The whole application is slow at that point - even pages that don't even make any database calls.

Sounds like thread pool if even non DB pages are slow.

You need to identify at what point the wait is happening, is it apache, tomcat or application code.

Find a request that has timed and and check the logs at each section to see how far it gets.
Check some others and see are the dying at the same point.
Whats the response/stack trace in the logs for the timed out connections?

Do you get partially loaded pages, static content missing for example? Or static content is served and dynamic is missing?

Finally, 1GB might be loads, but check your GC logs (you will need to enable logging) you might be thrashing.

Tomcat Application Slow & Timing Out (High Volume)

Comments