Poweredge & SBS2008 - Slow + Unusable; Low CPU and Mem usage

Zirconia

Having severe performance problems on a Poweredge T310 with SBS 2008 SP2.

We thought it was the RAID controller, and Dell replaced the Motherboard and RAID card earlier this week, but the problems keep coming back. It's so bad that we've had to move some applications off this machine to keep business users happy.

The strange thing is that the CPU only seems to be running about 5 to 10 % max and the physical RAM is only using 60 to 80 % (including SQL and Exchange), with no processes standing out in task manager. But Email is taking up to 50 minutes to process, mapped drives keep coming up as unavailable, and it takes 20 minutes to log on via RDP or the console, and the screen keeps freezing for one or two minutes at a time.

The Dell OpenManage tools, and online diagnostics show everything is working fine, and there's no related App or System events that show up.

Has anyone any ideas what would cause this, or know of some way we could trace what is the cause?

Thanks.

gerryk

Have you done any testing on your network infrastructure?

swampgas

Duplicated IP addresses can cause weird problems - I'd try to eliminate that as a potential problem as well.

Zirconia

Yes we've checked the network infrastructure, nothing going on there at all. We have other servers here as well on the same backbone switch which are zipping along (these are lower spec machines too), we had to move the accounts and CRM systems to these, and the users say they're running faster than ever.

Also no sign of IP conflicts either.

I notice that when we restart all Exchange services the problems usually go away for a couple of hours and then it slowly grinds to a halt again. Despite that, there are no exchange related errors in the event log either!

gerryk

What size is the Exchange repository?
Have you done any profiling on the storage? What RAID configuration do you have?
Presumably you have eliminated malware as a possibility? Do you log suspicious data traffic? Outbound to port 25 from other than Exchange, for instance.

ressem

I'd ask how long has this occurring, and for a list of hardware and software changes that occurred.

I encountered a similar horrendous slowdown, with no apparent CPU / IO load when an RDX backup drive was temporarily installed on a server.

I used procmon from sysinternals to record what operations the system was carrying out.
Start procmon, right click on the menu and add 'duration' as one of the columns. Most operations should be microseconds, and some IO in 10s of milliseconds.

See whether there's anything taking longer, excessively polling, or repeatedly giving results other than SUCCESS.

In my case the RDX drive had installed a service + driver to poll the eject button which killed the performance of the server. Even disconnecting the device didn't help. Once identified as the problem digging through the registry was required to remove. Always fun on a SBS domain controller.

longshotvalue

Zirconia wrote: »

Having severe performance problems on a Poweredge T310 with SBS 2008 SP2.

We thought it was the RAID controller, and Dell replaced the Motherboard and RAID card earlier this week, but the problems keep coming back. It's so bad that we've had to move some applications off this machine to keep business users happy.

The strange thing is that the CPU only seems to be running about 5 to 10 % max and the physical RAM is only using 60 to 80 % (including SQL and Exchange), with no processes standing out in task manager. But Email is taking up to 50 minutes to process, mapped drives keep coming up as unavailable, and it takes 20 minutes to log on via RDP or the console, and the screen keeps freezing for one or two minutes at a time.

The Dell OpenManage tools, and online diagnostics show everything is working fine, and there's no related App or System events that show up.

Has anyone any ideas what would cause this, or know of some way we could trace what is the cause?

Thanks.

How much ram does the server have, sbs 2008 needs a stupid amount of ram to run properly. Also there need to be a decent amount of free space avalable on the OS drive.

Also is it quicker after a restart and then gets slower. If so its probably a memory leak, which is often a printer driver or something. Try removing any new printer and restart. run the pfmon utility in the windows support pack to see whats using all the resources, it should be obvious as to what has a huge figure beside it and is growning..

rolion

Backup AD+Exchange+Sharepoint,format and reinstall OS(all offline),restore AD in Recovery mode and bet you problem resolved !

swampgas

Zirconia wrote: »

Yes we've checked the network infrastructure, nothing going on there at all. We have other servers here as well on the same backbone switch which are zipping along (these are lower spec machines too), we had to move the accounts and CRM systems to these, and the users say they're running faster than ever.

Also no sign of IP conflicts either.

I notice that when we restart all Exchange services the problems usually go away for a couple of hours and then it slowly grinds to a halt again. Despite that, there are no exchange related errors in the event log either!

Does the raid controller have its write cache enabled? Some need an additional battery pack before it can be enabled.

Zirconia

Thanks for all the replies. Here is one example of what we are seeing with Exchange. Although it says it's likely due to faulty hardware, we have had Dell out and they have replaced the Motherboard and RAID controller and we were seeing these errors before and after they did this. The only other hardware that could be causing this would be the physical disks, and I've run the Dell diags on them and they pass with no problems.

Log Name: Application
Source: ESE
Date: 06/10/2011 10:15:15
Event ID: 510
Task Category: Performance
Level: Warning
Keywords: Classic
User: N/A
Description:
MSExchangeIS (16260) First Storage Group: A request to write to the file "C:\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group\Mailbox Database.edb" at offset 91468193792 (0x000000154bede000) for 24576 (0x00006000) bytes succeeded, but took an abnormally long time (60 seconds) to be serviced by the OS. In addition, 29 other I/O requests to this file have also taken an abnormally long time to be serviced since the last message regarding this problem was posted 447 seconds ago. This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem.

I suspect it's either an OS problem or and Exchange problem (we were getting Database Corruption errors, but I repaired this with ESEUtil, and these errors are no longer appearing).

When I run ExTRA it shows a memory bottleneck issue, pointing at the paged and non-paged pools, but I don't know why as it's not using all the physical RAM (it's at 70 to 80 % usually).

I'm thinking of just exporting the mailboxes to PSTs, deleting the First Storage Group DB and re-creating it from scratch, and then re-importing the mailboxes, as I'm still suspicious after the database corruption errors we had last week.

longshotvalue

Zirconia wrote: »

Thanks for all the replies. Here is one example of what we are seeing with Exchange. Although it says it's likely due to faulty hardware, we have had Dell out and they have replaced the Motherboard and RAID controller and we were seeing these errors before and after they did this. The only other hardware that could be causing this would be the physical disks, and I've run the Dell diags on them and they pass with no problems.

Log Name: Application
Source: ESE
Date: 06/10/2011 10:15:15
Event ID: 510
Task Category: Performance
Level: Warning
Keywords: Classic
User: N/A
Description:
MSExchangeIS (16260) First Storage Group: A request to write to the file "C:\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group\Mailbox Database.edb" at offset 91468193792 (0x000000154bede000) for 24576 (0x00006000) bytes succeeded, but took an abnormally long time (60 seconds) to be serviced by the OS. In addition, 29 other I/O requests to this file have also taken an abnormally long time to be serviced since the last message regarding this problem was posted 447 seconds ago. This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem.

I suspect it's either an OS problem or and Exchange problem (we were getting Database Corruption errors, but I repaired this with ESEUtil, and these errors are no longer appearing).

When I run ExTRA it shows a memory bottleneck issue, pointing at the paged and non-paged pools, but I don't know why as it's not using all the physical RAM (it's at 70 to 80 % usually).

I'm thinking of just exporting the mailboxes to PSTs, deleting the First Storage Group DB and re-creating it from scratch, and then re-importing the mailboxes, as I'm still suspicious after the database corruption errors we had last week.

this might indicate a memory leak issue as i pointed out earlier.. did you install any printers or anything lately..?

longshotvalue

one other thing, have the exclusions on whatever Realtime AV your using been setup as required for SBS?

swampgas

Zirconia wrote: »

[...]
The only other hardware that could be causing this would be the physical disks, and I've run the Dell diags on them and they pass with no problems.
[...]

My own experience is that manufacturer's diagnostics need to be taken with a large grain of salt. Sometimes they find something, sometimes they don't - it often seems to need a particular workload to provoke the weirder hardware problems.

UnCorked

Have you done an offline defrag of your exchange? if not you should try this.

I wouldn't export to pst's first. Can you create a second mailstore and move the mailboxes to that?

If you wanted to rule out you Raid/Hard drives why dont you connect portable hard drive with enough capacity and create a second mailstore on that. move all your mailboxes and see if there are any improvements.

You also show that your mailstore is on your c drive which i presume is the same drive as the OS which is not a good idea so whatever way you make a new mailstore just make sure its on a different drive.

For memory issues you should start by reseating all your RAM - make sure you usea static cable - but many times this resolves memory issues. There are also plenty of online tools to test your memory so go try that too.

Hope this helps!

mickman

UnCorked wrote: »

Have you done an offline defrag of your exchange? if not you should try this.

I wouldn't export to pst's first. Can you create a second mailstore and move the mailboxes to that?

If you wanted to rule out you Raid/Hard drives why dont you connect portable hard drive with enough capacity and create a second mailstore on that. move all your mailboxes and see if there are any improvements.

You also show that your mailstore is on your c drive which i presume is the same drive as the OS which is not a good idea so whatever way you make a new mailstore just make sure its on a different drive.

For memory issues you should start by reseating all your RAM - make sure you usea static cable - but many times this resolves memory issues. There are also plenty of online tools to test your memory so go try that too.

Hope this helps!

Memory at 80% usage is too high. Install more or replace if faulty

Mr. Fancypants

As you can see from the replies it can be a number of things. Take what you can out of the equation. Is it possible to stop the exchange services for a while to see if the issue occurs when exchange is not running?

Nelbert

I had a similar problem before while using server 2008's built-in backup software. The incremental backups would just make the system slow to a crawl within a week or two. Did a full backup, wiped reinstalled and set the backups up as full backups fine ever since but they took 10 hours at a time.