Googlebot eating bandwidth

squibs

I have a site that does about 400Mb of traffic a month at the mo, but the googlebot is eating a further 1.2 GB. MSN and Yahoo bots combined only consume 10Mb. I've got my robots.txt set to keep the bots out of the admin area and the phpbb forum. It's a dynamic mysql/php site. Obviously I want to keep my rankings up but would like the bot to calm down a bit. Any ideas on why it's spidering me so often or tips to reduce the traffic?

aidan_walsh

Try using Google Sitemap?

seamus

Is there anything in particular which would account for big bandwidth - images, videos, etc?

squibs

Nope - not really. It's mainly text with mainly small images and the odd few 640*480 jpgs.

I had a look at sitemaps a while back but it seemed not to be well set up for database driven sites. I might be wrong in this.

mneylon

You might want to check which URLs are being hit the most.. I was doing a couple of megs constant earlier this year as I had a few Ubuntu packages on my blog

vito

Yep probably getting stuck in a loop somewhere. Usually caused by the way you are processing dynamic URL requests.

Things like dynamically generated calendars (where month/year is set through URL) can also cause issues as Googlebot doesn;t seem to know when to stop sometimes.

squibs

Thanks. I'll have a look at the logs. Do you guys use any particular tools for browsing logs? AWstats usually works for me, but it won't work for this. Regular text editor?

How might I search for a loop - multiple requests for the same page in a short period of time?

mneylon

Awstats will show you the top URLs requested

squibs

But it filters out the bot requests, right?

mneylon

squibs wrote:

But it filters out the bot requests, right?

Not really.

Have a closer look