Instant Google spider

corblimey

There's a certain other Irish site I frequent (it's a forum site dealing mostly with financial issues) and I've noticed that when I post a new thread, and then google for that thread title, it's almost always right there. Ie, as soon as it's been posted, it's been spidered by Google.

Could this be something they're doing on their site to have Google refresh instantly? It doesn't seem to happen with boards.ie posts. Anyone else seen this?

jebuz

Maybe they have a cron job set up to generate and submit a sitmap to google daily, or hourly?

Evil Phil

Boards.ie posts depend on the forum in my experience. But the website I think your referring too is probably getting indexed by Google at a very high rate because of the relevancy to the website topic all the posts have, and the quality of the information. That's Google's choice however, not the sites.

tricky D

2nd result

http://www.google.ie/search?q=Instant+Google+spider

within an hour

QED

Evil Phil

Yeah, this forum gets indexed almost instantly.

Gordon

Heh

Evil Phil

Too small, I can't read your funneh!

corblimey

tricky D wrote: »

2nd result

http://www.google.ie/search?q=Instant+Google+spider

within an hour

QED

You're right, I did do a quick search immediately after posting just to see, but it wasn't there. But yeah, that's still pretty quick. I guess it makes sense that it's up to Google and its parameters than letting the site owners determine it. Google is usually noisy enough without the extra pressure.

Gordon

Evil Phil wrote: »

Too small, I can't read your funneh!

Ah, it's the image you get with googles new preview magnifying glass thing. Although looking back at it, it generated the preview when I rolled over to get it, it didn't generate it when it crawled the page originally (I think).

Liam Byrne

corblimey wrote: »

I guess it makes sense that it's up to Google and its parameters than letting the site owners determine it.

Google is like the local delivery guy; if he calls once a week and you have nothing for him, he'll eventually only call once a fortnight, reducing to once a month, etc, if the pattern goes that way. Why waste resources if more-often-than-not you get nothing new ?

On the other hand, if you have a significant amount every week, he'll consider calling more often.

I have one site that's updated about twice a year and it takes a month or more for changes to register; other sites' changes show up within a few days.

Regular updates of relevant content is your best solution.

NeverSayDie

Not to mention that in their quest to gather more data than God, Google are regularly updating their tech to pull stuff in faster - a lot of the recent improvements probably came from this upgrade;
http://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html

A 100 million gigabyte index...