Google Spiders and Private Forums

romarr

Hi,

Can google spiders index private forums ? if not is it possible to allow them access the private pages ?

Thanks in advance

seamus

Generally no, but it depends on how you set up your forums.

The most obvious way to allow spiders to view private pages is to check for those visitors who identify themselves as spiders and tailor the content specifically for them. This however leaves you wide open to spoofing and you private pages aren't so private any more.

romarr

Thanks for your reply Seamus,

when you say private pages aren't so private any more ... will they become available for anyone to view through a google search, or cached once the spiders have been through ?

biko

Private forum are accessed through a login that the spider does not have.
Maybe seamus is referring to public URLs that are "non-disclosed" but can be accessed by anyone once you know the URL.

link8r

romarr wrote: »

Thanks for your reply Seamus,

when you say private pages aren't so private any more ... will they become available for anyone to view through a google search, or cached once the spiders have been through ?

Yes. Once the spiders have indexed your page, then they are cached and anyone can view the cache. Google can technically only access public pages - so if you make an exception for a Googlebot (or MSN etc) then anyone can spoof this and view the content. It's likely that Google would take a dim view of this as they advise against it.

Another thing you could do, is publish a list of private thread titles with the first 150chars and make this page public - but if a user wanted to read more, they'd have to login first or create an account etc. That would probably be the best and the most suitable.

Trojan

seamus wrote: »

The most obvious way to allow spiders to view private pages is to check for those visitors who identify themselves as spiders and tailor the content specifically for them. This however leaves you wide open to spoofing and you private pages aren't so private any more.

Warning. Do not do this unless you're willing to have Google delist the site completely if they realise what you're doing.

Delivering different content to human visitors as opposed to search engine spiders is a well known used spam technique. It's sometimes called with "cloaking" or "doorway pages" (depending on the precise technique) and is most famous for an incident where all of bmw.de was delisted by Google back in 2001 or so. BMW was big enough to buy it's way back in, but most of the rest of us cannot afford that risk.