robots.txt will not do what you want it to.
Just sniff for those robots' User-Agents (Google, MSN and Yahoo all publish
their UA strings on their websites, AFAIK) and send different content if
it's one of those.
they will hammer you for it eventually - AFAICT all major SEs send out their
spiders occasionally with faked user-agent strings - to catch out crap like
this.
google adsense won't. I explicity asked them about this. Well, what I
asked was that if I had a password protected area, could I allow them
access to spider the content so that normal users could see the ads. I
told them the layout would be different, but the content the same.
They said that was fine.
2cents.
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
