--On Wednesday, May 29, 2002 04:49:22 PM +0100 Alan Perkins <[EMAIL PROTECTED]> wrote: > > Are you suggesting a robot is checking that string against its UA???
Yes. The user-agent line is probably an in-joke from a Perl programmer. Also, I expect that it is very old. The specs seem to be in chronological order, with Excite (ArchiText) and Infoseek very early, and Google at the end. As for your underlying question, I expect that you won't get a useful answer. People at search engines really don't talk about how they detect spammers and hostile bots. Once a technique is public, it is dead. robots.txt is for cooperating robots. If a robot is not going to cooperate, then robots.txt is just a file. If a bot lies about its identity or intentions, that is clearly not "cooperating". Note that it is possible to have benign non-cooperating bots, like a checker for handicapped accessbility that sends a particular browser user-agent line (see http://www.cast.org/bobby/). wunder -- Walter Underwood [EMAIL PROTECTED] Senior Staff Engineer, Inktomi http://www.inktomi.com/
