On Fri, 2005-10-14 at 10:37, Gisle Aas wrote: > <[EMAIL PROTECTED]> writes: > > > The problem... if I include a space in my robot's user agent, it > > will fail to recognize robots.txt records targeted to my robot. > > You are not allowed to have space in the user agent name. See section > "3.8 Product Tokens" of RFC 2616 [1]. Isn't it an option to just > rename your spider to something that follows the spec?
Perhaps it would help if WWW::RobotRules were to warn/die when setting an agent with a space in? An excellent message would be "RFC2616 forbids spaces in an agent's names". -Nigel
