Gisle Aas wrote:
> <[EMAIL PROTECTED]> writes:
>
>> The problem... if I include a space in my robot's user agent, it
>> will fail to recognize robots.txt records targeted to my robot.
>
> You are not allowed to have space in the user agent name. See section
> "3.8 Product Tokens" of RFC 2616 [1]
Nigel Horne writes:
> Perhaps it would help if WWW::RobotRules were to warn/die when setting
> an agent with a space in? An excellent message would be "RFC2616 forbids
> spaces in an agent's names".
Impossible, because the whole user agent string *is* allowed to have
spaces in it. It's just th
On Fri, 2005-10-14 at 10:37, Gisle Aas wrote:
> <[EMAIL PROTECTED]> writes:
>
> > The problem... if I include a space in my robot's user agent, it
> > will fail to recognize robots.txt records targeted to my robot.
>
> You are not allowed to have space in the user agent name. See section
> "3.8
<[EMAIL PROTECTED]> writes:
> The problem... if I include a space in my robot's user agent, it
> will fail to recognize robots.txt records targeted to my robot.
You are not allowed to have space in the user agent name. See section
"3.8 Product Tokens" of RFC 2616 [1]. Isn't it an option to just
The problem... if I include a space in my robot's user agent, it will fail to
recognize robots.txt records targeted to my robot.
My robot's user agent:
Hispanic Business Inc. Spider/1.0
Robots.txt file:
User-agent: Hispanic Business Inc. Spider
Disallow:
User-agent: *
Disallow: /
My robot will
