On 14 Oct 2005 10:06:21 -0700, Gisle Aas <[EMAIL PROTECTED]> wrote:
> Those are pretty unreadable. At least try to use the "Follow TCP
> stream" function of ethereal so it is possible to read what the data
> passed between the peers are.
You are right, sorry. So I have replaced the 2 files with a
Fabio Marzocca <[EMAIL PROTECTED]> writes:
> I have put here: http://socrates.homedns.org/lwp/
> the two complete ethereal sessions (one from browser and the other
> from LWP).
Those are pretty unreadable. At least try to use the "Follow TCP
stream" function of ethereal so it is possible to read
I have put here: http://socrates.homedns.org/lwp/
the two complete ethereal sessions (one from browser and the other
from LWP). As you can see the browser reaches the correct page, while
LWP is diverted to an error page...
Gisle Aas wrote:
> <[EMAIL PROTECTED]> writes:
>
>> The problem... if I include a space in my robot's user agent, it
>> will fail to recognize robots.txt records targeted to my robot.
>
> You are not allowed to have space in the user agent name. See section
> "3.8 Product Tokens" of RFC 2616 [1]
Nigel Horne writes:
> Perhaps it would help if WWW::RobotRules were to warn/die when setting
> an agent with a space in? An excellent message would be "RFC2616 forbids
> spaces in an agent's names".
Impossible, because the whole user agent string *is* allowed to have
spaces in it. It's just th
On Fri, 2005-10-14 at 10:37, Gisle Aas wrote:
> <[EMAIL PROTECTED]> writes:
>
> > The problem... if I include a space in my robot's user agent, it
> > will fail to recognize robots.txt records targeted to my robot.
>
> You are not allowed to have space in the user agent name. See section
> "3.8
<[EMAIL PROTECTED]> writes:
> The problem... if I include a space in my robot's user agent, it
> will fail to recognize robots.txt records targeted to my robot.
You are not allowed to have space in the user agent name. See section
"3.8 Product Tokens" of RFC 2616 [1]. Isn't it an option to just
