[EMAIL PROTECTED] writes:

> If you spider several sites and one of them has a broken robots.txt file you
> can't tell which one since the warning doesn't tell you.

This will be better in 5.801.  I've applied a variation of Bill
Moseley's suggested patch for the same problem.

> Around line 73 of RobotRules.pm
> change:
>       warn "RobotRules: Disallow without preceding User-agent\n";
> to
>       # [EMAIL PROTECTED]: added $netloc
>         warn "RobotRules: $netloc Disallow without preceding User-agent\n";

Reply via email to