[EMAIL PROTECTED] writes: > If you spider several sites and one of them has a broken robots.txt file you > can't tell which one since the warning doesn't tell you.
This will be better in 5.801. I've applied a variation of Bill Moseley's suggested patch for the same problem. > Around line 73 of RobotRules.pm > change: > warn "RobotRules: Disallow without preceding User-agent\n"; > to > # [EMAIL PROTECTED]: added $netloc > warn "RobotRules: $netloc Disallow without preceding User-agent\n";
