If you spider several sites and one of them has a broken robots.txt file you can't tell which one since the warning doesn't tell you.
Around line 73 of RobotRules.pm
change:
warn "RobotRules: Disallow without preceding User-agent\n";
to
# [EMAIL PROTECTED]: added $netloc
warn "RobotRules: $netloc Disallow without preceding User-agent\n";
-Nigel
--
Nigel Horne. Arranger, Composer, Typesetter.
NJH Music, Barnsley, UK. ICQ#20252325
[EMAIL PROTECTED] http://www.bandsman.co.uk
