I've run into the same problem J and T ran into here:
http://www.mail-archive.com/[email protected]/msg05452.html

Namely, malformed robots.txt files out there are being parsed somewhat 
draconianly by WWW::RobotRules.pm.

I've cobbled together a "fixed" version of RobotRules.pm -- how can I get it 
reviewed and ultimately blessed by the LWP community?

http://www.geocities.com/mvaneerde/RobotRules.pm.txt

-- 
Matthew.van.Eerde (at) hbinc.com               805.964.4554 x902
Hispanic Business Inc./HireDiversity.com       Software Engineer

Reply via email to