On Sat, 18 Dec 2004, J and T wrote: > I recently came accross something that didn't seem right to me. I'm using > "WWW::RobotRules::AnyDBM_File", but the below sample script will return the > same thing. > > The URL I tested is: > http://www.midwestoffroad.com/ > > The robots.txt reads: > > User-agent: * > Disallow: admin.php > Disallow: error.php > Disallow: /admin/ > Disallow: /images/ > Disallow: /includes/ > Disallow: /themes/ > Disallow: /blocks/ > Disallow: /modules/ > Disallow: /language/ > User-agent: Baidu > Disallow: / > > RobotRules returns that the URL is denied by robots.txt which should not be > the case.
That's debatable. The robots.txt file is invalid according to <http://www.robotstxt.org/wc/norobots.html>: The file consists of one or more records separated by one or more blank lines [...] The record starts with one or more User-agent lines, followed by one or more Disallow lines So "Disallow: /" is part of the record begun with "User-agent: *". It's reasonable to ignore the misplaced "User-agent: Baidu" or to treat it as though it were placed at the start of the record. -- Liam Quinn
