On Sat, 18 Dec 2004, J and T wrote:

> I recently came accross something that didn't seem right to me. I'm using 
> "WWW::RobotRules::AnyDBM_File", but the below sample script will return the 
> same thing.
> 
> The URL I tested is:
> http://www.midwestoffroad.com/
> 
> The robots.txt reads:
> 
> User-agent: *
> Disallow: admin.php
> Disallow: error.php
> Disallow: /admin/
> Disallow: /images/
> Disallow: /includes/
> Disallow: /themes/
> Disallow: /blocks/
> Disallow: /modules/
> Disallow: /language/
> User-agent: Baidu
> Disallow: /
> 
> RobotRules returns that the URL is denied by robots.txt which should not be 
> the case.

That's debatable.  The robots.txt file is invalid according to 
<http://www.robotstxt.org/wc/norobots.html>:

    The file consists of one or more records separated by one or more 
    blank lines
    [...]
    The record starts with one or more User-agent lines, followed by one 
    or more Disallow lines

So "Disallow: /" is part of the record begun with "User-agent: *".  It's
reasonable to ignore the misplaced "User-agent: Baidu" or to treat it as
though it were placed at the start of the record.

-- 
Liam Quinn

Reply via email to