> So: > > I was looking at a robots.txt file and it had a series of disallow > instructions for various user agents, and then at the bottom was a full > disallow: [...] > Wouldn't this just disallow everyone from everything?
No, it would disallow everyone but aaaaa ... ddddd (with the specified restrictions). >From the spec: " The robot must obey the first record in /robots.txt that contains a User- Agent line whose value contains the name token of the robot as a substring. The name comparisons are case-insensitive. If no such record exists, it should obey the first record with a User-agent line with a "*" value, if present. If no record satisfied either condition, or no records are present at all, access is unlimited." Regards, Martin
