>This is wrong. > >The spec [1] doesn't say anything about order being significant. One of the >examples in the spec shows the robot "cybermapper" matching a user-agent >line which is after a "*" entry. > >A robot which implements "first match" is not following the spec. > >The spec text in the proposed RFC [2] (never adopted) is more specific >about this: > > These name tokens are used in User-agent lines in /robots.txt to > identify to which specific robots the record applies. The robot > must obey the first record in /robots.txt that contains a User- > Agent line whose value contains the name token of the robot as a > substring. The name comparisons are case-insensitive. If no such > record exists, it should obey the first record with a User-agent > line with a "*" value, if present. > >If you want to protect against this particular variety of robot >mis-coding, go ahead. I don't know if there is any evidence of this >behavior in the wild, so it might be a waste of time.
It may be wrong according to the specifications, but that's what seemed to work when I was trying to get rid of a couple of pesky spiders several years ago. With 'cut and paste', it isn't going to take a second to move it to the end of the file. Try it. It just might work for you. Fred Atkinson _______________________________________________ Robots mailing list Robots@mccmedia.com http://www.mccmedia.com/mailman/listinfo/robots