There are no wildcards in /robots.txt, only path prefixes and user-agent names. There is one special user-agent, "*", which means "all". I can't think of any good reason to always ignore the disallows for *.
I guess it is OK to implement the parts of a spec that you want. Just don't answer "yes" when someone asks if you honor robots.txt. A lot of spiders allow the admin to override /robots.txt for specific sites, or better, for specific URLs. wunder --On August 25, 2005 11:47:18 PM -0500 "Roger B." <[EMAIL PROTECTED]> wrote: > > Bob: It's one thing to ignore a wildcard rule in robots.txt. I don't > think its a good idea, but I can at least see a valid argument for it. > However, if I put something like: > > User-agent: PubSub > Disallow: / > > ...in my robots.txt and you ignore it, then you very much belong on > the Bad List. > > -- > Roger Benningfield > > -- Walter Underwood Principal Software Architect, Verity
