> Mhh. I have not looked into this. But is not every desktop aggregator > a robot?
Henry: Depends on who you ask. (See the Newsmonster debates from a couple years ago.) Right now, I obey all wildcard and/or my-user-agent-specific directives I find in robots.txt. If I were writing a desktop app, I would ignore any wildcard directives, and obey the specific stuff.. -- Roger Benningfield http://admin.support.journurl.com/
