--On August 26, 2005 9:51:10 AM -0700 James M Snell <[EMAIL PROTECTED]> wrote:
> Add a new link rel="readers" whose href points to a robots.txt-like file that > either allows or disallows the aggregator for specific URI's and establishes > polling rate preferences > > User-agent: {aggregator-ua} > Origin: {ip-address} > Allow: {uri} > Disallow: {uri} > Frequency: {rate} [{penalty}] > Max-Requests: {num-requests} {period} [{penalty}] No, on several counts. 1. Big, scalable spiders don't work like that. They don't do aggregate frequencies or rates. They may have independent crawlers visiting the same host. Yes, they try to be good citizens, but you can't force WWW search folk to redesign their spiders. 2. Frequencies and rates don't work well with either HTTP caching or with publishing schedules. Things are much cleaner with a single model (max-age and/or expires). 3. This is trying to be a remote-control for spiders instead of describing some characteristic of the content. We've rejected the remote control approach in Atom. 4. What happens when there are conflicting specs in this file, in robots.txt, and in a Google Sitemap? 5. Specifying all this detail is pointless if the spider ignores it. You still need to have enforceable rate controls in your webserver to handle busted or bad citizen robots. 6. Finally, this sort of thing has been proposed a few times and never caught on. By itself, that is a weak argument, but I think the causes are pretty strong (above). There are some proprietary extensions to robots.txt: Yahoo crawl-delay: <http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html> Google wildcard disallows: <http://www.google.com/remove.html#images> It looks like MSNbot does crawl-delay and an extension-only wildcard: <http://search.msn.com/docs/siteowner.aspx?t=SEARCH_WEBMASTER_REF_RestrictAccessToSite.htm> wunder -- Walter Underwood Principal Software Architect, Verity