Antone Roundy wrote:
> How could this all be related to aggregators that accept feed URL
> submissions?
My impression has always been that robots.txt was intended to stop
robots that crawl a site (i.e. they read one page, extract the URLs from it
and then read those pages). I don't believe robots.txt is intended to stop
processes that simply fetch one or more specific URLs with known names.
At PubSub we *never* "crawl" to discover feed URLs. The only feeds
we know about are:
1. Feeds that have announced their presence with a ping
2. Feeds that have been announced to us via a FeedMesh message.
3. Feeds that have been manually submitted to us via our "add-feed"
page.
We don't crawl.
I do not think we qualify as a "robot" in the sense that is relevant
to robots.txt. It would appear that Walter Underwood of Verity would agree
with me since he says in his recent post that: "I would call desktop clients
"clients" not "robots". The distinction is how they add feeds to the polling
list. Clients add them because of human decisions. Robots discover them
mechanically and add them." If Walter is correct, then he must agree with me
that robots.txt does not apply to PubSub! (and, we should not be on his
"bad" list.... Walter? Please take us off the list...)
bob wyman
