Antone Roundy wrote:
> I'm with Bob on this. If a person publishes a feed without limiting
> access to it, they either don't know what they're doing, or they're
> EXPECTING it to be polled on a regular basis. As long as PubSub
> doesn't poll too fast, the publisher is getting exactly what they
> should be expecting.
Because PubSub aggregates content for thousands of others, it
removes significant bandwidth load from publishers' sites. We only read a
feed from a site in response to an explicit ping from that site or, for
those sites that don't ping, we poll them on a scheduled basis. In fact, we
read scheduled, non-pinging feeds less frequently than most desktop systems
would. No one can claim that we do anything but reduce the load on
publishers systems. It should also be noted that we support gzip
compression, RFC3229+Feed, conditional-gets, etc. and thus do all the things
necessary to reduce our load on publishers sites in the event that we
actually do fetch data from them. This is a good thing and not something
that robots.txt was intended to prevent.
bob wyman
