Antone Roundy wrote: > I'm with Bob on this. If a person publishes a feed without limiting > access to it, they either don't know what they're doing, or they're > EXPECTING it to be polled on a regular basis. As long as PubSub > doesn't poll too fast, the publisher is getting exactly what they > should be expecting.
Because PubSub aggregates content for thousands of others, it removes significant bandwidth load from publishers' sites. We only read a feed from a site in response to an explicit ping from that site or, for those sites that don't ping, we poll them on a scheduled basis. In fact, we read scheduled, non-pinging feeds less frequently than most desktop systems would. No one can claim that we do anything but reduce the load on publishers systems. It should also be noted that we support gzip compression, RFC3229+Feed, conditional-gets, etc. and thus do all the things necessary to reduce our load on publishers sites in the event that we actually do fetch data from them. This is a good thing and not something that robots.txt was intended to prevent. bob wyman