Gentoo's portage has a neat feature that may work for the ratelimiting for the
rulesets. When emerge sync is run, a single small file is retrieved (couple
of bytes if I remember right), and this file is apparently compared to the
on-disk copy. If the contents are different (I think the file has a unix
timestamp in it), the full sync occurs, updating the portage tree with rsync.
If the contents are the same, the sync exits cleanly.
HTTP (which RDJ currently uses) already has a mechanism for this. The client can send an If-Modified-Since header, or compare an Etag from its cache, and the server will check, say "no, this file hasn't changed since you last downloaded it," and issue a "304 Not Modified" response consisting of only an HTTP header and no data. Just about every browser in existence uses this method to determine that it can load a file from its cache.
If RDJ is updated to handle this, as Chris(?) suggested, half the problem is solved. Even people downloading every two minutes are only grabbing a few dozen bytes of headers instead of several K of data.
Kelson Vibber
SpeedGate Communications <www.speed.net>
