On Fri, 26 Mar 2010, Jarkko Hietaniemi wrote:
The total size is not the problem. The number of files is. Vanilla
rsync is horribly inefficient (not the protocol, which is genius, mind)
because a client coming by and asking for updates basically ends up
requiring the moral equivalent of
"find . -type f -print". Let me repeat that: each client. Not fun.
Why use rsync, then? Why not have checkpointed logs on cpan with
additions/removals logged by date so you can roll forward on the client,
processing only those files? It would be trivial to set up and a lot more
efficient.
--Arthur Corliss
Live Free or Die
