I have a sensor collector system (very low-powered slow ARM cpu), and another system which daily pulls the data files from it for processing. There are about 1000 new files each day. As part of the processing it decides that certain of the files are of no interest, and adds them to an exclude file, which is used in future rsyncs. No files are ever deleted from the source system, just the receiver system. All done in cron jobs at night.
After about 18 months, there are about 350,000 files and the exclude list has about 72,000 filenames. I recently ran the pulling script manually and thought the system must have died. Rsync took almost 3 hours. Trying to narrow down the problem, I removed the "--exclude-file=" option from the rsync command -- and it took only 16 minutes -- including the time to transfer the 72,000 files that are of no interest. (con't)
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
