https://bugzilla.samba.org/show_bug.cgi?id=10678
Summary: performance problem with lots of hard links? Product: rsync Version: 3.0.6 Platform: All OS/Version: All Status: NEW Severity: normal Priority: P5 Component: core AssignedTo: way...@samba.org ReportedBy: d...@yost.com QAContact: rsync...@samba.org We use rsync to do backups. Each backed up file is hard linked to the corresponding file from the day before unless the file differs. On the backup disk there are 3 directory trees, one per backed-up machine, each with 45 directory trees. The backup disk has about 1TB of backup content. df says each disk being backed up is about 90GB. At the moment, we're copying an old backup disk to a new larger backup disk. It's taking forever and seems to be slowing down. After 8 hours it's copied only 37%. Maybe this is normal. Don't know. Thinking that perhaps a later version of rsync would do better, we started again, overwriting what was already copied. 8 hours later rsync hasn't started writing any new files yet. CPU time usage is about 1/10 of real time, so rsync seems I/O bound. It's possible that rsync is thrashing in the L2 cache, but there is a pretty big L3 cache on this Xeon chip. I wonder if you have ever profiled a case like this to see if there is anything that could be better in the algorithm that manages huge numbers of hard links. Perhaps yet another option flag could tell rsync that it's working on this kind of workload, and rsync could throw away bookkeeping data for hard links farther back than yesterday's backup tree. F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND 4 0 190742 152692 20 0 189568 3728 poll_s S+ pts/5 0:00 | \_ sudo /usr/local/rsync/3.1.1/bin/rsync -aSHAX /oldbackup/ /backup 4 0 190756 190742 20 0 937948 760856 poll_s S+ pts/5 18:45 | \_ /usr/local/rsync/3.1.1/bin/rsync -aSHAX /oldbackup/ /backup 5 0 190757 190756 20 0 1369872 1202448 sync_b D+ pts/5 19:44 | \_ /usr/local/rsync/3.1.1/bin/rsync -aSHAX /oldbackup/ /backup 1 0 190758 190757 20 0 792392 605480 poll_s S+ pts/5 0:49 | \_ /usr/local/rsync/3.1.1/bin/rsync -aSHAX /oldbackup/ /backup Filesystem 1K-blocks Used Available Use% Mounted_on /dev/sdb3 1,906,881,372 377,934,500 1,432,082,872 21% /backup /dev/sdc2 1,442,142,772 986,617,308 382,268,788 73% /oldbackup -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html