Re: rsync slowness
I got hit by that bug again, and this time I took the time to try to create a minimal recipe. It turns out that the culprit was the -y flag, which I had been using to save some bandwidth, but which doesn't work well when there are "too many" files in a directory. A simple reproducer (which does not exactly reproduce the issue described below, but seems close enough): #!/bin/bash mkdir -p src declare -i i=0 j=0 while : do echo $RANDOM$RANDOM$RANDOM > src/$(printf %09d $j).$RANDOM$RANDOM$RANDOM.foobarbaz.$RANDOM.$RANDOM.$RANDOM let i++ let j+=100 (($i == 10)) && break done cp -a src dst rm -f dst/00[7-9]* rsync -avy src/ dst/ After copying about 900 files, rsync will hang during a couple of minutes, before copying another batch of about 900 files. During the next iterations, the batches become smaller: about 200 files. After each batch, rsync waits during a couple of minutes. I stopped the process after two hours, and only 17500 files had been copied. By comparison, rsync -av src/ dst/ only takes 2 seconds to copy the 3 files. I removed the -y option from my scripts, but perhaps how -y works in directories with many files could be improved in one way or another? I've definitely not seen that. If you can produce a working example and tar it up for us to look at, that might be interesting/useful. Just to check, though: you do not have --checksum/-c on, right? I finally take the time to report an rsync slowness pattern that I've been seeing for years. Assuming: a directory with many (> 20K) files, for example a maildir, on the sending side, and a partial copy of that directory on the receiving side, with "enough" missing missing (say 5K), then the receiving side will do the following: it will take about one minute to start transferring the missing files, and will apparently hang about one minute every 200 files or so. After a few (about 10 or 20) iterations, the receiving side apparently hangs. On the receiving side rsync is using 100% CPU, on the sending side not more than a few percents. In case it matters, both sides are using ext4 filesystems, and Debian GNU/Linux. I tried --msgs2stderr -M--msgs2stderr, but that does not print any error message, I also tried to disable compression, to use --whole-file, to use --no-inc-recursive, ..., to no avail. Any hints? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync slowness
I've definitely not seen that. If you can produce a working example and tar it up for us to look at, that might be interesting/useful. Just to check, though: you do not have --checksum/-c on, right? On Sat, Aug 06, 2022 at 05:54:12PM +, Gregory Heytings via rsync wrote: > > I finally take the time to report an rsync slowness pattern that I've been > seeing for years. > > Assuming: > > a directory with many (> 20K) files, for example a maildir, on the sending > side, and > > a partial copy of that directory on the receiving side, with "enough" > missing missing (say 5K), > > then the receiving side will do the following: it will take about one minute > to start transferring the missing files, and will apparently hang about one > minute every 200 files or so. After a few (about 10 or 20) iterations, the > receiving side apparently hangs. > > On the receiving side rsync is using 100% CPU, on the sending side not more > than a few percents. In case it matters, both sides are using ext4 > filesystems, and Debian GNU/Linux. > > I tried --msgs2stderr -M--msgs2stderr, but that does not print any error > message, I also tried to disable compression, to use --whole-file, to use > --no-inc-recursive, ..., to no avail. > > Any hints? > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync slowness
I finally take the time to report an rsync slowness pattern that I've been seeing for years. Assuming: a directory with many (> 20K) files, for example a maildir, on the sending side, and a partial copy of that directory on the receiving side, with "enough" missing missing (say 5K), then the receiving side will do the following: it will take about one minute to start transferring the missing files, and will apparently hang about one minute every 200 files or so. After a few (about 10 or 20) iterations, the receiving side apparently hangs. On the receiving side rsync is using 100% CPU, on the sending side not more than a few percents. In case it matters, both sides are using ext4 filesystems, and Debian GNU/Linux. I tried --msgs2stderr -M--msgs2stderr, but that does not print any error message, I also tried to disable compression, to use --whole-file, to use --no-inc-recursive, ..., to no avail. Any hints? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync slowness issues with solaris 10
After installing rsync 2.6.8 on Solaris 10, I have noticed that it is extremely slow compared to older versions of rsync in solaris 8. Not sure if this is related to Solaris 10 or rsync, but please post your experiences with rsync in solaris 10. Have you noticed any slowness/performance issues ? Thanks and Regards. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
