Is rsync speed limited to 160MB/sec read speeds due to the chunk read size? Or, are we seeing just a coincidence and the limitation is somewhere else?
Rsync (single process) is used to locally copy files from one raid partition to another and we noticed rsync's read speed is limited. Whether I setup an rsync between two raid disks or two ram drives, rsync reads top out at around 160MB/sec. Writes are fine as ZFS easily sustains 2GB/sec, we just see the rsync read speeds are limited. All rsyncs are on the machine itself and do not cross the 10Gb network. We have five(5) raid boxes in the following configuration. Raid card caching is disabled which allows direct disk access. Intel E5-2630 64GB RAM DDR3 2400MHz LSI 9271-8i (caching disabled) 24x WD 4TB WD2001FYYG (2x 12 drive raid 6 arrays) Ubuntu 12.04.2 LTS (patched as of today) ZFS filesystem (v28 pool v5000, compression off, checksums on) The following tests were run on all five boxes to rule out hardware issues and tests were started when the machine was at a load of zero(0). There are no files on the destination drive when the rsync is started. While copies are running the machine's CPU usage never rises about 10% (90% idle). Test1: ram drive to ram drive rsync: setup the ram disks: sudo mount -t tmpfs -o size=1024M tmpfs /tmp/ram1/ sudo mount -t tmpfs -o size=1024M tmpfs /tmp/ram2/ rsync -avP /tmp/ram1/ /tmp/ram2/ Fedora-Live-Desktop.iso 997195776 100% 158.44MB/s 0:00:05 (xfer#1, to-check=0/2) Test 2: rsync the iso from one raid disk to another raid disk: rsync -avP /data1/ /data2/ Fedora-Live-Desktop.iso 997195776 100% 166.46MB/s 0:00:05 (xfer#1, to-check=0/2) Test 3: dd with an increasing chuck size was able to read from ram faster then rsync: dd if=/tmp/ram1/Fedora-Live-Desktop.iso of=/dev/null bs=<value> bs=512 1.2 GB/s bs=1K 2.2 GB/s bs=2K 3.5 GB/s bs=4K 5.2 GB/s bs=8K 6.8 GB/s bs=64K 7.0 GB/s Test 4: The reads from the raid with increasing read chunk size resulted in better speeds. Notice the block size of 512 bytes is 164MB/sec which is similar to what we are seeing with rsync. dd if=/data1/Fedora-Live-Desktop.iso of=/dev/null bs=<value> bs=128 42 MB/s bs=256 84 MB/s bs=512 164 MB/s (dd's default bs size) bs=1k 276 MB/s bs=2k 567 MB/s bs=4k 717 MB/s bs=8k 938 MB/s bs=16k 988 MB/s bs=32k 1.1 GB/s bs=64k 1.1 GB/s bs=128k 1.2 GB/s <-- we want to be here bs=256K 1.2 GB/s bs=512K 1.2 GB/s Test 5: Finally, a dd write of the iso to the raid partiitons from a ram drive to show the raids can write at gigabyte speeds: dd if=/tmp/ram1/Fedora-Live-Desktop.iso of=/data1/iso bs=<value> bs=512 70.6 MB/s bs=1K 137 MB/s bs=2K 271 MB/s bs=4K 472 MB/s bs=8K 804 MB/s bs=16K 1.2 GB/s bs=32K 1.7 GB/s bs=64K 2.0 GB/s bs=128k 2.9 GB/s bs=256K 3.0 GB/s bs=512K 3.0 GB/s What we are seeing is a speed limitation and _not_ a bug. Rsync works perfectly fine otherwise. We would be happy to test out any patches. Thanks for your time. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
