‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Sunday 28 June 2020 13:58, Matthias Schniedermeyer <m...@citd.de> wrote:
> On 27.06.2020 11:22, Rupert Gallagher via rsync wrote: > > > On Friday 26 June 2020 21:58, Rupert Gallagher via rsync > > rsync@lists.samba.org wrote: > > > > > Hello, > > > As disks are slow and rsync reads and writes so much that for the bus > > > this is the equivalent of context switching galore, would it be possible > > > to use RAM as a buffer? Say, you have 10GB of spare RAM, rsync uses the > > > bus to its peak for reading 10GB, then again for writing it down. This > > > would be more efficient than lot of small read/write operations. > > > Thank you > > > > Current task: rsync 752 GB > > source disk > > Writing speed: 77 MB/s > > Reading speed: 97 MB/s > > target disk > > Writing speed: 117 MB/s > > Reading speed: 99 MB/s > > Actual time: 380 min (6.3 hours) to copy 648 GB > > Actual speed: 28 MB/s (648/380 = 1.7 GB min =~ 1700MB min / 60 min = 28MB > > sec) > > Unfortunatly you left out every other detail. > Complete rsync commandline? /usr/local/bin/rsync --recursive --links --times --modify-window=1 --devices --specials --update --owner --group --perms --delete --delete-before --delete-excluded --exclude-from=/etc/excluded_from_backup.conf --numeric-ids --outbuf=Block --inplace --link-dest=/backup/latest/ /archive /backup > What hardware? (From the numbers it is only clear that you seem to talk about > HDDs.) > What HDDs? source: ST2000NX0403 sata hdd Writing speed : 117 MB/s Reading speed : 99 MB/s destination: ST5000LM000-2AN1 sata hdd Writing speed : 74 MB/s Reading speed : 89 MB/s > What computer? (Laptop? Desktop? Server? Raspberry Pi? Age?) Supermicro A2SDi-4C-HLN4F, newish > What "Buses"? ( a) Any modern "bus" is NOT saturated by those numbers. b) All > modern "buses" (Except USB, to some degree) are P2P, you can't even connect 2 > devices to the same bus. (Except USB, but there are usually several > controllers so you don't have to use same bus).) Supermicro CSE-M14TQC 4xSAS/SATA bay, connected with a CBL-SAST-0616(50cm) Mini-SAS HD to 4 SATA cable. The CSE receives the 4 sata cables, the mini-sas end is plugged on the main board. > With or without networking involved? no network involved > What Filesystem? What mount-options? FFS2 rw,nodev,nosuid,softdep,noatime > AVG Filesize? Directory structure? Fragmentation? mixed > Are you: > > - Copying 1 752GB File(*) no > - 752 1GB Files no > - 10.000.000 78KB Files inside a directory-Tree with 10.000.000 Million > subdirectories no > - 100.000.000 7.8KB Files in 100.000.000 Million subdirectories. no > Personally my usual copying-speed for the "from local XFS formated HDD to > local XFS formated USB-3-HDD"-case is about 350GB/Hour (or about 100MB/s.), > and that has been the case for years. I shall be so lucky... > *: > Even copying 2 different 752GB files can be quite a different monster > depending on file-fragmentation. > Copying a heavily fragmented file (from a HDD) is slow, even when it is a > single big file. > HDDs are very "sensitive" to sub-optimal usage, every 'seek' kills of > several ms where 0KB can be read/written. > The worst-case performance (or IOPS) of a HDD is counted in KB/s(!!), if > you have to seek after every sector read or written. > > -- > > Matthias > -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
