[Bug 11166] running with -vvv causes a hang
https://bugzilla.samba.org/show_bug.cgi?id=11166 --- Comment #9 from Michal Ruprich --- Hi, current rsync still hits this problem when transferring huge amounts of files. I am talking tens of thousands. I tried this with some 700k files and it got stuck at the same place as before. What worked for me was to create a bigger buffer from the beginning. I set the IO_BUFFER_SIZE from 32*1024 to 128*1024. I know that this does not make sense for every transfer but maybe it wouldn't be a bad idea to create a new option that could increase the size of the buffer from the beginning. If I have a usecase where I know I am going to be transferring a lot of files and for some reason I need the -vvv or more, I could just set the message buffer to a bigger value. Or do you have an idea how to re-write the logging completely? Thanks for any thoughts on this. Michal -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: RAM speedup
You may need a lot of ram. Take a look at how much RAM ZFS needs to operate well. Henri > On 27/06/2020, at 7:58 AM, Rupert Gallagher via rsync > wrote: > > Hello, > > As disks are slow and rsync reads and writes so much that for the bus this is > the equivalent of context switching galore, would it be possible to use RAM > as a buffer? Say, you have 10GB of spare RAM, rsync uses the bus to its peak > for reading 10GB, then again for writing it down. This would be more > efficient than lot of small read/write operations. > > Thank you > > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: RAM speedup
Not sure if it is useful. Will put a plug in for a rsync hard based backup system primarily focused on macOS. If it is speed you are after, then this is probably not the right / helpful tool. But feel free to pick out what ever is useful for you : http://www.lbackup.org Henri > On 29/06/2020, at 7:29 AM, Matthias Schniedermeyer via rsync > wrote: > > On 28.06.2020 16:46, Rupert Gallagher wrote: >> ? Original Message ? >> On Sunday 28 June 2020 13:58, Matthias Schniedermeyer wrote: >> >> destination: >> ST5000LM000-2AN1 sata hdd >> Writing speed : 74 MB/s >> Reading speed : 89 MB/s > > And this HDD is a SMR model(*) on top of beeing a 4k sector model emulating > 512 byte sectors. > So alignment needs to be correct and the filesystems must use 4k sectors. > > This HDD is NOT suitable for beeing used for small files and a hardlink-farm. > > SMR HDDs only reach best performance if used "like a tape drive" with large > and linear writing. > > I use SMR HDDs myself and only use them for "really large" files (>500MB per > file), otherwise they perform very poorly. > I also align my partitions correctly and set XFS to "4k sector"-size. > Which nowadaya means: I use 4k sectors for anything. As i only have been > using 4k sector HDDs for nearly as long as they are on the market (IIRC >1 > decade). > Also SSDs are usually optimized for 4k sectors too. > > > *: > https://en.wikipedia.org/wiki/Shingled_magnetic_recording > > > -- > > Matthias > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
