If you look at my previous exchange in the list, I raised the need for more ram
usage via a tool option. In the exchange I argued that proper use of ram as a
buffer would have cut down backup time to minutes instead of days. At the time,
my proposal was dismissed by someone saying that rsync uses as much ram as it
needs. I still feel the need to free rsync from this mindless constraint, while
also welcoming ram usage optimisations such as yours in this patch. How hard
can it be to allow rsync to use 1GB of ram instead of 100MB? The benefit would
be huge. In my case, where a supermicro server uses a shared bus to transfer
data from two disks, the overhead caused by frequent small buffer IO is so high
that backup time is still huge. And I am using server hardware! PC and laptops
are even worse.
RG
-------- Original Message --------
On Sep 26, 2021, 13:54, Jindřich Makovička via rsync < rsync@lists.samba.org>
wrote:
Hi,
When using rsync to back up the file system on my laptop, containing a pretty
much default linux desktop, I was wondering how rsync uses over 100MB of RAM it
allocates.
It turned out that most of the memory is used for the arrays of file_struct
pointers, most of which end up unused - much more than the actual file_struct
entries. In my case, the peak usage was 135MB of pointers, and just 1.5MB of
the file_struct entries themselves.
The problem seems to be that the default file_list allocation parameters
predate the incremental recursion, which allocates a huge number of small file
lists, while AFAICS originally rsync allocated just one large list.
Applying the attached patch, which reduces the default allocation to 32
pointers, and preallocates 32K pointers only for the main file lists in
send_file_list and recv_file_list, reduces the peak memory usage in my case
from 142MB to 12MB.
Regards,
--
Jindřich Makovička
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
