Hi, When using rsync to back up the file system on my laptop, containing a pretty much default linux desktop, I was wondering how rsync uses over 100MB of RAM it allocates.
It turned out that most of the memory is used for the arrays of file_struct pointers, most of which end up unused - much more than the actual file_struct entries. In my case, the peak usage was 135MB of pointers, and just 1.5MB of the file_struct entries themselves. The problem seems to be that the default file_list allocation parameters predate the incremental recursion, which allocates a huge number of small file lists, while AFAICS originally rsync allocated just one large list. Applying the attached patch, which reduces the default allocation to 32 pointers, and preallocates 32K pointers only for the main file lists in send_file_list and recv_file_list, reduces the peak memory usage in my case from 142MB to 12MB. Regards, -- Jindřich Makovička
From ef169c9157d312c63bad00e3bfc1d8eb70d56ccd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jind=C5=99ich=20Makovi=C4=8Dka?= <makov...@gmail.com> Date: Sun, 26 Sep 2021 12:01:21 +0200 Subject: [PATCH] Reduce memory usage Start only with 32 entries for the partial file lists, instead of 32k. --- flist.c | 2 ++ rsync.h | 5 +++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/flist.c b/flist.c index 3442d868..37f70b69 100644 --- a/flist.c +++ b/flist.c @@ -2186,6 +2186,7 @@ struct file_list *send_file_list(int f, int argc, char *argv[]) #endif flist = cur_flist = flist_new(0, "send_file_list"); + flist_expand(flist, FLIST_START_LARGE); if (inc_recurse) { dir_flist = flist_new(FLIST_TEMP, "send_file_list"); flags |= FLAG_DIVERT_DIRS; @@ -2541,6 +2542,7 @@ struct file_list *recv_file_list(int f, int dir_ndx) #endif flist = flist_new(0, "recv_file_list"); + flist_expand(flist, FLIST_START_LARGE); if (inc_recurse) { if (flist->ndx_start == 1) diff --git a/rsync.h b/rsync.h index 2f674bc5..708fd244 100644 --- a/rsync.h +++ b/rsync.h @@ -917,8 +917,9 @@ extern int xattrs_ndx; * Start the flist array at FLIST_START entries and grow it * by doubling until FLIST_LINEAR then grow by FLIST_LINEAR */ -#define FLIST_START (32 * 1024) -#define FLIST_LINEAR (FLIST_START * 512) +#define FLIST_START (32) +#define FLIST_START_LARGE (32 * 1024) +#define FLIST_LINEAR (FLIST_START_LARGE * 512) /* * Extent size for allocation pools: A minimum size of 128KB -- 2.33.0
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html