Re: [PATCH] Reduce memory usage
On Thu, Dec 23, 2021 at 07:55:00PM +0100, Roland via rsync wrote: > hello, > > it's fantastic to see that such optimizations still being found. > > out of curiosity - what is the status of this? will it get merged ? Yes, it was merged after being resubmitted nn github, see commit ae1f0029 (Reduce memory usage (#235), 2021-10-01) in https://github.com/WayneD/rsync -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] Reduce memory usage
hello, it's fantastic to see that such optimizations still being found. out of curiosity - what is the status of this? will it get merged ? roland Am 28.09.21 um 12:05 schrieb Jindřich Makovička via rsync: In 2004, an allocation optimization has been added to the file list handling code, that preallocates 32k of file_struct pointers in a file_list. This optimization predates the incremental recursion feature, for which it is not appropriate anymore. When copying a tree containing a large number of small directories, using the incremental recursion, rsync allocates many short file_lists. Suddenly, the unused file_struct pointers can easily take 90-95% of the memory allocated by rsync. This can be easily reproduced by using valgrind --tool=massif ./rsync -anx /usr /tmp/ and checking the memory profile of the first (sender) process. This patch changes the flist functions to start only with 32 entries for the partial file lists, instead of 32 * 1024. It also modifies the condition for the debug notification that the allocated memory moved, so it does not depend on the initial allocation size. Instead, it now simply checks if the file list has been already allocated. --- flist.c | 9 +++-- rsync.h | 5 +++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/flist.c b/flist.c index 3442d868..530d336e 100644 --- a/flist.c +++ b/flist.c @@ -305,7 +305,7 @@ static void flist_expand(struct file_list *flist, int extra) new_ptr = realloc_array(flist->files, struct file_struct *, flist->malloced); - if (DEBUG_GTE(FLIST, 1) && flist->malloced != FLIST_START) { + if (DEBUG_GTE(FLIST, 1) && flist->files) { rprintf(FCLIENT, "[%s] expand file_list pointer array to %s bytes, did%s move\n", who_am_i(), big_num(sizeof flist->files[0] * flist->malloced), @@ -2186,8 +2186,10 @@ struct file_list *send_file_list(int f, int argc, char *argv[]) #endif flist = cur_flist = flist_new(0, "send_file_list"); + flist_expand(flist, FLIST_START_LARGE); if (inc_recurse) { dir_flist = flist_new(FLIST_TEMP, "send_file_list"); + flist_expand(dir_flist, FLIST_START_LARGE); flags |= FLAG_DIVERT_DIRS; } else dir_flist = cur_flist; @@ -2541,10 +2543,13 @@ struct file_list *recv_file_list(int f, int dir_ndx) #endif flist = flist_new(0, "recv_file_list"); + flist_expand(flist, FLIST_START_LARGE); if (inc_recurse) { - if (flist->ndx_start == 1) + if (flist->ndx_start == 1) { dir_flist = flist_new(FLIST_TEMP, "recv_file_list"); + flist_expand(dir_flist, FLIST_START_LARGE); + } dstart = dir_flist->used; } else { dir_flist = flist; diff --git a/rsync.h b/rsync.h index 88319732..f8fcbffb 100644 --- a/rsync.h +++ b/rsync.h @@ -918,8 +918,9 @@ extern int xattrs_ndx; * Start the flist array at FLIST_START entries and grow it * by doubling until FLIST_LINEAR then grow by FLIST_LINEAR */ -#define FLIST_START(32 * 1024) -#define FLIST_LINEAR (FLIST_START * 512) +#define FLIST_START(32) +#define FLIST_START_LARGE (32 * 1024) +#define FLIST_LINEAR (FLIST_START_LARGE * 512) /* * Extent size for allocation pools: A minimum size of 128KB -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] Reduce memory usage
Original Message On Oct 2, 2021, 15:04, Jindřich Makovička < makov...@gmail.com> wrote: Just note this patch has nothing to do with memory consumption vs performance. It just avoids allocating memory that was left unused anyway. I can read-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Aw: Re: [PATCH] Reduce memory usage
Original Message On Oct 2, 2021, 12:36, < devz...@web.de> wrote: >>In the exchange I argued that proper use of ram as a buffer would have cut >>down backup time to minutes instead of days. >could you give an example where rsync is slowing things down so much due to >ram constraints or inefficient ram use? You find it all in the exchange I already referred to. >please mind that disk bandwith and file copy bandwith is not the same. random >i/o and seek time is the culprit. I am glad you did your homework. >why should rsync use ram for buffering data it copies, if the linux kernel / >vm subsystem already does this? Because my server is not using linux?-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Re: [PATCH] Reduce memory usage
Just note this patch has nothing to do with memory consumption vs performance. It just avoids allocating memory that was left unused anyway. On Sat, Oct 2, 2021, 12:51 devzero--- via rsync wrote: > >In the exchange I argued that proper use of ram as a buffer would have > cut down backup time to minutes instead of days. > > could you give an example where rsync is slowing things down so much due > to ram constraints or inefficient ram use? > > please mind that disk bandwith and file copy bandwith is not the same. > random i/o and seek time is the culprit. > > why should rsync use ram for buffering data it copies, if the linux kernel > / vm subsystem already does this? > > roland > > *Gesendet:* Samstag, 02. Oktober 2021 um 12:07 Uhr > *Von:* "Rupert Gallagher via rsync" > *An:* makov...@gmail.com, rsync@lists.samba.org > *Betreff:* Re: [PATCH] Reduce memory usage > If you look at my previous exchange in the list, I raised the need for > more ram usage via a tool option. In the exchange I argued that proper use > of ram as a buffer would have cut down backup time to minutes instead of > days. At the time, my proposal was dismissed by someone saying that rsync > uses as much ram as it needs. I still feel the need to free rsync from this > mindless constraint, while also welcoming ram usage optimisations such as > yours in this patch. How hard can it be to allow rsync to use 1GB of ram > instead of 100MB? The benefit would be huge. In my case, where a supermicro > server uses a shared bus to transfer data from two disks, the overhead > caused by frequent small buffer IO is so high that backup time is still > huge. And I am using server hardware! PC and laptops are even worse. > > RG > > > > > Original Message > On Sep 26, 2021, 13:54, Jindřich Makovička via rsync < > rsync@lists.samba.org> wrote: > Hi, > > When using rsync to back up the file system on my laptop, containing a > pretty much default linux desktop, I was wondering how rsync uses over > 100MB of RAM it allocates. > > It turned out that most of the memory is used for the arrays of > file_struct pointers, most of which end up unused - much more than the > actual file_struct entries. In my case, the peak usage was 135MB of > pointers, and just 1.5MB of the file_struct entries themselves. > > The problem seems to be that the default file_list allocation parameters > predate the incremental recursion, which allocates a huge number of small > file lists, while AFAICS originally rsync allocated just one large list. > > Applying the attached patch, which reduces the default allocation to 32 > pointers, and preallocates 32K pointers only for the main file lists in > send_file_list and recv_file_list, reduces the peak memory usage in my case > from 142MB to 12MB. > > Regards, > -- > Jindřich Makovička > -- Please use reply-all for most replies to avoid omitting the mailing > list. To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync Before posting, read: > http://www.catb.org/~esr/faqs/smart-questions.html > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html > -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Aw: Re: [PATCH] Reduce memory usage
>In the exchange I argued that proper use of ram as a buffer would have cut down backup time to minutes instead of days. could you give an example where rsync is slowing things down so much due to ram constraints or inefficient ram use? please mind that disk bandwith and file copy bandwith is not the same. random i/o and seek time is the culprit. why should rsync use ram for buffering data it copies, if the linux kernel / vm subsystem already does this? roland Gesendet: Samstag, 02. Oktober 2021 um 12:07 Uhr Von: "Rupert Gallagher via rsync" An: makov...@gmail.com, rsync@lists.samba.org Betreff: Re: [PATCH] Reduce memory usage If you look at my previous exchange in the list, I raised the need for more ram usage via a tool option. In the exchange I argued that proper use of ram as a buffer would have cut down backup time to minutes instead of days. At the time, my proposal was dismissed by someone saying that rsync uses as much ram as it needs. I still feel the need to free rsync from this mindless constraint, while also welcoming ram usage optimisations such as yours in this patch. How hard can it be to allow rsync to use 1GB of ram instead of 100MB? The benefit would be huge. In my case, where a supermicro server uses a shared bus to transfer data from two disks, the overhead caused by frequent small buffer IO is so high that backup time is still huge. And I am using server hardware! PC and laptops are even worse. RG Original Message On Sep 26, 2021, 13:54, Jindřich Makovička via rsync < rsync@lists.samba.org> wrote: Hi, When using rsync to back up the file system on my laptop, containing a pretty much default linux desktop, I was wondering how rsync uses over 100MB of RAM it allocates. It turned out that most of the memory is used for the arrays of file_struct pointers, most of which end up unused - much more than the actual file_struct entries. In my case, the peak usage was 135MB of pointers, and just 1.5MB of the file_struct entries themselves. The problem seems to be that the default file_list allocation parameters predate the incremental recursion, which allocates a huge number of small file lists, while AFAICS originally rsync allocated just one large list. Applying the attached patch, which reduces the default allocation to 32 pointers, and preallocates 32K pointers only for the main file lists in send_file_list and recv_file_list, reduces the peak memory usage in my case from 142MB to 12MB. Regards, -- Jindřich Makovička -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] Reduce memory usage
If you look at my previous exchange in the list, I raised the need for more ram usage via a tool option. In the exchange I argued that proper use of ram as a buffer would have cut down backup time to minutes instead of days. At the time, my proposal was dismissed by someone saying that rsync uses as much ram as it needs. I still feel the need to free rsync from this mindless constraint, while also welcoming ram usage optimisations such as yours in this patch. How hard can it be to allow rsync to use 1GB of ram instead of 100MB? The benefit would be huge. In my case, where a supermicro server uses a shared bus to transfer data from two disks, the overhead caused by frequent small buffer IO is so high that backup time is still huge. And I am using server hardware! PC and laptops are even worse. RG Original Message On Sep 26, 2021, 13:54, Jindřich Makovička via rsync < rsync@lists.samba.org> wrote: Hi, When using rsync to back up the file system on my laptop, containing a pretty much default linux desktop, I was wondering how rsync uses over 100MB of RAM it allocates. It turned out that most of the memory is used for the arrays of file_struct pointers, most of which end up unused - much more than the actual file_struct entries. In my case, the peak usage was 135MB of pointers, and just 1.5MB of the file_struct entries themselves. The problem seems to be that the default file_list allocation parameters predate the incremental recursion, which allocates a huge number of small file lists, while AFAICS originally rsync allocated just one large list. Applying the attached patch, which reduces the default allocation to 32 pointers, and preallocates 32K pointers only for the main file lists in send_file_list and recv_file_list, reduces the peak memory usage in my case from 142MB to 12MB. Regards, -- Jindřich Makovička-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] Reduce memory usage
Looks awesome, really nice catch! -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] Reduce memory usage
On Mon, 27 Sep 2021 22:03:01 +0200 Johannes Altmanninger wrote: > > On Sun, Sep 26, 2021 at 01:54:13PM +0200, Jindřich Makovička via > > rsync wrote: > > > > Applying the attached patch, which reduces the default allocation > > to 32 pointers, and preallocates 32K pointers only for the main > > file lists in send_file_list and recv_file_list, reduces the peak > > memory usage in my case from 142MB to 12MB. > > The patch looks very reasonable from what I can tell. > Out of curiosity, how did you measure peak memory? > I used "/bin/time -v rsync", and already get (very small) > improvements in maximum RSS when copying the rsync source tree :) Thanks for feedback. I profiled the memory usage using valgrind, simply by running valgrind --tool=massif ./rsync -anx /usr /tmp/ This issue affects only large trees with lots of small subdirectories, you will not see much improvement with just a couple of directories the rsync tree has. The outputs below are generated from the sender process output by ms_print . Before the fix: Command:./rsync -anx /usr /tmp/ Massif arguments: (none) ms_print arguments: massif.out.199013 MB 127.6^ # | # | # | # | @ # | @ # | @ # | @ # | @ # : | @ # : | @ #:: | : ::: @ @ #:: | : ::@ @ #:: | : : ::@ @ :# ::: : | : : : :: : :@:@:::# : ::: : | ::: :: :: :: : :@:@: :#:: : | : ::: :: :: : :@:@: :# :::: : | : :: :: :::@:@: :# ::@ ::: :: :: | : ::: ::: : @:: :@: :# ::@ :::@:: @: | : ::: :: @:@: @#@:@:@:@:@: 0 +--->Gi 0 4.574 ntime(i) total(B) useful-heap(B) extra-heap(B)stacks(B) 47 2,853,846,986 133,806,904 133,270,491 536,4130 99.60% (133,270,491B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc. ->99.12% (132,629,814B) 0x1329CC: my_alloc (in /home/henry/build/rsync/rsync) | ->95.80% (128,188,416B) 0x11383E: flist_expand (in /home/henry/build/rsync/rsync) | | ->95.80% (128,188,416B) 0x117AAB: send_file_name (in /home/henry/build/rsync/rsync) | | | ->95.80% (128,188,416B) 0x1184B4: send_directory (in /home/henry/build/rsync/rsync) | | | | ->95.80% (128,188,416B) 0x118CD1: send1extra (in /home/henry/build/rsync/rsync) | | | | ->95.80% (128,188,416B) 0x1190F4: send_extra_file_list (in /home/henry/build/rsync/rsync) | | | | ->95.80% (128,188,416B) 0x12A17A: send_files (in /home/henry/build/rsync/rsync) | | | | | ->95.80% (128,188,416B) 0x1362C5: client_run (in /home/henry/build/rsync/rsync) | | | | | ->95.80% (128,188,416B) 0x136F2D: start_client (in /home/henry/build/rsync/rsync) | | | | | ->95.80% (128,188,416B) 0x137604: main (in /home/henry/build/rsync/rsync) | | | | | | | | | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%) | | | | | | | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%) | | | | | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%) | | | ->02.16% (2,883,584B) 0x16E977: pool_alloc (in /home/henry/build/rsync/rsync) | | ->02.16% (2,883,584B) 0x116E12: make_file (in /home/henry/build/rsync/rsync) | | ->02.16% (2,883,584B) 0x11747F: send_file_name (in /home/henry/build/rsync/rsync) | | ->02.06% (2,752,512B) 0x1184B4: send_directory (in /home/henry/build/rsync/rsync) | | | ->02.06% (2,752,512B) 0x118CD1: send1extra (in /home/henry/build/rsync/
Re: [PATCH] Reduce memory usage
> On Sun, Sep 26, 2021 at 01:54:13PM +0200, Jindřich Makovička via rsync wrote: > > Applying the attached patch, which reduces the default allocation to 32 > pointers, and preallocates 32K pointers only for the main file lists in > send_file_list and recv_file_list, reduces the peak memory usage in my case > from 142MB to 12MB. The patch looks very reasonable from what I can tell. Out of curiosity, how did you measure peak memory? I used "/bin/time -v rsync", and already get (very small) improvements in maximum RSS when copying the rsync source tree :) On Mon, Sep 27, 2021 at 04:42:25PM +0200, Jindřich Makovička via rsync wrote: > Reduce memory usage > > Start only with 32 entries for the partial file lists, instead of 32k. > The log message could be a bit more detailed. You already mentioned that the big 32k allocation predates the recursive version, that's very useful information to add here. Maybe even explain why we change the first check from "flist->malloced != FLIST_START" to "flist->files". (Also I'd use "git send-email" to send patches inline but I'm not sure what's the convention here) > diff --git a/flist.c b/flist.c > index 3442d868..0f7a64e6 100644 > --- a/flist.c > +++ b/flist.c > @@ -303,11 +303,11 @@ static void flist_expand(struct file_list *flist, int > extra) > if (flist->malloced < flist->used + extra) > flist->malloced = flist->used + extra; > > new_ptr = realloc_array(flist->files, struct file_struct *, > flist->malloced); > > - if (DEBUG_GTE(FLIST, 1) && flist->malloced != FLIST_START) { > + if (DEBUG_GTE(FLIST, 1) && flist->files) { Yep, the new check makes more sense because now it's more obvious that the debug message is only printed when flist->files is realloc'd, and not when it's allocated for the first time. > rprintf(FCLIENT, "[%s] expand file_list pointer array to %s > bytes, did%s move\n", > who_am_i(), > big_num(sizeof flist->files[0] * flist->malloced), > (new_ptr == flist->files) ? " not" : ""); > } > @@ -2184,10 +2184,11 @@ struct file_list *send_file_list(int f, int argc, > char *argv[]) > if (preserve_hard_links && protocol_version >= 30 && !cur_flist) > init_hard_links(); > #endif > > flist = cur_flist = flist_new(0, "send_file_list"); > + flist_expand(flist, FLIST_START_LARGE); > if (inc_recurse) { > dir_flist = flist_new(FLIST_TEMP, "send_file_list"); This probably wants an flist_expand(dir_flist, FLIST_START_LARGE), because dir_flist is a global. I think the idea is that all flist_new() inside loops/recursive calls should start small, but lists that are only ever allocated once should stay at 32k. > flags |= FLAG_DIVERT_DIRS; > } else > dir_flist = cur_flist; > @@ -2539,10 +2540,11 @@ struct file_list *recv_file_list(int f, int dir_ndx) > if (preserve_hard_links && !first_flist) > init_hard_links(); > #endif > > flist = flist_new(0, "recv_file_list"); > + flist_expand(flist, FLIST_START_LARGE); > > if (inc_recurse) { > if (flist->ndx_start == 1) > dir_flist = flist_new(FLIST_TEMP, "recv_file_list"); Same here I guess. Maybe we should add an "initial size" parameter to flist_new(), so it can call flist_expand() automatically? > dstart = dir_flist->used; > diff --git a/rsync.h b/rsync.h > index 88319732..17f8700e 100644 > --- a/rsync.h > +++ b/rsync.h > @@ -916,12 +916,13 @@ extern int xattrs_ndx; > > /* > * Start the flist array at FLIST_START entries and grow it > * by doubling until FLIST_LINEAR then grow by FLIST_LINEAR > */ > -#define FLIST_START (32 * 1024) > -#define FLIST_LINEAR (FLIST_START * 512) > +#define FLIST_START (32) > +#define FLIST_START_LARGE(32 * 1024) > +#define FLIST_LINEAR (FLIST_START_LARGE * 512) Probably these should remain aligned (I'm assuming tab has width 8) -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] Reduce memory usage
On Mon, 27 Sep 2021 13:38:22 +0200 Jindřich Makovička wrote: > On Sun, 26 Sep 2021 13:54:13 +0200 > Jindřich Makovička wrote: > > Hi, > > > > ... > > > > Applying the attached patch, which reduces the default allocation to > > 32 pointers, and preallocates 32K pointers only for the main file > > lists in send_file_list and recv_file_list, reduces the peak memory > > usage in my case from 142MB to 12MB. > > The original patch breaks the testsuite due to extra messages in the > output. Actually it makes more sense to change the debug print to check that the original pointer is non-null and leave the testsuite as is. -- Jindrich Makovicka >From bdfdef1c5a4437e2492da148b824d39ba235704e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jind=C5=99ich=20Makovi=C4=8Dka?= Date: Sun, 26 Sep 2021 12:01:21 +0200 Subject: [PATCH] Reduce memory usage Start only with 32 entries for the partial file lists, instead of 32k. --- flist.c | 4 +++- rsync.h | 5 +++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/flist.c b/flist.c index 3442d868..0f7a64e6 100644 --- a/flist.c +++ b/flist.c @@ -305,7 +305,7 @@ static void flist_expand(struct file_list *flist, int extra) new_ptr = realloc_array(flist->files, struct file_struct *, flist->malloced); - if (DEBUG_GTE(FLIST, 1) && flist->malloced != FLIST_START) { + if (DEBUG_GTE(FLIST, 1) && flist->files) { rprintf(FCLIENT, "[%s] expand file_list pointer array to %s bytes, did%s move\n", who_am_i(), big_num(sizeof flist->files[0] * flist->malloced), @@ -2186,6 +2186,7 @@ struct file_list *send_file_list(int f, int argc, char *argv[]) #endif flist = cur_flist = flist_new(0, "send_file_list"); + flist_expand(flist, FLIST_START_LARGE); if (inc_recurse) { dir_flist = flist_new(FLIST_TEMP, "send_file_list"); flags |= FLAG_DIVERT_DIRS; @@ -2541,6 +2542,7 @@ struct file_list *recv_file_list(int f, int dir_ndx) #endif flist = flist_new(0, "recv_file_list"); + flist_expand(flist, FLIST_START_LARGE); if (inc_recurse) { if (flist->ndx_start == 1) diff --git a/rsync.h b/rsync.h index 88319732..17f8700e 100644 --- a/rsync.h +++ b/rsync.h @@ -918,8 +918,9 @@ extern int xattrs_ndx; * Start the flist array at FLIST_START entries and grow it * by doubling until FLIST_LINEAR then grow by FLIST_LINEAR */ -#define FLIST_START (32 * 1024) -#define FLIST_LINEAR (FLIST_START * 512) +#define FLIST_START (32) +#define FLIST_START_LARGE (32 * 1024) +#define FLIST_LINEAR (FLIST_START_LARGE * 512) /* * Extent size for allocation pools: A minimum size of 128KB -- 2.33.0 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] Reduce memory usage
On Sun, 26 Sep 2021 13:54:13 +0200 Jindřich Makovička wrote: > Hi, > > ... > > Applying the attached patch, which reduces the default allocation to > 32 pointers, and preallocates 32K pointers only for the main file > lists in send_file_list and recv_file_list, reduces the peak memory > usage in my case from 142MB to 12MB. The original patch breaks the testsuite due to extra messages in the output. Fix attached. Regards, -- Jindrich Makovicka >From aa907eabe701550cd5649b6e3da2b22b79b38a06 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jind=C5=99ich=20Makovi=C4=8Dka?= Date: Sun, 26 Sep 2021 12:01:21 +0200 Subject: [PATCH] Reduce memory usage Start only with 32 entries for the partial file lists, instead of 32k. --- flist.c | 2 ++ rsync.h | 5 +++-- testsuite/rsync.fns | 1 + 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/flist.c b/flist.c index 3442d868..37f70b69 100644 --- a/flist.c +++ b/flist.c @@ -2186,6 +2186,7 @@ struct file_list *send_file_list(int f, int argc, char *argv[]) #endif flist = cur_flist = flist_new(0, "send_file_list"); + flist_expand(flist, FLIST_START_LARGE); if (inc_recurse) { dir_flist = flist_new(FLIST_TEMP, "send_file_list"); flags |= FLAG_DIVERT_DIRS; @@ -2541,6 +2542,7 @@ struct file_list *recv_file_list(int f, int dir_ndx) #endif flist = flist_new(0, "recv_file_list"); + flist_expand(flist, FLIST_START_LARGE); if (inc_recurse) { if (flist->ndx_start == 1) diff --git a/rsync.h b/rsync.h index 88319732..17f8700e 100644 --- a/rsync.h +++ b/rsync.h @@ -918,8 +918,9 @@ extern int xattrs_ndx; * Start the flist array at FLIST_START entries and grow it * by doubling until FLIST_LINEAR then grow by FLIST_LINEAR */ -#define FLIST_START (32 * 1024) -#define FLIST_LINEAR (FLIST_START * 512) +#define FLIST_START (32) +#define FLIST_START_LARGE (32 * 1024) +#define FLIST_LINEAR (FLIST_START_LARGE * 512) /* * Extent size for allocation pools: A minimum size of 128KB diff --git a/testsuite/rsync.fns b/testsuite/rsync.fns index 1e2b399f..220b9e21 100644 --- a/testsuite/rsync.fns +++ b/testsuite/rsync.fns @@ -89,6 +89,7 @@ v_filt() { -e '/^total: /d' \ -e '/^client charset: /d' \ -e '/^server charset: /d' \ + -e '/ expand file_list pointer array /d' \ -e '/^$/,$d' } -- 2.33.0 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html