On Tue, 27 Apr 2004, Eric Whiting <[EMAIL PROTECTED]> wrote: > > I was watching the memory usage with a bash command line while loop > while(true);do cat /proc/$pid/status |grep VmSize ;sleep 60;done > > My last runs I changed the sleep to 5 at the end to get finer resolution on the > memory jump. I was not expecting to see that jump -- I was also curious.
I think I can explain the jump. When the file list is being received by the receiver, there is a steady increase in memory usage as the data structures are allocated for each file as the info is received. A list of pointers to the data structures is also being built. When that list fills up its current buffer, a new one twice the size is allocated, the pointers are copied, and the old buffer is freed. That freed memory is now availble to hold more file data structures. In the beginning, the memory allocation for the file data structures swamps the jumps due to allocation for the list. But once the list size reaches into the megabytes, then you see the effects of the jump. > some more cut/paste: > > dest_host:/usr/bin # while(true);do cat /proc/24624/status |grep VmSize;sleep > 60; done > VmSize: 6380 kB > VmSize: 8196 kB > VmSize: 9236 kB > VmSize: 10796 kB > VmSize: 12096 kB > VmSize: 14168 kB > VmSize: 15468 kB > VmSize: 16768 kB > VmSize: 18068 kB That was a jump of about 1200. > VmSize: 20408 kB <- memory jump starts after this point (sender stays stable at > 22M) That was a jump of about 2400. > VmSize: 26016 kB > (cntl-c to change the resolution) > > dest_host:/usr/bin # while(true);do cat /proc/24624/status |grep VmSize;sleep 5; > done That was a jump of about 4800. > VmSize: 35884 kB > VmSize: 35884 kB > VmSize: 35884 kB > VmSize: 37964 kB That was a jump of about 9600. The memory size stabilizes for awhile since the previous index buffer is reused for file info data. > VmSize: 22412 kB And THAT was a decrease of 15500. Since the rsync command was a 'push', the generator runs on the destination/receiver, so maybe the 15500 is the memory that is now shared between the generator and the receiver after the fork? Or maybe this is the effect of some final cleanup after receiving the file list? This reduction may be related to this bullet item in the NEWS file: * Changed hardlink info and file_struct + strings to use allocation pools. This reduces memory use for large file- sets and permits freeing memory to the OS. (J.W. Schultz) Wayne? JW? Any thoughts? -- John Van Essen Univ of MN Alumnus <[EMAIL PROTECTED]> -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html