John, > Well some more digging and a recompile of perl set to use system > malloc() instead of it's build in malloc() and the problem has gone > away (usemymalloc=n). Something the restore process does really > causes perl malloc to go crazy on lage restore jobs. Playing with > the write buffer size changed how the problem manifested (both smaller > - 65536 and larger - 2^24 - 16MB buffers made it better but didn't > eliminate it). The 1MB default buffer seemed to be the worst. > > There is a comment in the FreeBSD perl ports makefile about malloc > having problems with threads but perl was not built with threads on > my box as far as I know and BackupPC does not use threads. > > I just managed to get an archive of a 375GB backup set so I'm now much > happier.
Sorry about the delay in replying. Yes, I've heard reports of this once before. I spent some time looking at the BackupPC_tarCreate code, and adding mstat() (a debugging function included in Devel::Peek, which provides freelist and other malloc state information). I could see the memory usage grow unexpectedly in certain cases, but I didn't spend enough time to really figure out the root cause. If I recall correctly, in certain cases it was fragmenting the freed 1MB buffer blocks that BackupPC_tarCreate uses, and then needing to allocate another one since the last was now a bit smaller than 1MB. I suspect it is a subtle bug in perl's internal malloc that is triggered in certain complex sequences of large and small malloc()/free()s. Your workaround sounds like the right one. Craig ------------------------------------------------------------------------- SF.Net email is sponsored by: The Future of Linux Business White Paper from Novell. From the desktop to the data center, Linux is going mainstream. Let it simplify your IT future. http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4 _______________________________________________ BackupPC-users mailing list [email protected] List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
