Hamish Guthrie writes: > I am sorry to harp on about this, but, it gets back to my analysis of > rsync under backuppc a few months ago - I still think that we need to do > an implementation of File::RsyncP in C as opposed to perl. I have done > some memory requirement analysis of a raw rsync backup of a large > filesystem against a BackupPC backup of the same filesystem, and the > differences are astonishing in terms of memory requirements, not to > mention the raw speed of the actual backup. I know that there are some > additional requirements for File:RsyncP, but, I am sure that running in > C will be a huge amount more efficient than running in an interpreted > perl environment. > > Unfortunately, at the moment, I have a few other commercial projects I > am busy with and cannot immediately contribute to such a development, > but in my view, this would be the most impotant performance-enhancing > developments for BackupPC. > > I could quote specific examples of BackupPC (using File::RsyncP) vs > rsync itself, but I will not bore you or myself, but I can assure you > that the difference in speed is about 10:1. Memory usage is also > DRAMATICALLY less in the raw rsync environment, I would guess at least > 50% less RAM requirement. > > I am hoping to finish my current commercial projects in the next few > weeks, at which stage, I will have a look at File::RsyncP unless someone > else has taken the bait resulting from this e-mail and developed the > appropriate C code in the mean-time.
I agree with you and David Relson that the perl parts of File::RsyncP (and BackupPC), have higher memory usage beyond native rsync itself. Data that David sent suggests a factor of 2, which is believable. But remember that rsync (and BackupPC) suffer from the problem that the entire file list must be stored in memory. And the receiver forks, so two processes (whether rsync or BackupPC_dump) need the full file list. Sure, BackupPC does it less efficiently in perl, but it is a basic architectural issue for both native rsync and BackupPC. With a lot of effort, that factor of 2 could be reduced to maybe 1.1 or 1.2. But the basic problem of memory usage proportional to the number of files remains. Roy Keene's approach of developing a native backup client for BackupPC that doesn't suffer this limitation - and will be native C at both ends - looks like the most encouraging solution. It will not have memory usage proportional to the total number of files. Unfortunately, Huuricane Katrina has forced his office to relocate and development will stall for at least the next month. Craig ------------------------------------------------------- SF.Net email is Sponsored by the Better Software Conference & EXPO September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
