Thank you all again for your responses. I setup the backup server on a better computer now: Pentium 4 2.4GHz with 760M ram. The system hard drive interface is also faster (ATA133, instead of ATA66 on old computer). The backup files are still on the same USB drive. The data files are on a new computer with Q6600 2.4GHz + 4GB ram running Vista x64. Total backup files are about 170GB (over 500k files).All hash files are still on the backup server. So there should be no major file transfer over network (is this correct?).
I did a full backup via rsyncd. It takes about 1 hour for the client computer to prepare the file list before the transfer started. Total backup time is a little less than 8 hours and average transfer speed more than 6MB/sec. Big boost than before. I think the CPU is making the big difference and rsync is doing a lot of calculation. Can someone explain in detail what rsync is doing? I do not use any compression. 8 hours is long but tolerable for once a week. I suppose incremental backup will be faster. I don't think I will do any upgrade on the hard drive/usb/network since I believe the CPU/memory is the limiting step. Any thought on fine tune /configure the BackupPC to make it faster? Thanks. PS: the USB drive still shows as 40.000MB/s in dmesg. Is this a FreeBSD thing? I suppose it should be 60MB for usb2.0 (480Mbps). This may not be important here since the file transfer speed is far below 40MB/s. Les Mikesell wrote: > Jinshi wrote: >> Thank you all for your reply. Apparently everyone consider this is too >> slow since the difference between usb2.0 and usb1.1 is quite big. But I >> do tested all other usb ports. The dmesg says the transfer speed is >> either 40.000MB or 1.000MB/s. So, sorry everyone, I still insist I have >> usb2.0 here :) >> >> I didn't reply sooner because I want to wait and see how long the backup >> is. Now, total file about 170GB (I moved some files from other computer. >> Those files were backed up before, so the hash files are available on >> the backup computer). There is almost no new files (details below). And >> full backup by SMB transfer takes 32 hours (only 7.3MB new files), by >> rsyncd transfer takes 26 hours (1.8GB new files since my wife uploaded >> more photos during SMB backup time). > > Was the rsync run a full or incremental? A mostly unchanged incremental > should be much faster than a full because the full does a block checksum > comparison even on files where the timestamp and length match. > >> So, rsyncd helps, but not a lot, just like Serge predicted. Average >> transfer speed is 1.5 and 1.8 MB/sec, respectively. >> >> Now, back to my original question: where is the bottleneck? I am going >> to upgrade the computer to P4. If that still doesn't help much, I don't >> know what to do next?! > > Rsync transfers the entire directory listing before starting so there is > a certain amount of RAM needed per file - if you have less the server > might start swapping and slow down. Depending on your directory layout, > it might be possible to break the target up into several smaller chunks > to help with this. Also, a slightly more extreme approach would be to > make the subdirectory runs appear to be different hosts using the alias > feature. That way you could stagger the full and incremental runs so > you don't have to complete a full in one day. > ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
