David Rees wrote: > On 3/27/07, Evren Yurtesen <[EMAIL PROTECTED]> wrote: >> David Rees wrote: >>> Evren, I didn't see that you mentioned a wall clock time for your >>> backups? I want to know how many files are in a single backup, how >>> much data is in that backup and how long it takes to perform that >>> backup. >> I sent the status of the backups earlier today to mailing list? > > Still missing wall-clock time. Though we can extrapolate that it may > be taking over 6 hours for a full backup of the system below. Is that > correct?
That is true, full backups take about 500-600 minutes and incrementals take 200-300minutes. >> This is from 1 machine. >> >> Totals Existing Files New Files >> Backup# Type #Files Size/MB MB/sec #Files Size/MB >> #Files Size/MB >> 112 full 149290 1957.8 0.10 149251 1937.9 4601 20.7 > >> I dont know if the problem is hard links. This is not a FreeBSD or Linux >> problem. It exists on both. Just that people using ultra fast 5 disk >> raid 5 setups are seeing 2mbytes/sec transfer rate means that backuppc >> is very very inefficient. > > As mentioned earlier, RAID 5 is horrible for random small read/write > performance. It is often worse than a single disk. Parity information is not required for normal reads, so you get 4-5 times better performance with 5 disks. Plus usually those drives are 10K rpm. When I am making incremental or even full backups, backuppc does not write same files again so writes are very little. So you get a huge performance boost with raid5 and 10k rpm drives compared to a normal ide drive. > But still, I have a client which has 1.5 million files and 80-100GB of > data. A full backup takes 4-6 hours which is reasonable. Full backups > average 4.5-5MB/s. > >> For example this guy is using Linux (problem is OS independent) >> http://forum.psoft.net/showpost.php?p=107808&postcount=16 > > Your transfer rates are at least 5x slower than his and 10x slower > than what most people here are getting. There is something wrong > specifically with your setup. I suspect that if your backups were > going 5x faster you'd be happy and 10x faster you'd be very happy. I would be happy if backups were made with the same speed as other backup programs can do. >> On Linux with raid setup with async io etc. people are getting slightly >> better results. I think ufs2 is just fine. I wonder if there is >> something in my explanations...The problem is backuppc. People are >> getting ~2mbytes/sec(was it 2 or 5?) speed with raid5 and 5 drives, >> using Linux. It is a miracle that backup even finishes in 24 hours using >> a standart ide drive. > > With ext2 the default is async IO. With ext3 (the default system now) > the default is ordered which is similar to BSD's soft updates. > >> This is like the 'Contact' movie. The sphere took 30 seconds to download >> but there were 18 hours of recording. If what you said was true and >> backuppc would be backing up very small amount of files and skipping >> most, then backups would probably take less time than 2-4 hours each. > > Using the rsync transfer method, it does require at least a stat of > every single file to be backed up. For 150k files and 2GB of data, > you'd really expect this to be done within a hour with nearly any > machine. I am using rsync to sync 2GB of files from one machine to another and the number of files is about 100k and the operation takes 30seconds to 1 minute in average. > Are you using the --checksum-seed option? That will significantly > speed up (the 3rd and later) backups as well. No, it requires a patched rsync. > Are you also sure the backup server isn't swapping as well? Can you > get us some good logs from vmstat or similar during a backup? I can tell that it is not swapping because the disk where the swaps are idle while taking backup. ad0 is the disk with the swap. If there was such problem then it would be reading like crazy from swap. I have given this information before. > I also suspect that there is something else in this case slowing your > machine down. Unless you give us the information to help track it > down, we will not be able to figure it out. I feel like I am pulling > teeth. > I have given you all the information you asked for(didnt I?), even tried async option. Incremental backup of 1 machine still took about 300 minutes. The machine is working fine. I was using another backup program which was working way faster to backup the same machines. So I dont think that I have a hardware problem or such. Thanks, Evren ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
