David Rees wrote: > On 3/27/07, Evren Yurtesen <[EMAIL PROTECTED]> wrote: >> Yes it is terrible. I get much better performance if I do the tar option >> with the same files. As a matter of fact I was using a smaller script >> for taking backups earlier. (which I still use on some servers) and >> transfer files over NFS. It works way faster, especially incremental >> backups take 5-10 minutes compared to 400 minutes with backuppc > > So have you tried using tar for backups using BackupPC? > > I've said it before, there is something wrong with your setup because > on my server incrementals also only take 5-10 minutes as you'd expect. > And my server isn't that different than yours disk-wise, just RAID1 > instead of no raid, it's even the exact same disk.
Are you using tar? I will give it a try soon if I cant solve the proble any other way. I will make a post about the results. > On 3/27/07, Evren Yurtesen <[EMAIL PROTECTED]> wrote: >> David Rees wrote: >> That is true, full backups take about 500-600 minutes and incrementals >> take 200-300minutes. > > Is that from all 4 clients being backed up? Are all similar in having > ~150k files and ~2GB data to back up? All 4 machines are quite similar. I have given very optimistic examples, the data is about 3-7gb and latest full backups have been taking about 900minutes in one machine. Host User #Full Full Age (days) Full Size (GB) Speed (MB/s) clienthost 4 6.9 5.56 0.10 where backups Backup# Type Filled Level Start Date Duration/mins 245 full yes 0 3/21 20:00 913.9 6.9 246 incr no 1 3/22 20:00 280.4 5.9 >>> Your transfer rates are at least 5x slower than his and 10x slower >>> than what most people here are getting. There is something wrong >>> specifically with your setup. I suspect that if your backups were >>> going 5x faster you'd be happy and 10x faster you'd be very happy. >> I would be happy if backups were made with the same speed as other >> backup programs can do. > > What other backup programs are you comparing BackupPC to? I was running backup over nfs for 6 machines (I actually still run this few of the old machines) using afio to archive the files. Also feedback from some other people I know who were using http://www.r1soft.com/index.html >> I am using rsync to sync 2GB of files from one machine to another and >> the number of files is about 100k and the operation takes 30seconds to 1 >> minute in average. > > Is this using the same backup server and client in your earlier data example? No, it was another machine however I just ran rsync on the same machine and results are below.(the files were copied to the disk where backuppc is taking backups to) rsync -vaW -e 'ssh -c blowfish' --copy-dirlinks --keep-dirlinks --delete --exclude 'dev/*' --exclude 'proc/*' --exclude='usr/obj/*' Running 1st time to copy all files. sent 7938868 bytes received 6392356383 bytes 3283044.50 bytes/sec total size is 6365920146 speedup is 0.99 32m54.13s real 7m27.47s user 7m39.88s sys Eh I guess it is a good speed bump... Running 2nd time (I know this is not exactly same as incremental as there are very few changed files but it shows how long time it takes to compare files) sent 4432 bytes received 160088751 bytes 558789.47 bytes/sec total size is 6364731379 speedup is 39.76 5m3.21s real 24.84s user 1m2.82s sys >>> Are you using the --checksum-seed option? That will significantly >>> speed up (the 3rd and later) backups as well. >> No, it requires a patched rsync. > > You must have a very old version of rsync on your machine. > --checksum-seed has been available for quite some time in the official > rsync tree. It significantly improves rsync performance, I recommend > that you installed a recent rsync if possible. You need rsync 2.6.3 > (released 30 Sep 2004) or later. rsync version 2.6.8 protocol version 29 In the conf file it says that you need a patched rsync, but now I see in manual that mine supports it also. Thanks for the heads up. I will try this and let you know how it goes. >>> Are you also sure the backup server isn't swapping as well? Can you >>> get us some good logs from vmstat or similar during a backup? I can do that when next backups come and return back to you. But I am sure that it is not swapping. As I told before the ad0 disk (where the swap is) is almost idle during backups. >> I can tell that it is not swapping because the disk where the swaps are >> idle while taking backup. ad0 is the disk with the swap. If there was >> such problem then it would be reading like crazy from swap. I have given >> this information before. > > The information you provided before was just about impossible to read. > Please provide additional data in a readable format. I have sent this information earlier, Disks ad0 ad2 KB/t 4.00 25.50 tps 1 75 MB/s 0.00 1.87 % busy 1 96 The other, well it got wrapped I know... Sorry about that. I can later send iostat and vmstat seperately which should be more readable. But if the machine was actively using swap, ad0 couldnt be idle sitting there. >> The machine is working fine. I was using another backup program which >> was working way faster to backup the same machines. So I dont think that >> I have a hardware problem or such. > > OK, so it is using the same machines? Well the backup drive was the same drive so... The clients were about the same more or less. However I am sending the bonnie++ results also. CPU: Intel(R) Celeron(R) CPU 1.70GHz (1716.91-MHz 686-class CPU) real memory = 267321344 (254 MB) ad2: 238475MB <ST3250824A/3.AAH> [484521/16/63] at ata1-master UDMA100 Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP Backup 500M 15765 15 10722 9 57647 27 181.6 18 Latency 67620us 413ms 20515us 321ms Version 1.93c ------Sequential Create------ --------Random Create-------- Backup -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 7385 75 +++++ +++ 25633 93 5900 62 +++++ +++ 25816 95 Latency 194ms 272us 39123us 198ms 271us 26212us >> Earlier I sent the disk stats when the backup was working and I have >> pointed out that the disk with swap is not working at all while taking >> backup. We can easily conclude that swapping is not an issue. > > Can you provide stats during the entire backup? > >>> On all my production machines I make sure sysstat is running and then >>> run `sar -A` the minute before midnight and pipe the output to email >>> for analysis should the need arise. >> sar -A ? > > That provides detailed system statistics collected at specified intervals. > > Here's a sample from my backuppc server which has 3 disks while > backups are being run. The backuppc partition uses 2 disks in RAID1 > exactly the same as yours. The other disk is the system disk (also > 7200rpm ATA). > Hardware/OS: AthlonXP 2000+ 1GB RAM, Fedora Core 6. A max of 3 backups > will run at a time on this machine. This same data can be generated > using sar -b. > > 23:00:02 tps rtps wtps bread/s bwrtn/s > 23:10:02 639.17 219.11 420.06 5440.01 8864.10 > 23:20:03 757.28 215.81 541.46 5422.98 11696.18 > 23:30:04 497.98 211.21 286.78 3367.85 5903.83 > 23:40:04 984.20 322.85 661.35 7436.95 14135.63 > 23:50:03 619.68 391.74 227.93 10355.83 5160.97 > > You can see how much higher this system's IO/s seems to be that yours > from the only performance data you sent us. > > On 3/28/07, Evren Yurtesen <[EMAIL PROTECTED]> wrote: >> Adam Goryachev wrote: >>> anyway, could you please elaborate on the method that your 'other' >>> backup solution used, ie, if your other solution used rsync, and you are >>> using rsync with backuppc, then that might be helpful to know. If you >>> used tar before, and now use rsync, that would obviously make a difference. >> I have already said it earlier, the other backup was taking backup over >> NFS. I also have machines were for example 2GB data with 100k files is >> synced to another using rsync, it takes 30-60 seconds to go through 100k >> files if there are not so many files to sync. > > Same machines or different machines? Please be very exact when you > post information. > >> I disagree, I have given all the information requested from me. I have >> tried different things like mounting filesystems with async even though >> I know that it is not the problem just to make you guys happy. Now you >> say that my attitude is not helping the resolution? What makes you say so? > > The information you have been giving has been sparse and confusing. I > still feel like I am pulling teeth. > >> Other people have been using it with dual-processor systems with 2-3GB >> of memory and raid setups. I would hardly consider this 'similar' >> conditions. BackupPC is very inefficent at handling files so you guys >> have to use very fast hardware solutions to speed it up. So you are >> covering up the problem from another side and say that there is no problem. > > Others have posted in the same thread saying that they have much > slower hardware which runs faster than yours. One of my servers > appears to be similar to yours but it has 4x the memory and you still > haven't said what CPU is in your backuppc server. > > BTW, I think BackupPC is very efficient at handling files. > >>> I'd suggest you use whatever tools are available under your chosen OS (I >>> think it is freebsd), to measure: >>> 1) CPU utilisation on both the client being backed up, and the backuppc >>> server >> I have already given this information the machines are almost idle. > > If you are using rsync, you should see a good spike in client IO when > backups start. You had not given the information before now. > >>> 3) Disk utilisation/bandwidth on both client and server >> I have sent this information also. I didnt send it on client actually >> but server is almost idle diskwise. The main disk load is on the server. > > The data you sent for your server is lacking. More detail is needed. > >>> Also, if you are using tar as your transfer method, then I'd suggest you >>> try a 'manual' backup something like this (note, this command line is >>> wrong, but you will need to adjust to get the right one). >> This is a good idea, I will try it later and return back to you. (even >> though I am using rsync) > > Good. > >>> Of course, if you are using rsync, then do the same test with rsync >>> data, but I'm sure it has already been said that you should use tar for >>> your case with lots of small files. >>> >>> Just my 0.02c worth.... >> I will do the test with both actually, I will copy all files from the >> client to server using both rsync and tar and return back the results. > > Good. Please provide detailed information of your tests that is easy > to read. Also provide detailed system load information as well. > > You earlier said you were going to increase memory on the server, are > you still planning on doing that? > > Perhaps it would be helpful if you summarized all your data in one email. > > What types of files are being backed up? I wonder if there is > something about them which is triggering this performance issue... > > http://backuppc.sourceforge.net/faq/BackupPC.html#some_design_issues > > If a lot of files are ending up under the same pool directory, that > could definitely trigger some bad behavior, but given what we've seen > of your workload, I would expect a large number of identical files to > be backed up. > > What is the largest directory under your pool/cpool directory? Are > there any directories which have files with a lot of underscores in > them? > > -Dave > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys-and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/backuppc-users > http://backuppc.sourceforge.net/ ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
