Re: [BackupPC-users] Is there a speed setting?
Hi On Sun, 30 Aug 2009 01:30:25 -0400, Jeffrey J. Kosowsky backu...@kosowsky.org wrote: well, under normal conditions (ie: unix), i would say rsyncd is faster, as doesnt have the ssh overhead... But if bandwidth is your limitation (which it frequently is on any decent system) then rsync+ssh may be faster if there is a lot of fresh data to be transferred due to the compression of ssh transfers. bandwidth is the limit if you can both encrypt and compress faster than the network bandwidth... also, the bonus of the compression depends of the data and can also be gained by the normal rsync-rsyncd... backuppc rsync cant because doesnt support compression, so the main question might be how much the data compress Cya higuita -- Naturally the common people don't want war... but after all it is the leaders of a country who determine the policy, and it is always a simple matter to drag the people along, whether it is a democracy, or a fascist dictatorship, or a parliament, or a communist dictatorship. Voice or no voice, the people can always be brought to the bidding of the leaders. That is easy. All you have to do is tell them they are being attacked, and denounce the pacifists for lack of patriotism and exposing the country to danger. It works the same in every country. -- Hermann Goering, Nazi and war criminal, 1883-1946 signature.asc Description: PGP signature -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there a speed setting?
higuita wrote at about 01:28:38 +0100 on Tuesday, August 25, 2009: Hi all On Mon, 24 Aug 2009 15:05:55 -0500, Jim Leonard trix...@oldskool.org wrote: This brings up a good point: What is faster, using rsyncd or rsync+ssh? You'd think that rsyncd would be faster, yet I'm not getting times anywhere near these. well, under normal conditions (ie: unix), i would say rsyncd is faster, as doesnt have the ssh overhead... in windows i still didnt test... But if bandwidth is your limitation (which it frequently is on any decent system) then rsync+ssh may be faster if there is a lot of fresh data to be transferred due to the compression of ssh transfers. Higuita: How many of these are windows clients and how many non-windows? all are MacOSX or Linux, i will try to configure some windows machines during the next days. We already have a internal rsync script for windows, but its hard to manage and i finally manage to get a little more space for backuppc to migrate some windows to it and test how all works. i will report when i have more info cya higuita -- Naturally the common people don't want war... but after all it is the leaders of a country who determine the policy, and it is always a simple matter to drag the people along, whether it is a democracy, or a fascist dictatorship, or a parliament, or a communist dictatorship. Voice or no voice, the people can always be brought to the bidding of the leaders. That is easy. All you have to do is tell them they are being attacked, and denounce the pacifists for lack of patriotism and exposing the country to danger. It works the same in every country. -- Hermann Goering, Nazi and war criminal, 1883-1946 [GNUPG:] ERRSIG AEFD636B3B0C72B3 17 2 01 1251160118 9 [GNUPG:] NO_PUBKEY AEFD636B3B0C72B3 -- -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there a speed setting?
higuita wrote: Please note that this are rsync backups, not rsyncd, and my backuppc ssh settings are this: This brings up a good point: What is faster, using rsyncd or rsync+ssh? You'd think that rsyncd would be faster, yet I'm not getting times anywhere near these. Higuita: How many of these are windows clients and how many non-windows? -- Jim Leonard (trix...@oldskool.org)http://www.oldskool.org/ Help our electronic games project: http://www.mobygames.com/ Or check out some trippy MindCandy at http://www.mindcandydvd.com/ A child borne of the home computer wars: http://trixter.wordpress.com/ -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there a speed setting?
Hi all On Mon, 24 Aug 2009 15:05:55 -0500, Jim Leonard trix...@oldskool.org wrote: This brings up a good point: What is faster, using rsyncd or rsync+ssh? You'd think that rsyncd would be faster, yet I'm not getting times anywhere near these. well, under normal conditions (ie: unix), i would say rsyncd is faster, as doesnt have the ssh overhead... in windows i still didnt test... Higuita: How many of these are windows clients and how many non-windows? all are MacOSX or Linux, i will try to configure some windows machines during the next days. We already have a internal rsync script for windows, but its hard to manage and i finally manage to get a little more space for backuppc to migrate some windows to it and test how all works. i will report when i have more info cya higuita -- Naturally the common people don't want war... but after all it is the leaders of a country who determine the policy, and it is always a simple matter to drag the people along, whether it is a democracy, or a fascist dictatorship, or a parliament, or a communist dictatorship. Voice or no voice, the people can always be brought to the bidding of the leaders. That is easy. All you have to do is tell them they are being attacked, and denounce the pacifists for lack of patriotism and exposing the country to danger. It works the same in every country. -- Hermann Goering, Nazi and war criminal, 1883-1946 signature.asc Description: PGP signature -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there a speed setting?
Hi all here is my example, my top speed backups: icorreiadleite 4 3.4 7.1834.02 3 0.4 0.4 idleidle antonio-trind-l ajtrindade 46.1 10.90 34.17 15 0.1 0.1 idleidle drusso dleite 5 4.5 85.71 34.93 15 0.5 0.5 idleidle ltrindade dleite 4 2.1 94.77 35.11 15 0.1 0.1 idledone aacesar dleite 4 5.1 42.96 39.85 15 1.0 1.0 idleidle tvanez dleite 4 4.1 103.76 41.17 15 0.1 0.1 idledone jferreira dleite 5 4.5 13.72 41.21 14 18.34.5 idleno ping (no ping response) aprafaeldleite 5 4.1 50.43 45.10 15 2.1 2.1 idleno ping (no ping response) poliveira dleite 5 2.6 126.52 46.21 10 0.6 0.6 idleidle spimentadleite 4 8.1 62.63 46.43 15 3.1 3.1 idleno ping (no ping response) jppinto dleite 5 5.0 1374.13 55.76 12 3.0 3.0 idleno ping (no ping response) I even checked the first full backup of recent added machines and its giving also 20-40MB/s transfers. Please note that this are rsync backups, not rsyncd, and my backuppc ssh settings are this: Protocol 2 Ciphers arcfour,blowfish-cbc,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc,cast128-cbc BatchMode yes Compression no i have several HDs for backuppc to sustain this transfers rates. maybe you are using bad ssh settings if using rsync? good luck higuita -- Naturally the common people don't want war... but after all it is the leaders of a country who determine the policy, and it is always a simple matter to drag the people along, whether it is a democracy, or a fascist dictatorship, or a parliament, or a communist dictatorship. Voice or no voice, the people can always be brought to the bidding of the leaders. That is easy. All you have to do is tell them they are being attacked, and denounce the pacifists for lack of patriotism and exposing the country to danger. It works the same in every country. -- Hermann Goering, Nazi and war criminal, 1883-1946 signature.asc Description: PGP signature -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there a speed setting?
Hi I was reading this thread and here I have some figures: Host User #Full Full Age (days) Full Size (GB) Speed (MB/s) #Incr Incr Age (days) Last Backup (days) State Last attempt ibm - zseries admin 3 26.7 3.52 3.36 1 5.7 5.7 idle idle martini-lap admin 2 26.6 50.77 9.91 1 35.6 26.6 idle no ping (no ping response) mt.linux.net admin 6 18.1 0.29 1.98 40 0.8 0.8 idle idle piteriano-dri admin 2 27.9 14.04 9.15 6 22.9 22.9 idle no ping (no ping response) zeta-jones admin 5 5.5 339.49 15.94 40 0.5 0.5 idle idle I've sereveral remote hosts zseries,mt.linux.net that do backups over the net, the others are running on the local network 100/1000Mb the martini-lap is over 100Mb/s as the piteriano, zeta-jones is on 1000Mb/s network. My hardware config on the backuppc server is: processor : 3 vendor_id : AuthenticAMD cpu family : 16 model : 2 model name : AMD Phenom(tm) 9750 Quad-Core Processor stepping: 3 cpu MHz : 2400.000 8GB ram SuSE 11.0 x86_64 Backuppc 3.1 Software raid 5 with 3 disks (7.2K rpm) and software raid 0 with 2 disks (7.2Krpm) about that there is a lvm layer. cheers Pedro On Wednesday 19 August 2009 00:46:24 Jim Leonard wrote: Holger Parplies wrote: ah, so you're actually having a problem. Up to this point I wasn't sure if you weren't just misinterpreting some figures. No, he and I are seeing the same thing -- File::RsyncP is a real problem. I get decent transfers with actual rsync, but File::RsyncP has some serious design issues (see my other post with profiling information titled File::RsyncP issues). Is the author of that module (Craig Barratt) still around and/or maintaining it? If anyone is getting more than 10MB/s out of BackupPC rsyncd transfers, I would be quite surprised (and would like to know what the backup hardware was). -- -- Pedro M. S. Oliveira Pólo Tecnológico de Lisboa IT ConsultantEstrada do Paço do Lumiar, Lote 1 Email: pedro.olive...@dri.pt Sala 14 – 1600-546 Lisboa URL: http://www.dri.pt http://www.linux-geex.com Telefone: +351 21 715 30 55 Fax: +351 21 715 30 57 -- -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there a speed setting?
Jim writes: I get decent transfers with actual rsync, but File::RsyncP has some serious design issues (see my other post with profiling information titled File::RsyncP issues). Is the author of that module (Craig Barratt) still around and/or maintaining it? Yes - I also responded to your off-list email. As you can see, pollChild is called a ridiculously large number of times, which is eating up nearly 70% of the CPU time trying to do a backup. This is extremely inefficient and completely explains why my backups are taking so long over rsync (the CPU spends most of it's time in pollChild). I don't think there is enough information to conclusively support those conclusions. Craig -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there a speed setting?
Les Mikesell wrote: You still haven't said whether this is the first run where the files are actually copied or not - if not you shouldn't expect much network activity. How long would it take the target to read all it's files? Something like 'time tar -cf - / | cat /dev/null' would be a reasonable test. (Don't do -cf /dev/null with gnutar because it will cheat and not read the files.) That would be the fastest an rsync run could possibly complete with the --ignore times option even if you don't transfer any data or create new files on the server. This is the initial backup of the server. I am only running 1 session at a time until I solve this bandwidth problem. Now if you tell me my hardware isn't fast enough, the BackupPC server is a dual Opteron 2.2 Ghz with 8 GB RAM and 24 300GB drives in a 3ware RAID5 array, it isn't. Either end can limit the speed. How many concurrent runs do you do? Also, you should expect much faster rates if you have a few large files than if you have millions of tiny ones. I am retrying the same server with --ignore-times but it doesn't seem to be any faster. -- Jeremy Mann jer...@biochem.uthscsa.edu University of Texas Health Science Center Bioinformatics Core Facility http://www.bioinformatics.uthscsa.edu Phone: (210) 567-2672 -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there a speed setting?
Jeremy Mann wrote: You still haven't said whether this is the first run where the files are actually copied or not - if not you shouldn't expect much network activity. How long would it take the target to read all it's files? Something like 'time tar -cf - / | cat /dev/null' would be a reasonable test. (Don't do -cf /dev/null with gnutar because it will cheat and not read the files.) That would be the fastest an rsync run could possibly complete with the --ignore times option even if you don't transfer any data or create new files on the server. This is the initial backup of the server. I am only running 1 session at a time until I solve this bandwidth problem. Now if you tell me my hardware isn't fast enough, the BackupPC server is a dual Opteron 2.2 Ghz with 8 GB RAM and 24 300GB drives in a 3ware RAID5 array, it isn't. Either end can limit the speed. How many concurrent runs do you do? Also, you should expect much faster rates if you have a few large files than if you have millions of tiny ones. I am retrying the same server with --ignore-times but it doesn't seem to be any faster. The --ignore-times option is only relevant where you have the matching file in the previous copy - and adding the option should make it slower. Without it, files that have matching directory entries (timestamp/length) are skipped without looking at the contents. With it, the files are compared with a block-checksum exchange which would make it slow down to disk read speed. For example, my laptop's 15G 'My Documents' folder took approximately 73 minutes for the initial copy, 18 minutes for a full where nothing changed, and an incremental where nothing changes just shows 0.0 elapsed time since it did nothing but read a few hundred directory entries. -- Les Mikesell lesmikes...@gmail.com -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there a speed setting?
Jeremy Mann wrote: I'm watching a live output of Ganglia showing network usage while the backups are going. Also simple math.. I just finished one full backup, 16 GB in 143 minutes. That's simply unacceptable for a full backup. You should be able to get faster transfer rates than that. I just checked my last full backup and it was running at 8.2MB/s on a 100Mb/s network (296GB full backup in 613 minutes). I can't help with solving your problem, but I can verify that BackupPC with rsync is definitely capable of backup speeds higher than what you are seeing. BTW - I am connecting to an rsync server with no encryption. If you connect through SSH, that may affect your transfer rates. -- Bowie -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there a speed setting?
Holger Parplies wrote: ah, so you're actually having a problem. Up to this point I wasn't sure if you weren't just misinterpreting some figures. No, he and I are seeing the same thing -- File::RsyncP is a real problem. I get decent transfers with actual rsync, but File::RsyncP has some serious design issues (see my other post with profiling information titled File::RsyncP issues). Is the author of that module (Craig Barratt) still around and/or maintaining it? If anyone is getting more than 10MB/s out of BackupPC rsyncd transfers, I would be quite surprised (and would like to know what the backup hardware was). -- Jim Leonard (trix...@oldskool.org)http://www.oldskool.org/ Help our electronic games project: http://www.mobygames.com/ Or check out some trippy MindCandy at http://www.mindcandydvd.com/ A child borne of the home computer wars: http://trixter.wordpress.com/ -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Is there a speed setting?
Just curious if there is a bandwidth speed setting for BackupPC because I'm not seeing a lot of bandwidth when the backups occur. All my servers are on gigE and I'm not even seeing 50Mbit speeds between the servers and backupPC server. -- Jeremy Mann jer...@biochem.uthscsa.edu University of Texas Health Science Center Bioinformatics Core Facility http://www.bioinformatics.uthscsa.edu Phone: (210) 567-2672 -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there a speed setting?
Jeremy Mann wrote: Just curious if there is a bandwidth speed setting for BackupPC because I'm not seeing a lot of bandwidth when the backups occur. All my servers are on gigE and I'm not even seeing 50Mbit speeds between the servers and backupPC server. There isn't one built in, but 50Mb is probably close to the disk speed you can sustain after buffers are full and you factor in the seeks for new directory entries. If you are using rsync, a lot of the elapsed time is spend comparing existing files without using a lot of bandwidth. If you want to reduce bandwidth, you can add the --bwlimit option to the rsync command. -- Les Mikesell lesmikes...@gmail.com -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there a speed setting?
Hi, On Mon, Aug 17, 2009 at 14:55, Jeremy Mannjer...@biochem.uthscsa.edu wrote: Just curious if there is a bandwidth speed setting for BackupPC because I'm not seeing a lot of bandwidth when the backups occur. All my servers are on gigE and I'm not even seeing 50Mbit speeds between the servers and backupPC server. 50Mbps is actually quite a lot, and it's probably close to the bottleneck of your disks. You should use iostat on client and server while backups are running to see if you're getting 100%util of the disks that are being backed up. In BackupPC's case, as it will transfer only the differences, it will end up reading from disk much more than it actually sends on the network, so using network bandwidth as a measure of backup speed will not be very accurate. HTH, Filipe -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there a speed setting?
Filipe Brandenburger wrote: 50Mbps is actually quite a lot, and it's probably close to the bottleneck of your disks. You should use iostat on client and server while backups are running to see if you're getting 100%util of the disks that are being backed up. In BackupPC's case, as it will transfer only the differences, it will end up reading from disk much more than it actually sends on the network, so using network bandwidth as a measure of backup speed will not be very accurate. 50Mbit (5 MB/s) is *not* very quick when I'm used to our old rsync scripts that utilized our full gigE network. -- Jeremy Mann jer...@biochem.uthscsa.edu University of Texas Health Science Center Bioinformatics Core Facility http://www.bioinformatics.uthscsa.edu Phone: (210) 567-2672 -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there a speed setting?
Jeremy Mann wrote: Filipe Brandenburger wrote: 50Mbps is actually quite a lot, and it's probably close to the bottleneck of your disks. You should use iostat on client and server while backups are running to see if you're getting 100%util of the disks that are being backed up. In BackupPC's case, as it will transfer only the differences, it will end up reading from disk much more than it actually sends on the network, so using network bandwidth as a measure of backup speed will not be very accurate. 50Mbit (5 MB/s) is *not* very quick when I'm used to our old rsync scripts that utilized our full gigE network. What operations are you watching to see these numbers? The only one where network bandwidth matters much is the initial copy of a new host. The rest of the time you are mostly doing comparisions. Backuppc will be slower than native rsync because it is in perl and because it is working with a compressed copy for the comparison. And perhaps you didn't use the --ignore-times option on the runs you are using for comparison. Backuppc does this on fulls and it will slow things down to the speed that the remote can read the whole disk for the checksum comparisons - but it gives you an integrity check on your pooled copy. -- Les Mikesell lesmikes...@gmail.com -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there a speed setting?
Jeremy Mann wrote: What operations are you watching to see these numbers? The only one where network bandwidth matters much is the initial copy of a new host. The rest of the time you are mostly doing comparisions. Backuppc will be slower than native rsync because it is in perl and because it is working with a compressed copy for the comparison. And perhaps you didn't use the --ignore-times option on the runs you are using for comparison. Backuppc does this on fulls and it will slow things down to the speed that the remote can read the whole disk for the checksum comparisons - but it gives you an integrity check on your pooled copy. I'm watching a live output of Ganglia showing network usage while the backups are going. Also simple math.. I just finished one full backup, 16 GB in 143 minutes. That's simply unacceptable for a full backup. You still haven't said whether this is the first run where the files are actually copied or not - if not you shouldn't expect much network activity. How long would it take the target to read all it's files? Something like 'time tar -cf - / | cat /dev/null' would be a reasonable test. (Don't do -cf /dev/null with gnutar because it will cheat and not read the files.) That would be the fastest an rsync run could possibly complete with the --ignore times option even if you don't transfer any data or create new files on the server. Now if you tell me my hardware isn't fast enough, the BackupPC server is a dual Opteron 2.2 Ghz with 8 GB RAM and 24 300GB drives in a 3ware RAID5 array, it isn't. Either end can limit the speed. How many concurrent runs do you do? Also, you should expect much faster rates if you have a few large files than if you have millions of tiny ones. -- Les Mikesell lesmikes...@gmail.com -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there a speed setting?
On Mon, Aug 17, 2009 at 5:38 PM, Les Mikeselllesmikes...@gmail.com wrote: Jeremy Mann wrote: I'm watching a live output of Ganglia showing network usage while the backups are going. Also simple math.. I just finished one full backup, 16 GB in 143 minutes. That's simply unacceptable for a full backup. That equates to 21 MB/second backup rate. The speed on the network is irrelevant because that is affected by which files actually get transfered. The backup rate is affected by the time to calculate checksums on the server and the client and if you have compression turned on for your pool files then it includes the time to compress / decompress files to/from the pool (hash collisions require complete file comparison). What is your processor doing during this activity. Remember that each backup session is single threaded so your speed will be gated by the speed of a single processor. -- Jonathan Craig -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there a speed setting?
Jon Craig wrote: On Mon, Aug 17, 2009 at 5:38 PM, Les Mikeselllesmikes...@gmail.com wrote: Jeremy Mann wrote: I'm watching a live output of Ganglia showing network usage while the backups are going. Also simple math.. I just finished one full backup, 16 GB in 143 minutes. That's simply unacceptable for a full backup. That equates to 21 MB/second backup rate. The speed on the network is irrelevant because that is affected by which files actually get transfered. The backup rate is affected by the time to calculate checksums on the server and the client and if you have compression turned on for your pool files then it includes the time to compress / decompress files to/from the pool (hash collisions require complete file comparison). What is your processor doing during this activity. Remember that each backup session is single threaded so your speed will be gated by the speed of a single processor. Huh? 21MB/s would give me 100MB (megabytes) every 5 seconds. I think you are confusing MB (megabytes) and Mb (megabits) per second. 21Mb is right (about 2 MB - megabytes per second) which is what I saw for this particular server. -- Jeremy Mann jer...@biochem.uthscsa.edu University of Texas Health Science Center Bioinformatics Core Facility http://www.bioinformatics.uthscsa.edu Phone: (210) 567-2672 -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there a speed setting?
Hi, Les Mikesell wrote on 2009-08-17 16:38:23 -0500 [Re: [BackupPC-users] Is there a speed setting?]: Jeremy Mann wrote: I'm watching a live output of Ganglia showing network usage while the backups are going. Also simple math.. I just finished one full backup, 16 GB in 143 minutes. That's simply unacceptable for a full backup. ah, so you're actually having a problem. Up to this point I wasn't sure if you weren't just misinterpreting some figures. You still haven't said whether this is the first run where the files are actually copied or not - if not you shouldn't expect much network activity. That is true. As has been explained, BackupPC rsync full backups read all files on both sides but just send block checksums over the network. With checksum caching you can cut down file reads on the BackupPC server side. Also remember that on the first run (and whenever content is transferred that is not yet in the pool) your data will need to be compressed. I have no idea what the sustained compression rate of your Opteron is. Due to the way BackupPC works, large growing (log) files may also slow things down. If that is your problem, you might consider turning off compression (though I'm not positive that will solve things). Also check your server status page for hash collisions (xxx repeated files with longest chain yyy). Finally, concurrent backups compete for disk seeks. 24 spindles sounds quite impressive (even with RAID5 ;-), but just the same, you shouldn't be running more than the one backup while you are trying to determine your bottleneck. Also, you should expect much faster rates if you have a few large files than if you have millions of tiny ones. Can you give us any information on that (file count, average file size)? Regards, Holger -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
