Re: Rsync Throughput issues
You might try putting the rsycd (server) on the source. rsync as a server is really designed for read-only. The read-write support is an afterthough, and very flaky. Martin, Tridge, Wayne: correct me if that's not true any more. Make sure you're not using --bwlimit=, unless you need to parcel your bandwidth (In my situation, I do). Don't use -H, unless you actually use multiple hard links to files, as that adds tremendous overhead on large filesystems. If the files are nfs mounted, use -W, else you will read both files completely, before you transfer anything... might as well just send it if datestamp or size differs. I doubt ssh was giving you any slowdown, unless you're cpu-limited on one or the other box. If you don't need the security, and policy permits, plain rsh transport might be a bit faster. Tim Conway [EMAIL PROTECTED] 303.682.4917 Philips Semiconductor - Longmont TC 1880 Industrial Circle, Suite D Longmont, CO 80501 Available via SameTime Connect within Philips Available as n9hmg on AIM perl -e 'print pack(, 19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970), .\n ' There are some who call me Tim? Ben Ricker [EMAIL PROTECTED]@lists.samba.org on 10/08/2001 06:50:02 PM Sent by: [EMAIL PROTECTED] To: [EMAIL PROTECTED] cc: (bcc: Tim Conway/LMT/SC/PHILIPS) Subject: Rsync Throughput issues Classification: I am using Rsync between a Redhat Linux box and an AIX RS600. We have a about 30gb of database we need to sync to a backup server. Sounds good, right? The problem is that Rsync is so slow when we do the initial dump. We have files that are 1 - 5gb. It takes around 14-20 hours to Rsync the file structure, which means our daily backup will miss that days data because we are waiting for the data to rsync. At first we were using ssh as the agent but we thought that might be slowing us down. We then tried to use rsync as a server and push the data to the module/zone of the server. Consistently, it takes roughly 2 minutes to transfer 10meg of data I thought it was a network problem so we tried ftp. A 10 mb file took 90% the time (about 20 seconds. This is on a 100bit Full Duplex switched network with both boxes on the SAME subnet. Am I missing something? Should I expect these speeds? Could there be something on the App switch throttling rsync for some reason? Any thought will help, Ben Ricker System Administrator Wellinx.com
Re: Rsync Throughput issues
Ben Ricker wrote: I am using Rsync between a Redhat Linux box and an AIX RS600. We have a about 30gb of database we need to sync to a backup server. Sounds good, right? The problem is that Rsync is so slow when we do the initial dump. We have files that are 1 - 5gb. It takes around 14-20 hours to Rsync the file structure, which means our daily backup will miss that days data because we are waiting for the data to rsync. How many files are in the tree? Are all the files 1-5G? Are the source files on new/fast disk? For older systems, the 100Mbit network is often faster than disk. I have a 40G tree with 1 million files in it. I just killed the tree on a development box and restarted a rsync to it. I think it might take 10+ hours mainly because of how many files are in the tree -- not so much because of the total size. I'll let you know. I know it takes about 2 hours just to verify that all 1 million files are in sync even when there are no changes to the tree. At first we were using ssh as the agent but we thought that might be slowing us down. We then tried to use rsync as a server and push the data to the module/zone of the server. Consistently, it takes roughly 2 minutes to transfer 10meg of data That seems way too long. Sounds like a reverse dns lookup failure. Long shot debug question -- Do you have dns/nis/files all working properly? Sometimes reverse dns lookups can take a few minutes to fail or time out and thus affect the overall speed of transfers. Repeated reverse dns failures can add up fast. I thought it was a network problem so we tried ftp. A 10 mb file took 90% the time (about 20 seconds. This is on a 100bit Full Duplex switched network with both boxes on the SAME subnet. I think you still have other problems. A ftp of a 10Mbyte file takes my boxes about 1s on a 100Mbit wire. Be sure to try ftp both get and put. Oftentimes a duplex mismatch or speed problem will only affect traffic in one direction and not the other. I think you might have some network issues. Do a ifconfig on the linux box to see what sort of errors and collisions you are seeing. Am I missing something? Should I expect these speeds? Could there be something on the App switch throttling rsync for some reason? Any thought will help, Ben Ricker System Administrator Wellinx.com
Re[2]: Rsync Throughput issues
(Courtesy copy sent directly to Ben) Eric Whiting [EMAIL PROTECTED] wrote: Ben Ricker wrote: I am using Rsync between a Redhat Linux box and an AIX RS600. We have a about 30gb of database we need to sync to a backup server. Sounds good, right? The problem is that Rsync is so slow when we do the initial dump. We have files that are 1 - 5gb. It takes around 14-20 hours to Rsync the file structure, which means our daily backup will miss that days data because we are waiting for the data to rsync. ... I've got 2 areas that I sync across a T-1, as follows: Number of files: 8122 Number of files transferred: 626 Total file size: 2,101,157,198 bytes Total transferred file size: 1,855,602,142 bytes Literal data: 2,851,745 bytes Matched data: 1,852,750,397 bytes File list size: 197,222 Total bytes written: 10,194,719 Total bytes read: 487,766 wrote 10,194,719 bytes read 487,766 bytes 6,841.17 bytes/sec total size is 2,101,157,198 speedup is 196.69 Number of files: 13,502 Number of files transferred: 0 Total file size: Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 304636 Total bytes written: 2741 Total bytes read: 304,713 wrote 2,741 bytes read 304,713 bytes 3,967.15 bytes/sec total size is The above 2 things took 27 minutes 20 seconds total. Remeber again, this is across a T1 (full 1.5 Mbit T1), not 10Mbit or 100Mbit. Without ssh or anything else. At first we were using ssh as the agent but we thought that might be slowing us down. We then tried to use rsync as a server and push the data to the module/zone of the server. Consistently, it takes roughly 2 minutes to transfer 10meg of data That seems way too long. ... I thought it was a network problem so we tried ftp. A 10 mb file took 90% the time (about 20 seconds. This is on a 100bit Full Duplex 20 seconds for 10 MB (I'm assuming you meant MegaByte, not MegaBit ;-) is way slow. ESPECIALLY for ftp. A long time ago I ran an ftp test on a 10 Mbit link between 2 Sun Sparc2 machines. I was able to use 99.9% of the bandwidth (i.e. I got over 1 megaByte/sec transfer rates!). Put another way, using 2 sparcstation 2s on a slightly occupied 10Mbit network I should be able to transfer that file in 10 seconds. One word of warning - FTP transfers are way efficient, and can burn all the bandwidth up. Most other protocols are not nearly so efficient, and so you cannot get the speed from them. The same machines and all worked much slower doing rcp as transfer means (but I don't remember the numbers, this was what, 9 years ago?). So, whats the point? Its this: What's your network loading? Have you run anything (like Sun's SE performance toolkit (which unfortunatly only works on sun) to see if your network is working up to snuff? Going off in another direction: Do you have enough RAM? Programs running out of disk instead of out of ram are DEADLY slow... And the amount of data you have to worry about is non-trivial here... ;-) switched network with both boxes on the SAME subnet. I think you still have other problems. A ftp of a 10Mbyte file takes my boxes about 1s on a 100Mbit wire. probably due as much to file transfer rates as network speeds! What happens if you do the transfer twice? (So you use the cache for the file data instead of having to read it from the disk - assuming that you're not memory limited!) Be sure to try ftp both get and put. Oftentimes ... I think you might have some network issues. I'm going to have to agree. DNS is one possiblility, but that slow ftp reported tends to make me think its something else. Do a ifconfig on the linux box to see what sort of errors and collisions you are seeing. And on the AIX box also! What's the loading on that AIX box like? Can you try rsync-ing between 2 linux boxes? Between 2 aix boxes? (First, try ftp transfers and see what kind of performance you get, once THAT is fixed then see how well rsync works between the 'correct' machines, before you worry about trying between other than the machines you plan to use in the final setup) Am I missing something? Probably (not much help, eh? ;-) Should I expect these speeds? No, this is slow. Could there be something on the App switch throttling rsync for some reason? Umm, I think I missed something. What is the app switch? Unless you mean the network switching hub - maybe. Is is SNMP manageable? How much bandwidth is it currently handling? How much CAN it handle? One note on switching hubs - I have one at home, and I've noticed a really weird thing - when transferring data from my laptop (10Mbit PCMCIA on a P133 laptop) to my 'big machine' (Celeron 450MHz with 100Mbit ethernet, lotsa ram, etc) through the switching hub I get collisions(!!!) between the laptop and the hub. Scratch head time, and one of these days I'm going to have to figure out how and why... The point is - don't assume your hub does all its supposed to do and
Re: Re[2]: Rsync Throughput issues
Thanks for the advice. I ran an rsync test, using SSH as the transport agent, and I got the following throughput: wrote 178895529 bytes read 32 bytes 499011.33 bytes/sec total size is 1075112361 speedup is 6.01 It took the same amount of time as the ftp transfer did (about 6 minutes). So, it looks like the problem is with Rsync for AIX. We use the binary install. Someone suggested compiling rsync from source. Has anyone else run rsync on AIX? Ben Ricker System Administrator Wellinx.com On Tue, 2001-10-09 at 12:01, Rusty Carruth wrote: (Courtesy copy sent directly to Ben) Eric Whiting [EMAIL PROTECTED] wrote: Ben Ricker wrote: I am using Rsync between a Redhat Linux box and an AIX RS600. We have a about 30gb of database we need to sync to a backup server. Sounds good, right? The problem is that Rsync is so slow when we do the initial dump. We have files that are 1 - 5gb. It takes around 14-20 hours to Rsync the file structure, which means our daily backup will miss that days data because we are waiting for the data to rsync. ... I've got 2 areas that I sync across a T-1, as follows: Number of files: 8122 Number of files transferred: 626 Total file size: 2,101,157,198 bytes Total transferred file size: 1,855,602,142 bytes Literal data: 2,851,745 bytes Matched data: 1,852,750,397 bytes File list size: 197,222 Total bytes written: 10,194,719 Total bytes read: 487,766 wrote 10,194,719 bytes read 487,766 bytes 6,841.17 bytes/sec total size is 2,101,157,198 speedup is 196.69 Number of files: 13,502 Number of files transferred: 0 Total file size: Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 304636 Total bytes written: 2741 Total bytes read: 304,713 wrote 2,741 bytes read 304,713 bytes 3,967.15 bytes/sec total size is The above 2 things took 27 minutes 20 seconds total. Remeber again, this is across a T1 (full 1.5 Mbit T1), not 10Mbit or 100Mbit. Without ssh or anything else. At first we were using ssh as the agent but we thought that might be slowing us down. We then tried to use rsync as a server and push the data to the module/zone of the server. Consistently, it takes roughly 2 minutes to transfer 10meg of data That seems way too long. ... I thought it was a network problem so we tried ftp. A 10 mb file took 90% the time (about 20 seconds. This is on a 100bit Full Duplex 20 seconds for 10 MB (I'm assuming you meant MegaByte, not MegaBit ;-) is way slow. ESPECIALLY for ftp. A long time ago I ran an ftp test on a 10 Mbit link between 2 Sun Sparc2 machines. I was able to use 99.9% of the bandwidth (i.e. I got over 1 megaByte/sec transfer rates!). Put another way, using 2 sparcstation 2s on a slightly occupied 10Mbit network I should be able to transfer that file in 10 seconds. One word of warning - FTP transfers are way efficient, and can burn all the bandwidth up. Most other protocols are not nearly so efficient, and so you cannot get the speed from them. The same machines and all worked much slower doing rcp as transfer means (but I don't remember the numbers, this was what, 9 years ago?). So, whats the point? Its this: What's your network loading? Have you run anything (like Sun's SE performance toolkit (which unfortunatly only works on sun) to see if your network is working up to snuff? Going off in another direction: Do you have enough RAM? Programs running out of disk instead of out of ram are DEADLY slow... And the amount of data you have to worry about is non-trivial here... ;-) switched network with both boxes on the SAME subnet. I think you still have other problems. A ftp of a 10Mbyte file takes my boxes about 1s on a 100Mbit wire. probably due as much to file transfer rates as network speeds! What happens if you do the transfer twice? (So you use the cache for the file data instead of having to read it from the disk - assuming that you're not memory limited!) Be sure to try ftp both get and put. Oftentimes ... I think you might have some network issues. I'm going to have to agree. DNS is one possiblility, but that slow ftp reported tends to make me think its something else. Do a ifconfig on the linux box to see what sort of errors and collisions you are seeing. And on the AIX box also! What's the loading on that AIX box like? Can you try rsync-ing between 2 linux boxes? Between 2 aix boxes? (First, try ftp transfers and see what kind of performance you get, once THAT is fixed then see how well rsync works between the 'correct' machines, before you worry about trying between other than the machines you plan to use in the final setup) Am I missing something? Probably (not much help, eh? ;-) Should I expect these speeds? No, this is slow. Could there be something on the App switch throttling rsync for some reason? Umm, I think I
Re: Re[2]: Rsync Throughput issues
On Tue, 2001-10-09 at 14:51, meg wrote: we use rsync on aix 4.1.5 (apple hardware). it builds very easily. do you have specific questions? goodluck! We are running on an RS6000 (H80 with single CPU and 1gb memory). We are also running a rather large Oracle database on it. I noticed that when rsync was running, there was a considerable I/O wait (at one point, it was at 98%!). We installed Rsync from a binary dist, not compiled. I assume by your use of build that you compiled from source? Ben Ricker System Administrator Wellinx.com
Rsync Throughput issues
I am using Rsync between a Redhat Linux box and an AIX RS600. We have a about 30gb of database we need to sync to a backup server. Sounds good, right? The problem is that Rsync is so slow when we do the initial dump. We have files that are 1 - 5gb. It takes around 14-20 hours to Rsync the file structure, which means our daily backup will miss that days data because we are waiting for the data to rsync. At first we were using ssh as the agent but we thought that might be slowing us down. We then tried to use rsync as a server and push the data to the module/zone of the server. Consistently, it takes roughly 2 minutes to transfer 10meg of data I thought it was a network problem so we tried ftp. A 10 mb file took 90% the time (about 20 seconds. This is on a 100bit Full Duplex switched network with both boxes on the SAME subnet. Am I missing something? Should I expect these speeds? Could there be something on the App switch throttling rsync for some reason? Any thought will help, Ben Ricker System Administrator Wellinx.com
