Re: rsync very slow, but not a network issue
Reini (and Steven), Okay, I get. This benchmark you introduced below compares rsync on two variables: copy speed locally vs over the network, and copy speed for copies onto indentical target files vs different target files. And your benchmark showed that rsync was faster for copies onto pre-existing target files -- just as rsync is supposed to be. I just ran the same benchmark on my side and I get exactly the opposite results. For what it's worth, here's what I got. I started with two different binary files (based on different mp3's) each of size 434006. %[local copy, target file is different ] %time rsync test1.mp3 test2.mp3 real 0m6.422s user 0m0.430s sys 0m1.431s %[local copy, target file is the same ] %time rsync test1.mp3 test1-same.mp3 real 0m6.523s user 0m0.300s sys 0m1.261s %cp test2.bak test2.mp3 %[network copy, a pure copy with scp for benchmark] %time scp test2.mp3 $TEST: 260.6KB/s (calculated average) real 0m16.257s user 0m0.510s sys 0m1.091s %[network copy, target file is different ] %time rsync test1.mp3 $TEST:test2.mp3 real 0m13.797s user 0m1.421s sys 0m1.250s % (network copy, files are 100% the same) % (repeated twice to guarantee target file is there) %time rsync test2.mp3 $TEST:test2.bak real 0m28.329s user 0m1.390s sys 0m4.766s %rsync --version rsync version 2.6.2 protocol version 28 ssh $TEST rsync --version rsync version 2.6.2 protocol version 28 $ ssh -V OpenSSH_3.8.1p1, OpenSSL 0.9.7d 17 Mar 2004 $ ssh $TEST ssh -V OpenSSH_3.7.1p1, SSH protocols 1.5/2.0, Open SSL 0.9.7d 17 Mar 2004 (but ssh -v reports TEST's sshd server is responding with OpenSSH_3.6.1p1+CAN-2003-0693, SSH protocols 1.5/2.0, Open SSL 0x0090702f ) $ uname -r 1.5.10(0.116/4/2) As you can see, my network copy onto an identical waiting file is taking almost twice as long as a network copy onto a _different_ waiting file, and is also substantially slower than a pure copy. So I'm experiencing a net rsync slowdown rather than a speedup. Is it the processor? Thorsten Kampe And Steven Hartland suggested the problem is in effect that my computer is too slow, and that I should turn off all checksumming. I seem to have more bandwidth to my TEST machine than you, so I suppose my processor might be the bottleneck for me but not for you, which would explain our different results. How fast was your machine? Or Is it cygwin? I notice that the enormous slowdown for the network copy onto an identical target is all appearing in the 'real' component of the time's timing report, and that there's a bit of extra time in 'sys' but other than that there's nothing to account for it. I'm not sure how to interpret that. Would cygwin-specific delays appear in 'sys'? In other words, does this mean the checksums are using up all the time, and confirm that my processors are too slow to benefit from rsync? Best wishes, Alexis Reini Urban wrote: So check if the net overhead in the cygwin version is broken? Could be easily tested out with two local files: head -c 210 /dev/random test1.mp3 head -c 210 /dev/random test2.mp3 cp test1.mp3 test1-same.mp3 cp test2.mp3 test2.bak time rsync test1.mp3 test2.mp3 real0m0.518s user0m0.201s sys 0m0.171s time rsync test1.mp3 test1-same.mp3 real0m0.524s user0m0.154s sys 0m0.263s RSYNC_RSH=ssh TEST=othermachine cp test2.bak test2.mp3 time scp test2.mp3 $TEST: 63.2KB/s real0m36.619s user0m0.170s sys 0m0.233s # 100% different $ time rsync test1.mp3 $TEST:test2.mp3 Server is very old version of rsync, upgrade recommended. real0m37.162s user0m0.388s sys 0m0.202s # 100% same $ time rsync test2.mp3 $TEST:test2.bak Server is very old version of rsync, upgrade recommended. real0m7.298s user0m1.201s sys 0m2.217s 6x faster than scp, with same data. 0.5s (1.4%) slower than scp, with complete random data. $ rsync --version rsync version 2.6.2 protocol version 28 $ ssh $TEST rsync --version rsync version 2.4.6 protocol version 24 $ ssh -V OpenSSH_3.8.1p1, OpenSSL 0.9.7d 17 Mar 2004 $ ssh $TEST ssh -V OpenSSH_3.7.1p2, SSH protocols 1.5/2.0, OpenSSL 0.9.6c 21 dec 2001 $ uname -r 1.5.10(0.116/4/2) -- Reini Urban http://xarch.tu-graz.ac.at/home/rurban/ I see the final result you -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: rsync very slow, but not a network issue
Hello all, Thanks for the help so far, on this issue with the mysterious non-network based rsync slowdown. To recap... Brian Dessent wrote: Reini Urban wrote: Alexis Gallagher wrote: When the file is alredy there, rsync reports a speedup of about 70. (When the file is not already there, the speed up is 1, of course.) I am running rsync over ssh with pre-generated keys installed in my .ssh directories. This is a binary MP3. rsync (as diff) is not good in checking binary diffs. Please try it with a typical text file, where the patch is smaller than the source. That is not correct at all, and rsync would be much less useful if it were true. rsync works with arbitrary binary data just as well as it does with text. In any event I checked Reini's theory using a 5 MB text file that consists only of x's and newlines. So unlike an MP3, this file is non-binary and extremely compressible. And what I observed was as follows: scp: 348800 bytes/sec rsync (file not there): 303075 bytes/sec, with speedup of 1 rsync (file already there): 733 bytes/sec, with speedup of 70 This is pretty much exactly the same problem as I was having with the MP3 file. And I'm still quite puzzled. Cheers, Alexis -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: rsync very slow, but not a network issue
Hi Steve, Steven Hartland wrote: Is this not because its showing you the network transfer rate i.e. spending all its time doing compression and therefore not having to do actual network transfers? How long did each test take? I just performed the test again, this time timing the transfers with a stopwatch. (This time I was transferring a text file that consisted of 5 million zeros, which is about 5MB.) Here's what I found. scp: 19 seconds rsync (no there): 17 seconds rsync (already there): 93 seconds So it's taking much longer in real time when the file is already there, which is exactly the situation where rsync is supposed to accelerate teh transfer. The cygwin machine is a Pentium III 1Ghz, and the eMac is a bit faster I believe. This should be fast enough that it's not bottlenecking on the hash computation, I think. Cheers, Alexis -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: rsync very slow, but not a network issue
Ok, Now this is progress. When I use '--size-only' then I do see a speedup. The 'transfer' is virtually instantaneous on the stopwatch, and 'rsync --stats' reports a speed of a mere 25 bytes/sec but with a reported speedup of 78125. I can see why this works, but I remain confused why I should need to resort to it. This is basically dispensing with the rsync algorithm, no? The cygwin machine is a Pentium III, 966 Mhz, 512MB of RAM. And the OSX machine is a 1 Ghz PowerPC G4 with 256MB of RAM. I can't believe everyone would use rsync as much as they do if it were not useful on machines of such specifications. Is there a way to benchmark its hashing algorithm on both sides? Maybe the rsync process is getting insufficient priority on one side of the transfer? I remain puzzled, Alexis Gallagher Steven Hartland wrote: Alexis Gallagher wrote: So it's taking much longer in real time when the file is already there, which is exactly the situation where rsync is supposed to accelerate teh transfer. The cygwin machine is a Pentium III 1Ghz, and the eMac is a bit faster I believe. This should be fast enough that it's not bottlenecking on the hash computation, I think. Are you using --size-only? Depending on the processor the check of the file chunks can be slower. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
rsync very slow, but not a network issue
Hello, I am finding that rsync+ssh is giving extremely slow file transfers. But this slowdown is hitting not when it needs to send data over the network, but when it applies the rsync algorithm which is supposed to be faster than sending all the data over the network. This is very puzzling. For a benchmark, I tried transfering a 5.4 MB mp3 file three different ways. Here were my results, as reported by 'scp -v' and 'rsync --progress --stats': scp: 311000 B/s rsync (file not there): 309000 B/s rsync (file already there): 741 B/s When the file is alredy there, rsync reports a speedup of about 70. (When the file is not already there, the speed up is 1, of course.) I am running rsync over ssh with pre-generated keys installed in my .ssh directories. The receiving machine is running OS X 10.3.4. It has OpenSSH_3.6.1p1 (as judged from the 'ssh -v' output) and the fink installation of rsync2.6.2. The sending machine is running WinXPProSP1, with rsync 2.6.2 and OpenSSH_3.8.1p1. FWIW, I am attaching my cygcheck.out, as suggested in the webpage on using this list. Much obliged for any help, Alexis Cygwin Configuration Diagnostics Current System Time: Mon Jul 12 17:20:09 2004 Windows XP Professional Ver 5.1 Build 2600 Service Pack 1 Path: E:\cygwin\usr\local\bin E:\cygwin\bin E:\cygwin\bin E:\cygwin\usr\X11R6\bin e:\WINDOWS\system32 e:\WINDOWS e:\WINDOWS\system32\WBEM e:\Program Files\3dsmax5\backburner2\ e:\Program Files\Common Files\Autodesk Shared\ e:\jwsdp-1.3\jwsdp-shared\bin e:\Program Files\SSH Communications Security\SSH Secure Shell e:\jwsdp-1.3\bin e:\j2sdk1.4.2\bin e:\Python23 e:\jwsdp-1.3\apache-ant\bin Output from E:\cygwin\bin\id.exe (nontsec) UID: 1003(alexis) GID: 513(None) 513(None) Output from E:\cygwin\bin\id.exe (ntsec) UID: 1003(alexis) GID: 513(None) 0(root) 513(None) 544(Administrators) 545(Users) SysDir: E:\WINDOWS\System32 WinDir: E:\WINDOWS HOME = `E:\cygwin\home\alexis' MAKE_MODE = `unix' PWD = `/home/alexis/iTunesMusic/Jay-Z + DJ Danger Mouse/The Grey Album' USER = `alexis' ALLUSERSPROFILE = `E:\Documents and Settings\All Users' ANT_HOME = `e:\jwsdp-1.3\apache-ant' APPDATA = `E:\Documents and Settings\alexis\Application Data' CLIENTNAME = `Console' COMMONPROGRAMFILES = `E:\Program Files\Common Files' COMPUTERNAME = `8YGZF01' COMSPEC = `E:\WINDOWS\system32\cmd.exe' CVS_RSH = `/bin/ssh' HOMEDRIVE = `E:' HOMEPATH = `\Documents and Settings\alexis' HOSTNAME = `8ygzf01' INFOPATH = `/usr/local/info:/usr/info:/usr/share/info:/usr/autotool/devel/info:/usr/autotool/stable/info:' JAVA_HOME = `E:\j2sdk1.4.2' JSF_HOME = `e:\jwsdp-1.3\jsf-1_0-beta' JWSDP_HOME = `e:\jwsdp-1.3' LOGONSERVER = `\\8YGZF01' MANPATH = `/usr/local/man:/usr/man:/usr/share/man:/usr/autotool/devel/man::/usr/ssl/man:/usr/X11R6/man' NUMBER_OF_PROCESSORS = `1' OLDPWD = `/home/alexis/iTunesMusic/Jay-Z + DJ Danger Mouse' OS = `Windows_NT' PATHEXT = `.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH' PROCESSOR_ARCHITECTURE = `x86' PROCESSOR_IDENTIFIER = `x86 Family 6 Model 8 Stepping 6, GenuineIntel' PROCESSOR_LEVEL = `6' PROCESSOR_REVISION = `0806' PROGRAMFILES = `E:\Program Files' PROMPT = `$P$G' PS1 = `\[\033]0;\w\007 [EMAIL PROTECTED] \[\033[33m\w\033[0m\] $ ' PYTHONPATH = `E:\Python23\alexispackages\;E:\Python23\alexispackages\pyblogger-1.5' RSYNC_RSH = `ssh' SESSIONNAME = `Console' SHLVL = `1' SYSTEMDRIVE = `E:' SYSTEMROOT = `E:\WINDOWS' TEMP = `E:\DOCUME~1\alexis\LOCALS~1\Temp' TERM = `cygwin' TMP = `E:\DOCUME~1\alexis\LOCALS~1\Temp' USERDOMAIN = `8YGZF01' USERNAME = `alexis' USERPROFILE = `E:\Documents and Settings\alexis' WINDIR = `E:\WINDOWS' _ = `/usr/bin/cygcheck' POSIXLY_CORRECT = `1' HKEY_CURRENT_USER\Software\Cygnus Solutions HKEY_CURRENT_USER\Software\Cygnus Solutions\Cygwin HKEY_CURRENT_USER\Software\Cygnus Solutions\Cygwin\mounts v2 HKEY_CURRENT_USER\Software\Cygnus Solutions\Cygwin\Program Options HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2 (default) = `/cygdrive' cygdrive flags = 0x0022 HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2\/ (default) = `E:\cygwin' flags = 0x000a HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2\/usr/bin (default) = `E:\cygwin/bin' flags = 0x000a HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2\/usr/lib (default) = `E:\cygwin/lib' flags = 0x000a HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2\/usr/X11R6/lib/X11/fonts (default) = `E:\cygwin\usr\X11R6\lib\X11\fonts' flags = 0x000a HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\Program Options a: fd N/AN/A c: fd N/AN/A d: cd N/AN/A e: hd NTFS 57207Mb
