Re: rsync very slow, but not a network issue

2004-07-15 Thread Steven Hartland
- Original Message - 
From: "Alexis Gallagher" 

I've repeated your tests with a P4 2.4Ghz windows XP using SFU
and a PIII 800Mhz running FreeBSD 5.1


> %[local copy, target file is different ]
> %time rsync test1.mp3 test2.mp3
> 
> real 0m6.422s
> user 0m0.430s
> sys  0m1.431s

P4: time rsync test1.mp3 test2.mp3
real0m1.906s
user0m0.344s
sys 0m0.343s
P3:time rsync test1.mp3 test2.mp3
real0.380u 
user   0.233s 
sys0:00.81

> %[local copy, target file is the same ]
> %time rsync test1.mp3 test1-same.mp3
> 
> real 0m6.523s
> user 0m0.300s
> sys  0m1.261s

P4(SFU): time rsync test1.mp3 test2.mp3
real0m0.906s
user0m0.500s
sys 0m0.157s

P4(cygwin): time rsync test1.mp3 test2.mp3
real0m26.005s
user0m1.060s
sys 0m3.732s

P3:time rsync test1.mp3 test2.mp3
real0.326u 
user   0.264s
user   0:00.72

> %cp test2.bak test2.mp3
> %[network copy, a pure copy with scp for benchmark]
> %time scp test2.mp3 $TEST:
> 
> 260.6KB/s (calculated average)
> real 0m16.257s
> user 0m0.510s
> sys  0m1.091s

P4(SFU)->P3:
4.5MB/s
real0m3.188s
user0m1.312s
sys 0m0.438s
P4(cygwin)->P3:
2.6MB/s
real0m4.433s
user0m0.639s
sys 0m0.561s

P3->P4
5.3MB/s:
real   1.356u 
user   0.439s 
sys0:05.38

[network copy, ftp]
P3 -> P4
10.85 MB/s
P4(windows)->P3
10.85 MB/s
P4(cygwin)->P3
9.43 MB/s

> %[network copy, target file is different ]
> %time rsync test1.mp3 $TEST:test2.mp3
> 
> real 0m13.797s
> user 0m1.421s
> sys  0m1.250s

P3->P4
real 3.652u
user 0.411s
sys  0:05.39
P4(SFU)->P3
real0m4.516s
user0m1.968s
sys 0m1.093s
P4(cygwin)->P3
real0m7.722s
user0m1.357s
sys 0m1.030s

> 
> % (network copy, files are 100% the same)
> % (repeated twice to guarantee target file is there)
> %time rsync test2.mp3 $TEST:test2.bak
> 
> real 0m28.329s
> user 0m1.390s
> sys  0m4.766s
> 

P4: rsync --version
rsync  version 2.6.0  protocol version 27
OpenSSH_3.7.1p2, SSH protocols 1.5/2.0, OpenSSL 0.9.7c 30 Sep 2003
P3: rsync --version
rsync  version 2.6.0  protocol version 27
OpenSSH_3.6.1p1 FreeBSD-20030423, SSH protocols 1.5/2.0, OpenSSL 0x0090701f

> %rsync --version
> rsync version 2.6.2 protocol version 28
> ssh $TEST rsync --version
> rsync version 2.6.2 protocol version 28
> $ ssh -V
> OpenSSH_3.8.1p1, OpenSSL 0.9.7d 17 Mar 2004
> $ ssh $TEST ssh -V
> OpenSSH_3.7.1p1, SSH protocols 1.5/2.0, Open SSL 0.9.7d 17 Mar 2004
> (but ssh -v reports TEST's sshd server is responding with
> OpenSSH_3.6.1p1+CAN-2003-0693, SSH protocols 1.5/2.0, Open SSL 0x0090702f )
> $ uname -r
> 1.5.10(0.116/4/2)

Result is although the copy is cpu limited in ssh and cygwin ssh is slower
than SFU ssh it doesn't explain the same file test results:
SFU: real0m0.906s
Cygwin: real0m26.005s

Strange behaviour noticed. It appears that for each block tested cygwin
is opening a new connection instead of using the current one this could
well be where the performance is being lost as this doesn't happen
under SFU.

Steve



This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or 
entity to whom it is addressed. In the event of misdirection, the recipient is 
prohibited from using, copying, printing or otherwise disseminating it or any 
information contained in it. 

In the event of misdirection, illegible or incomplete transmission please telephone 
(023) 8024 3137
or return the E.mail to [EMAIL PROTECTED]


--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/



Re: rsync very slow, but not a network issue

2004-07-15 Thread Alexis Gallagher
Reini (and Steven),
Okay, I get. This benchmark you introduced below compares rsync on two 
variables: copy speed locally vs over the network, and copy speed for 
copies onto indentical target files vs different target files. And your 
benchmark showed that rsync was faster for copies onto pre-existing 
target files -- just as rsync is supposed to be.

I just ran the same benchmark on my side and I get exactly the opposite 
results. For what it's worth, here's what I got. I started with two 
different binary files (based on different mp3's) each of size 434006.

%[local copy, target file is different ]
%time rsync test1.mp3 test2.mp3
real 0m6.422s
user 0m0.430s
sys  0m1.431s
%[local copy, target file is the same ]
%time rsync test1.mp3 test1-same.mp3
real 0m6.523s
user 0m0.300s
sys  0m1.261s
%cp test2.bak test2.mp3
%[network copy, a pure copy with scp for benchmark]
%time scp test2.mp3 $TEST:
260.6KB/s (calculated average)
real 0m16.257s
user 0m0.510s
sys  0m1.091s
%[network copy, target file is different ]
%time rsync test1.mp3 $TEST:test2.mp3
real 0m13.797s
user 0m1.421s
sys  0m1.250s
% (network copy, files are 100% the same)
% (repeated twice to guarantee target file is there)
%time rsync test2.mp3 $TEST:test2.bak
real 0m28.329s
user 0m1.390s
sys  0m4.766s
%rsync --version
rsync version 2.6.2 protocol version 28
ssh $TEST rsync --version
rsync version 2.6.2 protocol version 28
$ ssh -V
OpenSSH_3.8.1p1, OpenSSL 0.9.7d 17 Mar 2004
$ ssh $TEST ssh -V
OpenSSH_3.7.1p1, SSH protocols 1.5/2.0, Open SSL 0.9.7d 17 Mar 2004
(but ssh -v reports TEST's sshd server is responding with
OpenSSH_3.6.1p1+CAN-2003-0693, SSH protocols 1.5/2.0, Open SSL 0x0090702f )
$ uname -r
1.5.10(0.116/4/2)
As you can see, my network copy onto an identical waiting file is taking 
almost twice as long as a network copy onto a _different_ waiting file, 
and is also substantially slower than a pure copy. So I'm experiencing a 
net rsync slowdown rather than a speedup.

Is it the processor? Thorsten Kampe And Steven Hartland suggested the 
problem is in effect that my computer is too slow, and that I should 
turn off all checksumming. I seem to have more bandwidth to my TEST 
machine than you, so I suppose my processor might be the bottleneck for 
me but not for you, which would explain our different results. How fast 
was your machine?

Or Is it cygwin? I notice that the enormous slowdown for the network 
copy onto an identical target  is all appearing in the 'real' component 
of the time's timing report, and that there's a bit of extra time in 
'sys' but other than that there's nothing to account for it. I'm not 
sure how to interpret that. Would cygwin-specific delays appear in 
'sys'? In other words, does this mean the checksums are using up all the 
time, and confirm that my processors are too slow to benefit from rsync?

Best wishes,
Alexis
Reini Urban wrote:
So check if the net overhead in the cygwin version is broken?
Could be easily tested out with two local files:
head -c 210 < /dev/random >test1.mp3
head -c 210 < /dev/random >test2.mp3
cp test1.mp3 test1-same.mp3
cp test2.mp3 test2.bak
time rsync test1.mp3 test2.mp3
  real0m0.518s
  user0m0.201s
  sys 0m0.171s
time rsync test1.mp3 test1-same.mp3
  real0m0.524s
  user0m0.154s
  sys 0m0.263s
RSYNC_RSH=ssh
TEST=othermachine
cp test2.bak test2.mp3
time scp test2.mp3 $TEST:
  63.2KB/s
real0m36.619s
user0m0.170s
sys 0m0.233s
# 100% different
$ time rsync test1.mp3 $TEST:test2.mp3
Server is very old version of rsync, upgrade recommended.
real0m37.162s
user0m0.388s
sys 0m0.202s
# 100% same
$ time rsync test2.mp3 $TEST:test2.bak
Server is very old version of rsync, upgrade recommended.
real0m7.298s
user0m1.201s
sys 0m2.217s
6x faster than scp, with same data.
0.5s (1.4%) slower than scp, with complete random data.
$ rsync --version
rsync  version 2.6.2  protocol version 28
$ ssh $TEST rsync --version
rsync version 2.4.6  protocol version 24
$ ssh -V
OpenSSH_3.8.1p1, OpenSSL 0.9.7d 17 Mar 2004
$ ssh $TEST ssh -V
OpenSSH_3.7.1p2, SSH protocols 1.5/2.0, OpenSSL 0.9.6c 21 dec 2001
$ uname -r
1.5.10(0.116/4/2)
--
Reini Urban
http://xarch.tu-graz.ac.at/home/rurban/

I see the final result you
--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/


Re: rsync very slow, but not a network issue

2004-07-13 Thread Reini Urban
Thorsten Kampe schrieb:
* Alexis Gallagher (2004-07-12 19:46 +0200)
I am finding that rsync+ssh is giving extremely slow file transfers. But 
this slowdown is hitting not when it needs to send data over the 
network, but when it applies the rsync algorithm which is supposed to be 
faster than sending all the data over the network. This is very puzzling.

For a benchmark, I tried transfering a 5.4 MB mp3 file three different 
ways. Here were my results, as reported by 'scp -v' and 'rsync 
--progress --stats':

   scp: 311000 B/s
   rsync (file not there): 309000 B/s
   rsync (file already there): 741 B/s
When the file is alredy there, rsync reports a speedup of about 70. 
(When the file is not already there, the speed up is 1, of course.) I am 
running rsync over ssh with pre-generated keys installed in my .ssh 
directories.

rsync under Cygwin is extremly slow and CPU intensive. For example a
simple script that syncs my dot files from my local NetWare server
takes 1:10 minutes to run and eats about 60% CPU while on my Gentoo
box (which has exactly the same hardware) it takes 7 seconds and 2%
CPU.
Try disabling checksumming, zipping and transfer the whole file
('-W'). Try it without ssh.
I rechecked my test, this time with cygserver enabled. Before I ran it 
without cygserver and without ipc-daemon2. (Forgot if rsync uses now the 
old ipc-daemon2 or the new cygserver or no ipc at all.)
Same (fast) results as expected, for both cases with local files.
--
Reini Urban
http://xarch.tu-graz.ac.at/home/rurban/

--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/


Re: rsync very slow, but not a network issue

2004-07-13 Thread Alexis Gallagher
Ok,
Now this is progress. When I use '--size-only' then I do see a speedup. 
The 'transfer' is virtually instantaneous on the stopwatch, and 'rsync 
--stats' reports a speed of a mere 25 bytes/sec but with a reported 
speedup of 78125.

I can see why this works, but I remain confused why I should need to 
resort to it. This is basically dispensing with the rsync algorithm, no?

The cygwin machine is a Pentium III, 966 Mhz, 512MB of RAM. And the OSX 
machine is a 1 Ghz PowerPC G4 with 256MB of RAM. I can't believe 
everyone would use rsync as much as they do if it were not useful on 
machines of such specifications.

Is there a way to benchmark its hashing algorithm on both sides? Maybe 
the rsync process is getting insufficient priority on one side of the 
transfer? I remain

puzzled,
Alexis Gallagher
Steven Hartland wrote:
Alexis Gallagher wrote:
So it's taking much longer in real time when the file is already
there, which is exactly the situation where rsync is supposed to
accelerate teh transfer.
The cygwin machine is a Pentium III 1Ghz, and the eMac is a bit
faster I believe. This should be fast enough that it's not
bottlenecking on the hash computation, I think.

> Are you using "--size-only"? Depending on the processor the check of
> the file chunks can be slower.
>
--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/


Re: rsync very slow, but not a network issue

2004-07-13 Thread Steven Hartland
Are you using "--size-only"? Depending on the processor the check of
the file chunks can be slower.

Steve / K
- Original Message - 
From: "Alexis Gallagher"

> So it's taking much longer in real time when the file is already there, 
> which is exactly the situation where rsync is supposed to accelerate teh 
> transfer.
> 
> The cygwin machine is a Pentium III 1Ghz, and the eMac is a bit faster I 
> believe. This should be fast enough that it's not bottlenecking on the 
> hash computation, I think.




This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or 
entity to whom it is addressed. In the event of misdirection, the recipient is 
prohibited from using, copying, printing or otherwise disseminating it or any 
information contained in it. 

In the event of misdirection, illegible or incomplete transmission please telephone 
(023) 8024 3137
or return the E.mail to [EMAIL PROTECTED]


--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/



Re: rsync very slow, but not a network issue

2004-07-13 Thread Thorsten Kampe
* Alexis Gallagher (2004-07-12 19:46 +0200)
> I am finding that rsync+ssh is giving extremely slow file transfers. But 
> this slowdown is hitting not when it needs to send data over the 
> network, but when it applies the rsync algorithm which is supposed to be 
> faster than sending all the data over the network. This is very puzzling.
> 
> For a benchmark, I tried transfering a 5.4 MB mp3 file three different 
> ways. Here were my results, as reported by 'scp -v' and 'rsync 
> --progress --stats':
> 
> scp: 311000 B/s
> rsync (file not there): 309000 B/s
> rsync (file already there): 741 B/s
> 
> When the file is alredy there, rsync reports a speedup of about 70. 
> (When the file is not already there, the speed up is 1, of course.) I am 
> running rsync over ssh with pre-generated keys installed in my .ssh 
> directories.

rsync under Cygwin is extremly slow and CPU intensive. For example a
simple script that syncs my dot files from my local NetWare server
takes 1:10 minutes to run and eats about 60% CPU while on my Gentoo
box (which has exactly the same hardware) it takes 7 seconds and 2%
CPU.

Try disabling checksumming, zipping and transfer the whole file
('-W'). Try it without ssh.

Thorsten


--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/



Re: rsync very slow, but not a network issue

2004-07-13 Thread Alexis Gallagher
Hi Steve,
Steven Hartland wrote:
Is this not because its showing you the network transfer rate i.e. spending
all its time doing compression and therefore not having to do actual
network transfers? How long did each test take?
I just performed the test again, this time timing the transfers with a 
stopwatch. (This time I was transferring a text file that consisted of 5 
million zeros, which is about 5MB.) Here's what I found.

   scp: 19 seconds
   rsync (no there): 17 seconds
   rsync (already there): 93 seconds
So it's taking much longer in real time when the file is already there, 
which is exactly the situation where rsync is supposed to accelerate teh 
transfer.

The cygwin machine is a Pentium III 1Ghz, and the eMac is a bit faster I 
believe. This should be fast enough that it's not bottlenecking on the 
hash computation, I think.

Cheers,
Alexis

--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/


Re: rsync very slow, but not a network issue

2004-07-13 Thread Reini Urban
Brian Dessent schrieb:
Reini Urban wrote:
When the file is alredy there, rsync reports a speedup of about 70.
(When the file is not already there, the speed up is 1, of course.) I am
running rsync over ssh with pre-generated keys installed in my .ssh
directories.
This is a binary MP3.
rsync (as diff) is not good in checking binary diffs.
Please try it with a typical text file, where the patch
is smaller than the source.
That is not correct at all, and rsync would be much less useful if it
were true.  rsync works with arbitrary binary data just as well as it
does with text.  It has no conception of 'lines' or any other type of
delimited text.  All it does is split up the file into fixed-length
chunks and calculates a rolling checksum of each of these.  It does not
use anything resembling 'diff' or 'patch'. 

http://rsync.samba.org/tech_report/node2.html
Thanks for clarification. Makes sense.
So as punishment I made another bench.
What is the maximum overhead?
It cannot be the reported difference (factor 500).
http://rsync.samba.org/tech_report/node6.html has about a similar 
filesize (2.1MB binary).
In this example with a hit:miss ration of 64247:948 the overhead is 
5312200:5629158+1632284 (if read and write is synchronous) compared to rcp.
So about max 2.5 overhead, and not 500.

So check if the net overhead in the cygwin version is broken?
Could be easily tested out with two local files:
head -c 210 < /dev/random >test1.mp3
head -c 210 < /dev/random >test2.mp3
cp test1.mp3 test1-same.mp3
cp test2.mp3 test2.bak
time rsync test1.mp3 test2.mp3
  real0m0.518s
  user0m0.201s
  sys 0m0.171s
time rsync test1.mp3 test1-same.mp3
  real0m0.524s
  user0m0.154s
  sys 0m0.263s
RSYNC_RSH=ssh
TEST=othermachine
cp test2.bak test2.mp3
time scp test2.mp3 $TEST:
  63.2KB/s
real0m36.619s
user0m0.170s
sys 0m0.233s
# 100% different
$ time rsync test1.mp3 $TEST:test2.mp3
Server is very old version of rsync, upgrade recommended.
real0m37.162s
user0m0.388s
sys 0m0.202s
# 100% same
$ time rsync test2.mp3 $TEST:test2.bak
Server is very old version of rsync, upgrade recommended.
real0m7.298s
user0m1.201s
sys 0m2.217s
6x faster than scp, with same data.
0.5s (1.4%) slower than scp, with complete random data.
$ rsync --version
rsync  version 2.6.2  protocol version 28
$ ssh $TEST rsync --version
rsync version 2.4.6  protocol version 24
$ ssh -V
OpenSSH_3.8.1p1, OpenSSL 0.9.7d 17 Mar 2004
$ ssh $TEST ssh -V
OpenSSH_3.7.1p2, SSH protocols 1.5/2.0, OpenSSL 0.9.6c 21 dec 2001
$ uname -r
1.5.10(0.116/4/2)
--
Reini Urban
http://xarch.tu-graz.ac.at/home/rurban/
--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/


Re: rsync very slow, but not a network issue

2004-07-13 Thread Steven Hartland

- Original Message - 
From: "Alexis Gallagher"

> In any event I checked Reini's theory using a 5 MB text file that
> consists only of x's and newlines. So unlike an MP3, this file is
> non-binary and extremely compressible. And what I observed was as follows:
> 
>scp: 348800 bytes/sec
>rsync (file not there): 303075 bytes/sec, with speedup of 1
>rsync (file already there): 733 bytes/sec, with speedup of 70
> 
> This is pretty much exactly the same problem as I was having with the
> MP3 file. And I'm still quite puzzled.

Is this not because its showing you the network transfer rate i.e. spending
all its time doing compression and therefore not having to do actual
network transfers? How long did each test take?

Steve



This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or 
entity to whom it is addressed. In the event of misdirection, the recipient is 
prohibited from using, copying, printing or otherwise disseminating it or any 
information contained in it. 

In the event of misdirection, illegible or incomplete transmission please telephone 
(023) 8024 3137
or return the E.mail to [EMAIL PROTECTED]


--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/



Re: rsync very slow, but not a network issue

2004-07-13 Thread Alexis Gallagher
Hello all,
Thanks for the help so far, on this issue with the mysterious
non-network based rsync slowdown. To recap...
Brian Dessent wrote:
Reini Urban wrote:
Alexis Gallagher wrote:
When the file is alredy there, rsync reports a speedup of about 
70. (When the file is not already there, the speed up is 1, of 
course.) I am running rsync over ssh with pre-generated keys 
installed in my .ssh directories.
This is a binary MP3. rsync (as diff) is not good in checking 
binary diffs. Please try it with a typical text file, where the 
patch is smaller than the source.

That is not correct at all, and rsync would be much less useful if it
 were true.  rsync works with arbitrary binary data just as well as 
it does with text.
In any event I checked Reini's theory using a 5 MB text file that
consists only of x's and newlines. So unlike an MP3, this file is
non-binary and extremely compressible. And what I observed was as follows:
  scp: 348800 bytes/sec
  rsync (file not there): 303075 bytes/sec, with speedup of 1
  rsync (file already there): 733 bytes/sec, with speedup of 70
This is pretty much exactly the same problem as I was having with the
MP3 file. And I'm still quite puzzled.
Cheers,
Alexis
--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/


Re: rsync very slow, but not a network issue

2004-07-12 Thread Brian Dessent
Reini Urban wrote:

> > When the file is alredy there, rsync reports a speedup of about 70.
> > (When the file is not already there, the speed up is 1, of course.) I am
> > running rsync over ssh with pre-generated keys installed in my .ssh
> > directories.
> 
> This is a binary MP3.
> rsync (as diff) is not good in checking binary diffs.
> Please try it with a typical text file, where the patch
> is smaller than the source.

That is not correct at all, and rsync would be much less useful if it
were true.  rsync works with arbitrary binary data just as well as it
does with text.  It has no conception of 'lines' or any other type of
delimited text.  All it does is split up the file into fixed-length
chunks and calculates a rolling checksum of each of these.  It does not
use anything resembling 'diff' or 'patch'. 

http://rsync.samba.org/tech_report/node2.html

Brian

--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/



Re: rsync very slow, but not a network issue

2004-07-12 Thread Reini Urban
Alexis Gallagher schrieb:
I am finding that rsync+ssh is giving extremely slow file transfers. But 
this slowdown is hitting not when it needs to send data over the 
network, but when it applies the rsync algorithm which is supposed to be 
faster than sending all the data over the network. This is very puzzling.

For a benchmark, I tried transfering a 5.4 MB mp3 file three different 
ways. Here were my results, as reported by 'scp -v' and 'rsync 
--progress --stats':

   scp: 311000 B/s
   rsync (file not there): 309000 B/s
   rsync (file already there): 741 B/s
When the file is alredy there, rsync reports a speedup of about 70. 
(When the file is not already there, the speed up is 1, of course.) I am 
running rsync over ssh with pre-generated keys installed in my .ssh 
directories.
This is a binary MP3.
rsync (as diff) is not good in checking binary diffs.
Please try it with a typical text file, where the patch
is smaller than the source.
--
Reini Urban
http://xarch.tu-graz.ac.at/home/rurban/
--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/


rsync very slow, but not a network issue

2004-07-12 Thread Alexis Gallagher
Hello,
I am finding that rsync+ssh is giving extremely slow file transfers. But 
this slowdown is hitting not when it needs to send data over the 
network, but when it applies the rsync algorithm which is supposed to be 
faster than sending all the data over the network. This is very puzzling.

For a benchmark, I tried transfering a 5.4 MB mp3 file three different 
ways. Here were my results, as reported by 'scp -v' and 'rsync 
--progress --stats':

   scp: 311000 B/s
   rsync (file not there): 309000 B/s
   rsync (file already there): 741 B/s
When the file is alredy there, rsync reports a speedup of about 70. 
(When the file is not already there, the speed up is 1, of course.) I am 
running rsync over ssh with pre-generated keys installed in my .ssh 
directories.

The receiving machine is running OS X 10.3.4. It has OpenSSH_3.6.1p1 (as 
judged from the 'ssh -v' output) and the fink installation of 
rsync2.6.2. The sending machine is running WinXPProSP1, with rsync 2.6.2 
and OpenSSH_3.8.1p1. FWIW, I am attaching my cygcheck.out, as suggested 
in the webpage on using this list.

Much obliged for any help,
Alexis

Cygwin Configuration Diagnostics
Current System Time: Mon Jul 12 17:20:09 2004

Windows XP Professional Ver 5.1 Build 2600 Service Pack 1

Path:   E:\cygwin\usr\local\bin
E:\cygwin\bin
E:\cygwin\bin
E:\cygwin\usr\X11R6\bin
e:\WINDOWS\system32
e:\WINDOWS
e:\WINDOWS\system32\WBEM
e:\Program Files\3dsmax5\backburner2\
e:\Program Files\Common Files\Autodesk Shared\
e:\jwsdp-1.3\jwsdp-shared\bin
e:\Program Files\SSH Communications Security\SSH Secure Shell
e:\jwsdp-1.3\bin
e:\j2sdk1.4.2\bin
e:\Python23
e:\jwsdp-1.3\apache-ant\bin

Output from E:\cygwin\bin\id.exe (nontsec)
UID: 1003(alexis) GID: 513(None)
513(None)

Output from E:\cygwin\bin\id.exe (ntsec)
UID: 1003(alexis) GID: 513(None)
0(root)  513(None)
544(Administrators)  545(Users)

SysDir: E:\WINDOWS\System32
WinDir: E:\WINDOWS

HOME = `E:\cygwin\home\alexis'
MAKE_MODE = `unix'
PWD = `/home/alexis/iTunesMusic/Jay-Z + DJ Danger Mouse/The Grey Album'
USER = `alexis'

ALLUSERSPROFILE = `E:\Documents and Settings\All Users'
ANT_HOME = `e:\jwsdp-1.3\apache-ant'
APPDATA = `E:\Documents and Settings\alexis\Application Data'
CLIENTNAME = `Console'
COMMONPROGRAMFILES = `E:\Program Files\Common Files'
COMPUTERNAME = `8YGZF01'
COMSPEC = `E:\WINDOWS\system32\cmd.exe'
CVS_RSH = `/bin/ssh'
HOMEDRIVE = `E:'
HOMEPATH = `\Documents and Settings\alexis'
HOSTNAME = `8ygzf01'
INFOPATH = 
`/usr/local/info:/usr/info:/usr/share/info:/usr/autotool/devel/info:/usr/autotool/stable/info:'
JAVA_HOME = `E:\j2sdk1.4.2'
JSF_HOME = `e:\jwsdp-1.3\jsf-1_0-beta'
JWSDP_HOME = `e:\jwsdp-1.3'
LOGONSERVER = `\\8YGZF01'
MANPATH = 
`/usr/local/man:/usr/man:/usr/share/man:/usr/autotool/devel/man::/usr/ssl/man:/usr/X11R6/man'
NUMBER_OF_PROCESSORS = `1'
OLDPWD = `/home/alexis/iTunesMusic/Jay-Z + DJ Danger Mouse'
OS = `Windows_NT'
PATHEXT = `.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH'
PROCESSOR_ARCHITECTURE = `x86'
PROCESSOR_IDENTIFIER = `x86 Family 6 Model 8 Stepping 6, GenuineIntel'
PROCESSOR_LEVEL = `6'
PROCESSOR_REVISION = `0806'
PROGRAMFILES = `E:\Program Files'
PROMPT = `$P$G'
PS1 = `\[\033]0;\w\007
[EMAIL PROTECTED] \[\033[33m\w\033[0m\]
$ '
PYTHONPATH = `E:\Python23\alexispackages\;E:\Python23\alexispackages\pyblogger-1.5'
RSYNC_RSH = `ssh'
SESSIONNAME = `Console'
SHLVL = `1'
SYSTEMDRIVE = `E:'
SYSTEMROOT = `E:\WINDOWS'
TEMP = `E:\DOCUME~1\alexis\LOCALS~1\Temp'
TERM = `cygwin'
TMP = `E:\DOCUME~1\alexis\LOCALS~1\Temp'
USERDOMAIN = `8YGZF01'
USERNAME = `alexis'
USERPROFILE = `E:\Documents and Settings\alexis'
WINDIR = `E:\WINDOWS'
_ = `/usr/bin/cygcheck'
POSIXLY_CORRECT = `1'

HKEY_CURRENT_USER\Software\Cygnus Solutions
HKEY_CURRENT_USER\Software\Cygnus Solutions\Cygwin
HKEY_CURRENT_USER\Software\Cygnus Solutions\Cygwin\mounts v2
HKEY_CURRENT_USER\Software\Cygnus Solutions\Cygwin\Program Options
HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions
HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin
HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2
  (default) = `/cygdrive'
  cygdrive flags = 0x0022
HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2\/
  (default) = `E:\cygwin'
  flags = 0x000a
HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2\/usr/bin
  (default) = `E:\cygwin/bin'
  flags = 0x000a
HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2\/usr/lib
  (default) = `E:\cygwin/lib'
  flags = 0x000a
HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2\/usr/X11R6/lib/X11/fonts
  (default) = `E:\cygwin\usr\X11R6\lib\X11\fonts'
  flags = 0x000a
HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\Program Options

a:  fd   N/AN/A
c:  fd   N/AN/A
d:  cd   N/AN/A
e:  hd  NTFS   57207Mb  9