Rsync on WinNT and Solaris

[Victor Templonuevo]

Hi!

  Ive been trying to make rsync run for the past week
or so on my two machines running on Solaris and NT but
to no avail.

  Heres my setup. 

  I was able to setup rsync on my Winnt machine as a
service based on Mike McHenry Documentation on the
Rsync Web page. I have the following .conf file

use chroot = false
strict modes = false
hosts allow = *
transfer logging = true

[backup]
path = /export/home/rsyncUser

  On the Solaris Machine i just installed the binaries
of rsync. 

   Im able to rsh to my Winnt and Solaris Machine and
vice versa. 

   But when i tried executing rsync on my Solaris
machine it gave me this error:

protocol version mismatch - is your shell clean?
(see the rsync man page for an explanation)

   I checked the man pages of rsync, but since im
running Winnt on the other end, i cannot execute the
diagnostic command specified in the man pages to
troubleshoot the above error since i dont have a
/bin/true file on my winnt. So what i did is tried the
diagnostic command on my other solaris machine and it
produces a zero length file. i dont know if theres a
different behavior of rsync when executed over Winnt.


   When i executed rsync now on my winnt, it gave me a
different error:

Terminal read: The parameter is incorrect.
unexpected EOF in read_timeout
read error: Connection reset by peer

Can anyone help me on this.

Thanks in advance.

Victor Templonuevo

__
Do You Yahoo!?
Yahoo! Auctions - buy the things you want at great prices
http://auctions.yahoo.com/

FW: Problem with large include files

[Wilson, Mark - MST]

-Original Message-
From: Wilson, Mark - MST 
Sent: Tuesday, 15 May 2001 15:31
To: 'Dave Dykstra'
Subject: RE: Problem with large include files



> No, rsync can't run over UDP.  Perhaps there are some TCP parameters you
> can tweak.  I wouldn't be surprised if the combination of the high speed
> and fairly high latency would require a larger window size.  It may
require
> some tcpdumps and an analysis by somebody familiar with how to read them.

Shame, will try to look at TCP configurable items.

Do you have any idea what the maximum number of files you can have in an
include file is (for the current version)?

How do you want your test on 2.3.2 done? ie LAN or high speed WAN, numbers
of file, sizes of files, things to time, daemon v rsh.

Cheers

Mark


CAUTION - This message may contain privileged and confidential 
information intended only for the use of the addressee named above.
If you are not the intended recipient of this message you are hereby 
notified that any use, dissemination, distribution or reproduction 
of this message is prohibited. If you have received this message in 
error please notify Air New Zealand immediately. Any views expressed 
in this message are those of the individual sender and may not 
necessarily reflect the views of Air New Zealand.
_
For more information on the Air New Zealand Group, visit us online
at http://www.airnewzealand.com or http://www.ansett.com.au
_

Re: Problem with large include files

[Dave Dykstra]

On Tue, May 15, 2001 at 08:54:08AM +1200, Wilson, Mark - MST wrote:
> 
> > Try rsync 2.3.2 on both ends and see what the result is.  
> 
> > What transport method are you using (rsh, ssh, or rsync --daemon mode)?
> 
> --daemon mode. Some tests a colleague did showed a performance advantage.
> 
> 
> > Come to think of it, I know from experience that rsync is able to keep the
> > TCP queues full in both directions during a transfer.  There must be
> > something else going on.  Does your link perhaps lose a percentage of the
> > packets, forcing TCP to timeout and retransmit, slowing it down?
> 
> I thought rsync *should* be quite good, given what it is used for. The
> network guys tell me that the link only sheds packets (ATM frames) in an
> intelligent way when the bandwidth goes over 29.9 Mbit. If only a single
> stream is used no packets are lost. Up until I started using parallel rsync
> streams, no one had been able to soak the link. I have a theory that TCP
> requires some kind of ACK or NAK for every packet sent. Or if it uses
> "sliding windows" the window size is too small for the speed/latency of this
> link, hence TCP doesn't send flat out. Unfortunately I've never delved into
> TCP enough to find out how this part of it works. Is there are version of
> rsync that uses UDP? If implemented correctly this would get around any
> latency problems.


No, rsync can't run over UDP.  Perhaps there are some TCP parameters you
can tweak.  I wouldn't be surprised if the combination of the high speed
and fairly high latency would require a larger window size.  It may require
some tcpdumps and an analysis by somebody familiar with how to read them.

- Dave Dykstra

FW: Problem with large include files

[Wilson, Mark - MST]

> Try rsync 2.3.2 on both ends and see what the result is.  

> What transport method are you using (rsh, ssh, or rsync --daemon mode)?

--daemon mode. Some tests a colleague did showed a performance advantage.


> Come to think of it, I know from experience that rsync is able to keep the
> TCP queues full in both directions during a transfer.  There must be
> something else going on.  Does your link perhaps lose a percentage of the
> packets, forcing TCP to timeout and retransmit, slowing it down?

I thought rsync *should* be quite good, given what it is used for. The
network guys tell me that the link only sheds packets (ATM frames) in an
intelligent way when the bandwidth goes over 29.9 Mbit. If only a single
stream is used no packets are lost. Up until I started using parallel rsync
streams, no one had been able to soak the link. I have a theory that TCP
requires some kind of ACK or NAK for every packet sent. Or if it uses
"sliding windows" the window size is too small for the speed/latency of this
link, hence TCP doesn't send flat out. Unfortunately I've never delved into
TCP enough to find out how this part of it works. Is there are version of
rsync that uses UDP? If implemented correctly this would get around any
latency problems.

Cheers

Mark


CAUTION - This message may contain privileged and confidential 
information intended only for the use of the addressee named above.
If you are not the intended recipient of this message you are hereby 
notified that any use, dissemination, distribution or reproduction 
of this message is prohibited. If you have received this message in 
error please notify Air New Zealand immediately. Any views expressed 
in this message are those of the individual sender and may not 
necessarily reflect the views of Air New Zealand.
_
For more information on the Air New Zealand Group, visit us online
at http://www.airnewzealand.com or http://www.ansett.com.au
_

Re: Problem with large include files

[Dave Dykstra]

On Mon, May 14, 2001 at 01:32:01PM -0700, Wayne Davison wrote:
> On Fri, 11 May 2001, Dave Dykstra wrote:
> > The optimization bypassed the normal recursive traversal of all the
> > files and directly opened the included files and sent the list over.
> 
> There's an alternative to this optimization:  if we could read the
> source files from a file (or stdin), we could use -R to preserve the
> source path.  For instance:
> 
> cd /srcpath
> rsync -R --source-files-from=file-list desthost:/destpath
> 
> This would allow the behavior of the --include code to remain unchanged
> since the user would be specifying a huge list of files/directories to
> transfer, not a huge list of include directives with a "*" exclude at
> the end.

Uh, I think you're thinking of the same result as I was with the --files-from
option, but a different implementation.  With --files-from it will leave
the behavior of --include alone, although internally I expect it will still
use the include file list portion of the rsync wire protocol.  I would
still want to have a "source" path as part of the command line syntax, which
you left out above.  The paths in the --files-from file-list will be relative
to the top level.

- Dave Dykstra

Re: Problem with large include files

[Wayne Davison]

On Fri, 11 May 2001, Dave Dykstra wrote:
> The optimization bypassed the normal recursive traversal of all the
> files and directly opened the included files and sent the list over.

There's an alternative to this optimization:  if we could read the
source files from a file (or stdin), we could use -R to preserve the
source path.  For instance:

cd /srcpath
rsync -R --source-files-from=file-list desthost:/destpath

This would allow the behavior of the --include code to remain unchanged
since the user would be specifying a huge list of files/directories to
transfer, not a huge list of include directives with a "*" exclude at
the end.

..wayne..

Re: Problem with large include files

[Dave Dykstra]

On Mon, May 14, 2001 at 11:13:40AM -0500, Dave Dykstra wrote:
...
> > As a thought, have you or any of the other developers thought of getting
> > rsync to operate over a number of streams or to use "sliding windows" to
> > overcome latency effects?
> 
> I don't recall that that subject has been discussed much on the mailing list
> since I've been participating.  It was my understanding that rsync already
> pipelines pretty well in both directions. 

Come to think of it, I know from experience that rsync is able to keep the
TCP queues full in both directions during a transfer.  There must be
something else going on.  Does your link perhaps lose a percentage of the
packets, forcing TCP to timeout and retransmit, slowing it down?

- Dave Dykstra

Re: Problem with large include files

[Dave Dykstra]

On Mon, May 14, 2001 at 03:12:27PM +1200, Wilson, Mark - MST wrote:
> Dave
> 
> A couple of points:
> 
> 1. I think you are telling me that if I go back to 2.3.2 my problems should
> go away. Is this correct?

Hopefully, assuming you trigger the optimization (no wildcards in the
includes followed by an exclude '*', no --delete).  It depends on what
exactly is failing though.

> 2. I rather oversimplified how I am using rsync... Perhaps I had better
> explain. If I can do some testing for you I am happy to do so, however there
> is quite a bit of pressure for me to get my problem fixed. This I must do
> first.
...
> Anyway, the purpose of all this verbosity is two fold. Firstly you need to
> tell me, given my environment, how you want your testing done and secondly
> if you had any ideas of how to fix my problem. If we can't fix it we will
> have to do the backup to tape and send it on a plane method -which we really
> want to avoid. 

Try rsync 2.3.2 on both ends and see what the result is.  

What transport method are you using (rsh, ssh, or rsync --daemon mode)?




> As a thought, have you or any of the other developers thought of getting
> rsync to operate over a number of streams or to use "sliding windows" to
> overcome latency effects?

I don't recall that that subject has been discussed much on the mailing list
since I've been participating.  It was my understanding that rsync already
pipelines pretty well in both directions. 

- Dave Dykstra

RE: Problem with large include files

[Safi, Sam (S.M.)]

Mark,

I reverted back to rsync 2.3.2 after a massive complain from our user trying to get 
their data from their machine to the target sites.  Since rsync been running very 
smoothly.

Just as a background.  I have been using rsync for about 3 years now and been using 
2.3.2 since it came out.  I transfer updated data daily from 40 sources to 70 targets 
globally including Australia, Europe, South America and the U.S., and the size of the 
data is about 320 GB per target where the size of the total update can be in the range 
of 50 GB per target site per day.  Our files are CAD files and the file name can be in 
access of 100 character long (Which used to be a problem in one of the releases of 
rsync, if I recall)

Regards / Mit freundlichen Gruessen
Sam Safi
EAI/Ford Motor Co.  Alpha, A401 
Digital Buck & Visual Collaboration *  
Data Management & Security  *  313-39-01744




-Original Message-
From: Wilson, Mark - MST [mailto:[EMAIL PROTECTED]]
Sent: Sunday, May 13, 2001 11:12 PM
To: 'Dave Dykstra'
Cc: RSync List (E-mail)
Subject: RE: Problem with large include files 


Dave

A couple of points:

1. I think you are telling me that if I go back to 2.3.2 my problems should
go away. Is this correct?

2. I rather oversimplified how I am using rsync... Perhaps I had better
explain. If I can do some testing for you I am happy to do so, however there
is quite a bit of pressure for me to get my problem fixed. This I must do
first.

Environment:
I am shifting large applications from a Sun E10k domain backended with a
NetApp F840 Filer to another Sun E10k backended with a NetApp F840 Filer.
The source machine is in Auckland, New Zealand and the destination machine
is in Melbourne, Australia. The link is a 10Mbit pipe that is burstable to
30Mbit (actually I can run it full time at 30Mbit at the moment, I'm sure
that wont last). The latency on the link is about 20ms one way.

Original Problem:
Because of the "slow" link and requirement to maintain all the files, links,
etc we were limited to a few mechanisms. Originally we were ftping tgz files
but finding sufficient scratch space was a problem. Also the compress time
made things slow. We also tried various versions of rsh tar gzip, etc.
Eventually I tried rsync because it did on the fly compression and correctly
handled permissions, files, links, etc.

Locally, especially across our 1Gbit links, rsync flew. Everyone was very
happy. However it wasn't so fast across the Tasman (sea between NZ and Oz).
In fact with a bit of sniffing by the network guys we found that rsync
wasn't even using 1Mbit of the link. H, latency methinks. So I wrote a
fancy perl script that took the required directories to be transferred and
split all the files into 20 (configurable) balanced streams. This meant 20
copies of rsync running at the same time. Of course the balancing was done
on the uncompressed file sizes so took no account of how "dense" the files
were. This is unfortunate as different streams take different amounts of
times. However the results were quite spectacular! At best, on DB files, we
were getting disk to disk transfer rates of up to 1.5Gbyte per minute. Not
bad over a 10Mbit link... It completely soaked the link and maxed out the 12
processors. Fun having all that power at your command!

The current problem:
However the euphoria rapidly wore off when I tried to transfer 128Gb of test
data, some of it quite dense with many, many small files. In fact 104755
files in total. Unfortunately if there are too many files in an include file
(--include-from) the streams quit part way through. Very upsetting... I
tried turning up the debugging and re running it. I didn't find out anything
except how many files it did before it quit. Interestingly enough, with
heaps of debug, it processed more files... Weird.

Anyway, the purpose of all this verbosity is two fold. Firstly you need to
tell me, given my environment, how you want your testing done and secondly
if you had any ideas of how to fix my problem. If we can't fix it we will
have to do the backup to tape and send it on a plane method -which we really
want to avoid. 

As a thought, have you or any of the other developers thought of getting
rsync to operate over a number of streams or to use "sliding windows" to
overcome latency effects?

I look forward to your reply.

Cheers

Mark
-Original Message-
From: Dave Dykstra [mailto:[EMAIL PROTECTED]]
Sent: Saturday, 12 May 2001 01:43
To: Wilson, Mark - MST
Cc: RSync List (E-mail)
Subject: Re: Problem with large include files


On Fri, May 11, 2001 at 11:41:41AM +1200, Wilson, Mark - MST wrote:
> Hi there
>  
> I recently tried to do a transfer of a directory tree with about 120,000
> files. I needed to select various files and used the --include-from option
> to select the files I needed to transfer. The include file had 103,555
> filenames in it. The problem I have is that