Re: performance problem of using parallel rsync to stage data from 1 source to multiple destination

2005-09-02 Thread Paul Slootman
On Thu 01 Sep 2005, Xuehai Zhang wrote:
 
  If (similar) tasks are run in parallel, then the data of the files being
  handled may still be in the buffer cache so that it doesn't need to get
  read in from disk again. This will save time...
 
 I agree with you that caching effect might be the cause. The buffer cache
 you mentioned refers to the cache on the data source, right?

Yes.


Paul Slootman
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: Status of rsync on win32

2005-09-02 Thread Misha Bergal
Wayne Davison [EMAIL PROTECTED] writes:

 On Sun, Aug 28, 2005 at 10:15:55AM -0500, Misha Bergal wrote:
 rsync on the server machine makes System process to consume about 90%
 of CPU time.

 If you are sending over a fast network, using -W (--whole-file) will
 save you CPU (and disk I/O).  In addition (or alternately), if you're
 using an encrypted transport (such as ssh), try switching it to a faster
 cipher (e.g. -e ssh -c arcfour) or use an unencrypted transport (such
 as rsh or a direct rsync daemon setup).


I am sending file through rather slow network. I am not using ssh.


-- 
Misha Bergal
MetaCommunications Engineering

-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: performance problem of using parallel rsync to stage data from 1 source to multiple destination

2005-09-02 Thread Wayne Davison
Since it sounds like disk I/O is your limiting factor, you may wish to
look into updating multiple systems using a batch file.  This requires
all the receiving systems to have identical files in the destination
hierarchy.  You would first create a batch file by performing the
synchronization either to one of the destination systems, or even to a
second physical disk on the sending system in an effort to create the
batch file more quickly (if you avoid a copy to the same physical disk*,
it will avoid having both the sending and receiving disk I/O hitting a
single disk).  The batch-writing command would be:

  rsync -av --write-batch=xfer1 /src/ /dest/

Then, you would update all the (remaining) destinations by reading that
batch file:

  rsync -av --read-batch=xfer1 $dest:/tmp/ 

As long as you're using at least 2.6.3, batch mode should work quite
well (older versions used an experimental implementation that is not
recommended).

*One other thing you might try to create the batch file quickly using a
local copy on a machine that doesn't have a second physical disk:  use
the option --only-write-batch instead of --write-batch.  This would use
an extra copy of the destination files somewhere on the sending system,
but not update it right away, which _might_ save some elapsed time in
the creating of the batch file (you'd have to try it and see).  Then, at
the end of the updating of the remote systems, you would use the batch
file to update your local destination mirror.

..wayne..
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html