Re: performance problem of using parallel rsync to stage data from 1 source to multiple destination

2005-09-02 Thread Paul Slootman
On Thu 01 Sep 2005, Xuehai Zhang wrote: If (similar) tasks are run in parallel, then the data of the files being handled may still be in the buffer cache so that it doesn't need to get read in from disk again. This will save time... I agree with you that caching effect might be the

Re: performance problem of using parallel rsync to stage data from 1 source to multiple destination

2005-09-02 Thread Wayne Davison
Since it sounds like disk I/O is your limiting factor, you may wish to look into updating multiple systems using a batch file. This requires all the receiving systems to have identical files in the destination hierarchy. You would first create a batch file by performing the synchronization

Re: performance problem of using parallel rsync to stage data from 1 source to multiple destination

2005-09-01 Thread Paul Slootman
On Wed 31 Aug 2005, xuehai zhang wrote: results. Why the time of transferring the file to 2N nodes is shorter than twice of the time of transferring the same file to N nodes? Does it make If the network is not the bottleneck, then cpu or the disks are. If (similar) tasks are run in parallel,

Re: performance problem of using parallel rsync to stage data from 1 source to multiple destination

2005-09-01 Thread Xuehai Zhang
Paul, Thanks for your response. results. Why the time of transferring the file to 2N nodes is shorter than twice of the time of transferring the same file to N nodes? Does it make If the network is not the bottleneck, then cpu or the disks are. The network is 100Mbps LAN. If (similar)

performance problem of using parallel rsync to stage data from 1 source to multiple destination

2005-08-31 Thread xuehai zhang
Hi all, I am new to rsync and I apologize in advance if my question is shallow. I write a simple script to use rsync to transfer a big file (~600MB)from a single source to variable number of destinations in a parallel way. When I transfer the file to 4 destination machines, I get X overall