Re: Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-29 Thread Alberto Accomazzi
In message <[EMAIL PROTECTED]>, Dave Dykstra writes: > On Thu, Nov 29, 2001 at 11:02:07AM -0500, Alberto Accomazzi wrote: > ... > > These numbers show that reading the filenames this way rather than using > > the code in place to deal with the include/exclude list cuts the startup > > time down t

Re: Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-29 Thread Dave Dykstra
On Thu, Nov 29, 2001 at 11:02:07AM -0500, Alberto Accomazzi wrote: ... > These numbers show that reading the filenames this way rather than using > the code in place to deal with the include/exclude list cuts the startup > time down to 0 (from 1hr). The actual sending of the filenames is down > f

RE: Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-29 Thread Keating, Tim
> From: Alberto Accomazzi [mailto:[EMAIL PROTECTED]] > Sent: Thursday, November 29, 2001 10:02 AM > To: Dave Dykstra > Cc: [EMAIL PROTECTED] > Subject: Re: Rsync: Re: patch to enable faster mirroring of large > filesystems

Re: Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-29 Thread Alberto Accomazzi
Here are some more results from my tests towards implementing --files-from. I have modified (actually "hacked" is more appropriate here -- read on) the source of rsync-2.3.2 to implement the command line option --files-from in an effort to test how well this feature would work in a real-case scen

Re: Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-28 Thread Eric Whiting
Dave Dykstra wrote: > > Note that his case is rather pathological because he's got over a million > files in only 400 directories, so he must have an average of over 2500 > files per directory, which are very large directories. He's got about 65% > of the files explicitly listed in his --include-

Re: Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-28 Thread Dave Dykstra
Rsync list: Alberto and I have done a couple more exchanges by private email, and we found that he wasn't turning on my include/exclude optimization in his test because he had an "exclude" directive in rsyncd.conf. He has now removed that and run the test again. His very interesting results are

Re: Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-28 Thread Dave Dykstra
On Tue, Nov 27, 2001 at 05:00:14PM -0500, Lenny Foner wrote: ... > [ . . . ] > > I'm pretty sure that rsync won't use up memory for excluded files so it > would make no difference. > > ...though this also implies (since you say it'd probably use basically > the same mechanism interna

Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-27 Thread Lenny Foner
Date: Tue, 27 Nov 2001 15:34:15 -0600 From: Dave Dykstra <[EMAIL PROTECTED]> [ . . . ] No, the difficulty of turning on the optimization is irrelevant because the optimization is no longer in the current version of rsync. It is only needed to do the performance test whic

Re: Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-27 Thread Dave Dykstra
On Tue, Nov 27, 2001 at 02:34:22PM -0500, Lenny Foner wrote: ... > I know you're trying to get reliable statistics so it's clear what > sort of performance we're talking about here. But may I respectfully > suggest that -having- to be so careful about whether optimization > actually got turned on

Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-27 Thread Lenny Foner
Date: Tue, 27 Nov 2001 10:49:11 -0600 From: Dave Dykstra <[EMAIL PROTECTED]> Thank you very much for doing the test Alberto. I didn't have any set of files that large on which I could do a test, and as I said when I tested the worse case I could think of with my application I

Re: Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-27 Thread Dave Dykstra
Thank you very much for doing the test Alberto. I didn't have any set of files that large on which I could do a test, and as I said when I tested the worse case I could think of with my application I couldn't measure an appreciable difference. First, I want to make sure that you really did get t

Re: Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-27 Thread Alberto Accomazzi
Dear all, here's my own (renewed) pitch to throw in a --files-from patch. As Dave has suggested in the past, transferring a list of files can be accomplished using --include and --exclude, and has called for people to test the performance gains of his old optimization when using these options (s

Re: Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-20 Thread Dave Dykstra
On Tue, Nov 20, 2001 at 11:45:44AM +, Lachlan Cranswick wrote: > > Is there any chance this can be added into the distribution as it sounds > really nifty. I exchanged some off-list email with the patch author and besides the fact that it adds too many options I object to it because it only

Rsync: Re: patch to enable faster mirroring of large filesystems

2001-11-20 Thread Lachlan Cranswick
Is there any chance this can be added into the distribution as it sounds really nifty. Another suggestion unless I have read the following - would it be useful to have a command option in rsync to generate the file list by doing the "find" and outputting into a standard format? (As this would m