Re: FW: rsync performance
On Sat, 13 Sep 2003 20:28:21 -0700 jw schultz <[EMAIL PROTECTED]> wrote: > The referenced mail message describes the benchmark as: > | The directory backed up or restored had 1 1-byte files > > That isn't a very good benchmark. 10,000 files is not that > many and being 1 byte means that all that is measured is the > filesystem meta-data, node-creation time and overhead. The purpose of that benchmark was to measure those factors. > That the test used the --delete option indicates that some > percentage of the files would have not been touched by rsync. My > guess is that the unmodified files account for cp -a being slower; > rsync processed fewer files than cp. The fact that the benchmark > description does not indicate the actual rate of change (a > determining factor for rsync and, i assume, rdiff-backup) makes it > decidedly dubious. The description was slightly unclear; the numbers in the first group indicate copying into an empty directory, so both processed the same number of files. > I believe the speed complaint had to do with files that have > significant amounts of data in them. My remark was off-topic then. -- Ben Escoto pgp0.pgp Description: PGP signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: FW: rsync performance
On Sat, Sep 13, 2003 at 07:46:05PM -0700, Ben Escoto wrote: > On Fri, 12 Sep 2003 17:05:09 -0700 jw schultz <[EMAIL PROTECTED]> wrote: > > Rsync is not an efficient local copy utility. It can be > > used for local copying but local and high-bandwidth network > > speed is sacrificed for low-bandwidth performance and for > > data integrity. > > I've been surprised at how fast rsync can copy locally. For instance, > > http://mail.nongnu.org/archive/html/rdiff-backup-users/2002-12/msg00066.html > > finds 'rsync -aH --delete' faster than 'cp -a'. Perhaps though this > was some idiosyncracy of my test (the main purpose wasn't to benchmark > rsync anyway). The referenced mail message describes the benchmark as: | The directory backed up or restored had 1 1-byte files That isn't a very good benchmark. 10,000 files is not that many and being 1 byte means that all that is measured is the filesystem meta-data, node-creation time and overhead. That the test used the --delete option indicates that some percentage of the files would have not been touched by rsync. My guess is that the unmodified files account for cp -a being slower; rsync processed fewer files than cp. The fact that the benchmark description does not indicate the actual rate of change (a determining factor for rsync and, i assume, rdiff-backup) makes it decidedly dubious. The test may well be invalidated by caches. I believe the speed complaint had to do with files that have significant amounts of data in them. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: FW: rsync performance
On Fri, 12 Sep 2003 17:05:09 -0700 jw schultz <[EMAIL PROTECTED]> wrote: > Rsync is not an efficient local copy utility. It can be > used for local copying but local and high-bandwidth network > speed is sacrificed for low-bandwidth performance and for > data integrity. I've been surprised at how fast rsync can copy locally. For instance, http://mail.nongnu.org/archive/html/rdiff-backup-users/2002-12/msg00066.html finds 'rsync -aH --delete' faster than 'cp -a'. Perhaps though this was some idiosyncracy of my test (the main purpose wasn't to benchmark rsync anyway). -- Ben Escoto pgp0.pgp Description: PGP signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync error: error in rsync protocol data stream (code 12) at io.c(463)
On Sat, 13 Sep 2003, Dag Wieers wrote: > Using rsync-2.5.6 I get the exact same error: > > building file list ... > 28844 files to consider > apt/packages/ > apt/packages/avifile/ > apt/packages/avifile/avifile-0.7.34-1.dag.rh90.i386.rpm > rsync: writefd_unbuffered failed to write 4 bytes: phase "unknown": Broken pipe > rsync error: error in rsync protocol data stream (code 12) at io.c(515) > > I'm now going to test with an unpatched rsync, although looking at the Red > Hat patches I don't see anything that could cause this. Ok, I found the FAQ ;) The remote disk is full. Is there a chance that rsync could give a more descriptive error ? The reason why I didn't look in the FAQ right away was because it looked like a real bug, not something that is known. -- dag wieers, [EMAIL PROTECTED], http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors] -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync error: error in rsync protocol data stream (code 12) at io.c(463)
On Sat, 13 Sep 2003, Dag Wieers wrote:
> I'm having a problem rsyncing one file (since I signed it). It seems that
> the content of a file is able to cause problems in the protocol.
>
> building file list ...
> 28820 files to consider
> apt/packages/avifile/
> apt/packages/avifile/avifile-0.7.34-1.dag.rh90.i386.rpm
> rsync: error writing 4 unbuffered bytes - exiting: Broken pipe
> rsync error: error in rsync protocol data stream (code 12) at io.c(463)
>
> I'm using rsync-2.5.5-4 (the rsync shipped with RH9). The first time rsync
> halted (indefinitely), every other run gives the above error.
Using rsync-2.5.6 I get the exact same error:
building file list ...
28844 files to consider
apt/packages/
apt/packages/avifile/
apt/packages/avifile/avifile-0.7.34-1.dag.rh90.i386.rpm
rsync: writefd_unbuffered failed to write 4 bytes: phase "unknown": Broken pipe
rsync error: error in rsync protocol data stream (code 12) at io.c(515)
I'm now going to test with an unpatched rsync, although looking at the Red
Hat patches I don't see anything that could cause this.
PS: Is there a reason why the Red Hat patches are not applied to the rsync
sourcecode ? I've attached them for inspection.
-- dag wieers, [EMAIL PROTECTED], http://dag.wieers.com/ --
[Any errors in spelling, tact or fact are transmission errors]
--- io.c.orig 2003-09-13 22:36:40.0 +0200
+++ io.c2003-09-13 22:39:13.0 +0200
@@ -509,7 +509,7 @@
* across the stream */
io_multiplexing_close();
rprintf(FERROR, RSYNC_NAME
- ": writefd_unbuffered failed to write %ld
bytes: phase \"%s\": %s\n",
+ ": writefd_unbuffered failed to write %lu
bytes: phase \"%s\": %s\n",
(long) len, io_write_phase,
strerror(errno));
exit_cleanup(RERR_STREAMIO);
@@ -605,7 +605,7 @@
}
while (len) {
- int n = MIN((int) len, IO_BUFFER_SIZE-io_buffer_count);
+ int n = MIN((ssize_t) len, IO_BUFFER_SIZE-io_buffer_count);
if (n > 0) {
memcpy(io_buffer+io_buffer_count, buf, n);
buf += n;
--- match.c.orig2003-09-13 22:39:22.0 +0200
+++ match.c 2003-09-13 22:42:59.0 +0200
@@ -153,12 +153,12 @@
last_i = -1;
if (verbose > 2)
- rprintf(FINFO,"hash search b=%ld len=%.0f\n",
+ rprintf(FINFO,"hash search b=%lu len=%.0f\n",
(long) s->n, (double)len);
/* cast is to make s->n signed; it should always be reasonably
* small */
- k = MIN(len, (OFF_T) s->n);
+ k = MIN(len, (ssize_t) s->n);
map = (schar *)map_ptr(buf,0,k);
@@ -173,7 +173,7 @@
end = len + 1 - s->sums[s->count-1].len;
if (verbose > 3)
- rprintf(FINFO, "hash search s->n=%ld len=%.0f count=%ld\n",
+ rprintf(FINFO, "hash search s->n=%lu len=%.0f count=%lu\n",
(long) s->n, (double) len, (long) s->count);
do {
@@ -190,13 +190,13 @@
sum = (s1 & 0x) | (s2 << 16);
tag_hits++;
- for (; j < (int) s->count && targets[j].t == t; j++) {
+ for (; j < (ssize_t) s->count && targets[j].t == t; j++) {
int l, i = targets[j].i;
if (sum != s->sums[i].sum1) continue;
/* also make sure the two blocks are the same length */
- l = MIN(s->n,len-offset);
+ l = MIN((ssize_t) s->n,len-offset);
if (l != s->sums[i].len) continue;
if (verbose > 3)
@@ -216,7 +216,7 @@
/* we've found a match, but now check to see
if last_i can hint at a better match */
- for (j++; j < (int) s->count && targets[j].t == t; j++) {
+ for (j++; j < (ssize_t) s->count && targets[j].t == t; j++) {
int i2 = targets[j].i;
if (i2 == last_i + 1) {
if (sum != s->sums[i2].sum1) break;
@@ -232,7 +232,7 @@
matched(f,s,buf,offset,i);
offset += s->sums[i].len - 1;
- k = MIN((len-offset), s->n);
+ k = MIN((len-offset), (ssize_t) s->n);
map = (schar *)map_ptr(buf,offset,k);
sum = get_checksu
rsync error: error in rsync protocol data stream (code 12) at io.c(463)
Hi, I'm having a problem rsyncing one file (since I signed it). It seems that the content of a file is able to cause problems in the protocol. building file list ... 28820 files to consider apt/packages/avifile/ apt/packages/avifile/avifile-0.7.34-1.dag.rh90.i386.rpm rsync: error writing 4 unbuffered bytes - exiting: Broken pipe rsync error: error in rsync protocol data stream (code 12) at io.c(463) I'm using rsync-2.5.5-4 (the rsync shipped with RH9). The first time rsync halted (indefinitely), every other run gives the above error. Kind regards, -- dag wieers, [EMAIL PROTECTED], http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors] -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
