Re: Which rsync version?
Wayne Davison wrote: > On Fri, Aug 03, 2007 at 09:20:22PM +0200, [EMAIL PROTECTED] wrote: >> #0 0x2b4167a93ee5 in free () from /lib64/libc.so.6 >> #1 0x00431c9b in pool_free_old (p=0x66fb90, >> addr=) at lib/pool_alloc.c:266 > > I found and fixed a bug in pool_free_old() that could leave some freed > extents on the list of currently allocated extents (in one code path). > If that were to happen, rsync would try to double-free the memory. > > Hopefully the latest version in CVS (and the "nightly" tar file) will > avoid this crash now. Great! I've been experimenting with the incremental recursion in rsync 3.0-cvs to simplify our backup process for Wikipedia's uploaded media files -- about 2 TB in several million files, too big for rsync 2.x without breaking it up into a huge number of sub-invocations. I've been getting segfaults anywhere from a few minutes to a couple days into the copy, and the backtrace I managed to get seems to match this one. Thanks for the fix -- will let you know if we encounter any more problems. :) -- brion vibber (brion @ wikimedia.org) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
Mon, 20 Aug 2007 22:11:12 -0700, wayned wrote: > On Fri, Aug 03, 2007 at 09:20:22PM +0200, Sven.Hartrumpf wrote: > > #0 0x2b4167a93ee5 in free () from /lib64/libc.so.6 > > #1 0x00431c9b in pool_free_old (p=0x66fb90, > > addr=) at lib/pool_alloc.c:266 > > I found and fixed a bug in pool_free_old() that could leave some freed > extents on the list of currently allocated extents (in one code path). > If that were to happen, rsync would try to double-free the memory. > > Hopefully the latest version in CVS (and the "nightly" tar file) will > avoid this crash now. Yes, it does! Many thanks. Sven pgp4UAsGfimoy.pgp Description: PGP signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
On Fri, Aug 03, 2007 at 09:20:22PM +0200, [EMAIL PROTECTED] wrote: > #0 0x2b4167a93ee5 in free () from /lib64/libc.so.6 > #1 0x00431c9b in pool_free_old (p=0x66fb90, > addr=) at lib/pool_alloc.c:266 I found and fixed a bug in pool_free_old() that could leave some freed extents on the list of currently allocated extents (in one code path). If that were to happen, rsync would try to double-free the memory. Hopefully the latest version in CVS (and the "nightly" tar file) will avoid this crash now. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
On Wed, Aug 08, 2007 at 01:07:09PM +1200, Jason Haar wrote: > will v3 also allow compression of the initial directory meta-data? Not at present. That is something to consider adding, but may not make it into the first 3.0.0 release. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
Paul Slootman wrote: > No, which suggests that when not preserving hardlinks, the CVS version > is much faster (it starts transferring during the building of the file > list, instead of waiting until both ends have generated the file list). > That sounds pretty cool - will v3 also allow compression of the initial directory meta-data? (for others: currently the "-z" option only compresses files - not the initial data that allows the rsync client and server to compare file details - so they can decide what to send) (I saw some comment via Google from years back that the initial "file listing" command is "nearly" compressed and so doesn't need it. However, in my totally unscientific test I just sniffed a transfer, extracted the file listing data, and compressed it with gzip. Knocked 66% off the size...) -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
Fri, 3 Aug 2007 15:06:51 -0400, hashproduct+rsync wrote: > No, that's just another SIGPIPE. There are a total of three rsync > processes, forked in the following pattern: > > sender > \_ generator > \_ receiver > > Your first backtrace was for the sender. Setting gdb to always follow > the child gave you the backtrace for the receiver. But the generator > is the one that is crashing and taking the others down with it. Thanks Matt. So here goes the backtrace from the third process, the generator: (gdb) catch fork Catchpoint 1 (fork) (gdb) run -a --progress -v /cl/ /data/cl Starting program: /programs/linux/rsync-3.0.0-64/bin/rsync -a --progress -v /cl/ /data/cl Catchpoint 1 (forked process 28707), 0x2b4167ab95bd in fork () from /lib64/libc.so.6 (gdb) set follow-fork-mode child (gdb) continue Continuing. sending incremental file list [Switching to process 28707] Catchpoint 1 (forked process 28708), 0x2b4167ab95bd in fork () from /lib64/libc.so.6 (gdb) set follow-fork-mode parent (gdb) continue Continuing. ./ Program received signal SIGSEGV, Segmentation fault. 0x2b4167a93ee5 in free () from /lib64/libc.so.6 (gdb) bt #0 0x2b4167a93ee5 in free () from /lib64/libc.so.6 #1 0x00431c9b in pool_free_old (p=0x66fb90, addr=) at lib/pool_alloc.c:266 #2 0x00404354 in flist_free (flist=0x730db0) at flist.c:2229 #3 0x0040da9c in check_for_finished_files (itemizing=1, code=FLOG, check_redo=1) at generator.c:1851 #4 0x0040df44 in generate_files (f_out=1, local_name=) at generator.c:1974 #5 0x00417007 in do_recv (f_in=0, f_out=1, local_name=0x0) at main.c:774 #6 0x004177cd in start_server (f_in=0, f_out=1, argc=1, argv=0x7fff4329d6a8) at main.c:884 #7 0x00418875 in child_main (argc=1768711524, argv=0x4) at main.c:891 #8 0x0042af95 in local_child (argc=2, argv=0x7fff4329d6a0, f_in=0x7fff4329f68c, f_out=0x7fff4329f688, child_main=0x418860 ) at pipe.c:147 #9 0x0041817b in main (argc=2, argv=0x66c9b0) at main.c:445 (gdb) > uname -a Linux ... 2.6.18.8-0.5-default #1 SMP Fri Jun 22 12:17:53 UTC 2007 x86_64 x86_64 x86_64 GNU/Linux -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
On 8/3/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > Thanks. Now gdb's output it might be more useful? > > (gdb) show follow-fork-mode > Debugger response to a program call of fork or vfork is "parent". > (gdb) set follow-fork-mode child > (gdb) show follow-fork-mode > Debugger response to a program call of fork or vfork is "child". > (gdb) run -a --progress -v /cl/ /data/cl > Starting program: /programs/linux/rsync-3.0.0-64/bin/rsync -a --progress -v > /cl/ /data/cl > sending incremental file list > Program received signal SIGPIPE, Broken pipe. > [Switching to process 18620] > 0x2b89a0b4c280 in __write_nocancel () from /lib64/libc.so.6 > (gdb) bt > #0 0x2b89a0b4c280 in __write_nocancel () from /lib64/libc.so.6 > #1 0x00420acc in writefd_unbuffered (fd=7, > buf=0x66cb20 "8n&/schulorganisation,property=source.txt", len=2122) > at io.c:1387 > #2 0x0042126a in io_flush (flush_it_all=1) at io.c:1442 > #3 0x00409c8c in read_ndx_and_attrs (f_in=0, > iflag_ptr=0x7fff0a2284c8, type_ptr=0x7fff0a2284cf "", > buf=0x7fff0a2263d0 "", len_ptr=0x7fff0a2284c4) at rsync.c:184 > #4 0x0040f18e in recv_files (f_in=0, local_name=0x0) at > receiver.c:371 > #5 0x00416f39 in do_recv (f_in=0, f_out=1, local_name=0x0) > at main.c:726 > #6 0x004177cd in start_server (f_in=0, f_out=1, argc=1, > argv=0x7fff0a229638) at main.c:884 > #7 0x00418875 in child_main (argc=2122, argv=0x) > at main.c:891 > #8 0x0042af95 in local_child (argc=2, argv=0x7fff0a229630, > f_in=0x7fff0a22b61c, f_out=0x7fff0a22b618, > child_main=0x418860 ) at pipe.c:147 > #9 0x0041817b in main (argc=2, argv=0x66c9b0) at main.c:445 > (gdb) No, that's just another SIGPIPE. There are a total of three rsync processes, forked in the following pattern: sender \_ generator \_ receiver Your first backtrace was for the sender. Setting gdb to always follow the child gave you the backtrace for the receiver. But the generator is the one that is crashing and taking the others down with it. To trace the generator, try `set follow-fork-mode ask'. When you are asked about the first fork, choose the child (the generator). When you are asked about the second, stick with the parent (the generator). If your gdb doesn't support `set follow-fork-mode ask' (mine doesn't), you can set a catchpoint on the fork to give you the opportunity to change the follow-fork-mode from child to parent, like this: [EMAIL PROTECTED] test]$ gdb rsync GNU gdb Red Hat Linux (6.6-15.fc7rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"... (no debugging symbols found) Using host libthread_db library "/lib/libthread_db.so.1". (gdb) catch fork Catchpoint 1 (fork) (gdb) run -rni src/ dest/ Starting program: /usr/bin/rsync -rni src/ dest/ (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) Catchpoint 1 (forked process 2957), 0x00110402 in __kernel_vsyscall () (gdb) set follow-fork-mode child (gdb) continue Continuing. [New process 2957] [Switching to process 2957] Catchpoint 1 (forked process 2958), 0x00110402 in __kernel_vsyscall () (gdb) set follow-fork-mode parent (gdb) continue Continuing. [Detaching after fork from child process 2958. (Try `set detach-on-fork off'.)] skipping non-regular file "bar" skipping non-regular file "baz" Program exited normally. (gdb) quit Of course, you'll get a crash instead of "Program exited normally". Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
Fri, 3 Aug 2007 11:48:02 +0200, paul wrote: > On Fri 03 Aug 2007, [EMAIL PROTECTED] wrote: > > > > I understand, but I have no idea how I can solve this problem. > > > > Is rsync starting a second rsync instance? > > Yes. > > http://www.delorie.com/gnu/docs/gdb/gdb_26.html may help. Thanks. Now gdb's output it might be more useful? (gdb) show follow-fork-mode Debugger response to a program call of fork or vfork is "parent". (gdb) set follow-fork-mode child (gdb) show follow-fork-mode Debugger response to a program call of fork or vfork is "child". (gdb) run -a --progress -v /cl/ /data/cl Starting program: /programs/linux/rsync-3.0.0-64/bin/rsync -a --progress -v /cl/ /data/cl sending incremental file list Program received signal SIGPIPE, Broken pipe. [Switching to process 18620] 0x2b89a0b4c280 in __write_nocancel () from /lib64/libc.so.6 (gdb) bt #0 0x2b89a0b4c280 in __write_nocancel () from /lib64/libc.so.6 #1 0x00420acc in writefd_unbuffered (fd=7, buf=0x66cb20 "8n&/schulorganisation,property=source.txt", len=2122) at io.c:1387 #2 0x0042126a in io_flush (flush_it_all=1) at io.c:1442 #3 0x00409c8c in read_ndx_and_attrs (f_in=0, iflag_ptr=0x7fff0a2284c8, type_ptr=0x7fff0a2284cf "", buf=0x7fff0a2263d0 "", len_ptr=0x7fff0a2284c4) at rsync.c:184 #4 0x0040f18e in recv_files (f_in=0, local_name=0x0) at receiver.c:371 #5 0x00416f39 in do_recv (f_in=0, f_out=1, local_name=0x0) at main.c:726 #6 0x004177cd in start_server (f_in=0, f_out=1, argc=1, argv=0x7fff0a229638) at main.c:884 #7 0x00418875 in child_main (argc=2122, argv=0x) at main.c:891 #8 0x0042af95 in local_child (argc=2, argv=0x7fff0a229630, f_in=0x7fff0a22b61c, f_out=0x7fff0a22b618, child_main=0x418860 ) at pipe.c:147 #9 0x0041817b in main (argc=2, argv=0x66c9b0) at main.c:445 (gdb) Sven pgpHrFeFfIDb8.pgp Description: PGP signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
On 8/3/07, Paul Slootman <[EMAIL PROTECTED]> wrote: > On Fri 03 Aug 2007, [EMAIL PROTECTED] wrote: > > > Program received signal SIGPIPE, Broken pipe. > > 0xe410 in __kernel_vsyscall () > > Unfortunately this means you were not tracing the process that failed; > you were tracing the process that forked another process and got a > SIGPIPE when trying to write to that process after that process died due > to the double free error... Hints for debugging multiple processes: http://sourceware.org/gdb/current/onlinedocs/gdb_5.html#SEC28 Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
On Fri 03 Aug 2007, [EMAIL PROTECTED] wrote: > Program received signal SIGPIPE, Broken pipe. > 0xe410 in __kernel_vsyscall () Unfortunately this means you were not tracing the process that failed; you were tracing the process that forked another process and got a SIGPIPE when trying to write to that process after that process died due to the double free error... Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
Fri, 3 Aug 2007 10:10:30 +0200, paul wrote: > On Fri 03 Aug 2007, Sven wrote: > > > > Yes, the cvs version from 20070803T08 started very modestly in terms of RAM, > > but after some minutes: > > > rsync -a --progress -v /c/ /data/c (source is mounted via nfs) > > ... > > ces/002495.ces > > 105851 100% 859.88kB/s0:00:00 (xfer#23, to-check=5762/13367) > > *** glibc detected *** rsync: munmap_chunk(): invalid pointer: 0x080a8cb0 > > *** > > === Backtrace: = > > /lib/libc.so.6[0x2a97e6e1] > > rsync[0x807c0f0] > > rsync[0x804b782] > > Do you have any means of translating this backtrace to functions and > source files with line numbers? E.g. as the gdb 'bt' command does, when > the code is compiled with debugging info? (-g in gcc). Yes, but the error looks somewhat different now (but is reproducable, at least): > gdb64 rsync GNU gdb 6.4 Copyright 2005 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i686-pc-linux-gnu"...Using host libthread_db library "/lib/libthread_db.so.1". (gdb) run -a --progress -v /cl/ /data/cl Starting program: /programs/linux/rsync-cvs/bin/rsync -a --progress -v /cl/ /data/cl warning: Lowest section in system-supplied DSO at 0xe000 is .hash at e0b4 sending incremental file list *** glibc detected *** /programs/linux/rsync-cvs/bin/rsync: double free or corruption (out): 0x2aa42180 *** === Backtrace: = /lib/libc.so.6[0x2a97b6e1] /lib/libc.so.6(cfree+0x89)[0x2a97cd79] /programs/linux/rsync-cvs/bin/rsync[0x807c0f0] /programs/linux/rsync-cvs/bin/rsync[0x804b782] /programs/linux/rsync-cvs/bin/rsync[0x8056215] /programs/linux/rsync-cvs/bin/rsync[0x8056825] /programs/linux/rsync-cvs/bin/rsync[0x806053a] /programs/linux/rsync-cvs/bin/rsync[0x8060d89] /programs/linux/rsync-cvs/bin/rsync[0x8062108] /programs/linux/rsync-cvs/bin/rsync[0x807556f] /programs/linux/rsync-cvs/bin/rsync[0x80618c5] /lib/libc.so.6(__libc_start_main+0xdc)[0x2a92cf9c] /programs/linux/rsync-cvs/bin/rsync[0x804a6c1] === Memory map: 08048000-08092000 r-xp 00:14 389384 /programs/linux/rsync-cvs/bin/rsync 08092000-08094000 rwxp 0004a000 00:14 389384 /programs/linux/rsync-cvs/bin/rsync 08094000-0810a000 rwxp 08094000 00:00 0 [heap] 2a60-2a621000 rwxp 2a60 00:00 0 2a621000-2a70 ---p 2a621000 00:00 0 2a732000-2a773000 rwxp 2a732000 00:00 0 2a7f5000-2a82a000 r-xs 08:02 1577380 /var/run/nscd/passwd 2a84b000-2a88d000 rwxp 2a84b000 00:00 0 2a88d000-2a8c2000 r-xs 08:02 1577476 /var/run/nscd/group 2a8e3000-2a916000 r-xp 08:02 526055 /usr/lib/locale/en_US/LC_CTYPE 2a916000-2a917000 rwxp 2a916000 00:00 0 2a917000-2aa3f000 r-xp 08:02 2232595 /lib/libc-2.5.so 2aa3f000-2aa4 r-xp 00128000 08:02 2232595 /lib/libc-2.5.so 2aa4-2aa42000 rwxp 00129000 08:02 2232595 /lib/libc-2.5.so 2aa42000-2aa45000 rwxp 2aa42000 00:00 0 2aa45000-2aa4c000 r-xp 08:02 480803 /usr/lib/libpopt.so.0.0.0 2aa4c000-2aa4e000 rwxp 6000 08:02 480803 /usr/lib/libpopt.so.0.0.0 2aa5f000-2aa69000 r-xp 08:02 2232639 /lib/libgcc_s.so.1 2aa69000-2aa6b000 rwxp 9000 08:02 2232639 /lib/libgcc_s.so.1 2aa6b000-2aa72000 r-xs 08:02 542502 /usr/lib/gconv/gconv-modules.cache 2aa72000-2aa73000 rwxp 2aa72000 00:00 0 2aa73000-2aa8e000 r-xp 08:02 2232588 /lib/ld-2.5.so 2aa8e000-2aa9 rwxp 0001a000 08:02 2232588 /lib/ld-2.5.so ffd85000-ffd8e000 rw-p ffd85000 00:00 0 [stack] e000-f000 r-xp e000 00:00 0 Program received signal SIGPIPE, Broken pipe. 0xe410 in __kernel_vsyscall () (gdb) bt #0 0xe410 in __kernel_vsyscall () #1 0x2a9c7903 in __write_nocancel () from /lib/libc.so.6 #2 0x0806ac22 in writefd_unbuffered (fd=7, buf=0xffd84c30 "ü\017", len=4) at io.c:1387 #3 0x0806b26b in mplex_write (fd=7, code=, buf=0x80ad6d8 "\030DdÓѺ \0019", len=4092, convert=0) at io.c:482 #4 0x0806b423 in io_flush (flush_it_all=0) at io.c:1440 #5 0x0806b58c in writefd (fd=, buf=0xffd86095 "qØÿhqØÿKÓ\004\b\a", len=0) at io.c:1474 #6 0x0806b70c in write_byte (f=7, c=32 ' ') at io.c:1584 #7 0x0804d34b in send_file_name (f=7, flist=, fname=, stp=0x0, flags=65540, filter_level=2) at flist.c:495 #8 0x0804dce2 in send_directory (f=7, flist=0x80b2
Re: Which rsync version?
On Fri 03 Aug 2007, [EMAIL PROTECTED] wrote: > > Yes, the cvs version from 20070803T08 started very modestly in terms of RAM, > but after some minutes: > > rsync -a --progress -v /c/ /data/c (source is mounted via nfs) > ... > ces/002495.ces > 105851 100% 859.88kB/s0:00:00 (xfer#23, to-check=5762/13367) > *** glibc detected *** rsync: munmap_chunk(): invalid pointer: 0x080a8cb0 *** > === Backtrace: = > /lib/libc.so.6[0x2a97e6e1] > rsync[0x807c0f0] > rsync[0x804b782] Do you have any means of translating this backtrace to functions and source files with line numbers? E.g. as the gdb 'bt' command does, when the code is compiled with debugging info? (-g in gcc). Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
Thu, 2 Aug 2007 16:52:53 +0200, paul wrote: > On Thu 02 Aug 2007, Sven wrote: > > > > I once tried to rsync around 100 GB (10 million files), but version > > 2.6.6 needed too much RAM and was too slow. > > I believe 2.6.7 had a number of memory-saving options; each subsequent > version will of course have other improvements. 2.6.9 is the current > official release. > > If those files don't have hardlinks (or you're not interested > in preserving hardlinks), No hardlinks. > you can greatly improve performance > by trying the 3.0.0 cvs snapshot. Yes, the cvs version from 20070803T08 started very modestly in terms of RAM, but after some minutes: > rsync -a --progress -v /c/ /data/c (source is mounted via nfs) ... ces/002495.ces 105851 100% 859.88kB/s0:00:00 (xfer#23, to-check=5762/13367) *** glibc detected *** rsync: munmap_chunk(): invalid pointer: 0x080a8cb0 *** === Backtrace: = /lib/libc.so.6[0x2a97e6e1] rsync[0x807c0f0] rsync[0x804b782] rsync[0x8056215] rsync[0x8056825] rsync[0x806053a] rsync[0x8060d89] rsync[0x8062108] rsync[0x807556f] rsync[0x80618c5] /lib/libc.so.6(__libc_start_main+0xdc)[0x2a92ff9c] rsync[0x804a6c1] === Memory map: 08048000-08092000 r-xp 00:14 389384 /linux/rsync-cvs/bin/rsync 08092000-08094000 rw-p 0004a000 00:14 389384 /linux/rsync-cvs/bin/rsync 08094000-0866a000 rw-p 08094000 00:00 0 [heap] 2a7c8000-2a7d2000 r-xp 08:02 2232639 /lib/libgcc_s.so.1 2a7d2000-2a7d4000 rw-p 9000 08:02 2232639 /lib/libgcc_s.so.1 2a7f8000-2a82d000 r--s 08:02 1577380 /var/run/nscd/passwd 2a84e000-2a89 rw-p 2a84e000 00:00 0 2a89-2a8c5000 r--s 08:02 1577476 /var/run/nscd/group 2a8e6000-2a919000 r--p 08:02 526055 /usr/lib/locale/en_US/LC_CTYPE 2a919000-2a91a000 rw-p 2a919000 00:00 0 2a91a000-2aa42000 r-xp 08:02 2232595 /lib/libc-2.5.so 2aa42000-2aa43000 r--p 00128000 08:02 2232595 /lib/libc-2.5.so 2aa43000-2aa45000 rw-p 00129000 08:02 2232595 /lib/libc-2.5.so 2aa45000-2aa48000 rw-p 2aa45000 00:00 0 2aa48000-2aa4f000 r-xp 08:02 480803 /usr/lib/libpopt.so.0.0.0 2aa4f000-2aa51000 rw-p 6000 08:02 480803 /usr/lib/libpopt.so.0.0.0 2aa6e000-2aa75000 r--s 08:02 542502 /usr/lib/gconv/gconv-modules.cache 2aa75000-2aa76000 rw-p 2aa75000 00:00 0 2aa76000-2aa91000 r-xp 08:02 2232588 /lib/ld-2.5.so 2aa91000-2aa93000 rw-p 0001a000 08:02 2232588 /lib/ld-2.5.so ffd49000-ffd54000 rw-p ffd49000 00:00 0 [stack] e000-f000 r-xp e000 00:00 0 rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken pipe (32) rsync: connection unexpectedly closed (2322 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.0cvs] [ERROR: 12] > Is this a known problem? Sven pgpQQdRiTXCbS.pgp Description: PGP signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
The issue is that asking to preserve hardlinks turns off incremental recursion mode, which was what gives the performance benefit. Ahh... ok, sorry... -- Best regards, Charles -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
On Thu 02 Aug 2007, Charles Marcus wrote: > >If those files don't have hardlinks (or you're not interested > >in preserving hardlinks), you can greatly improve performance > >by trying the 3.0.0 cvs snapshot. > > Which suggests that the current CVS version won't preserve hardlinks? No, which suggests that when not preserving hardlinks, the CVS version is much faster (it starts transferring during the building of the file list, instead of waiting until both ends have generated the file list). When preserving hardlinks (by passing the -H option as usual) it will still need to first collect the whole file list. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
On 8/2/07, Charles Marcus <[EMAIL PROTECTED]> wrote: > > If those files don't have hardlinks (or you're not interested > > in preserving hardlinks), you can greatly improve performance > > by trying the 3.0.0 cvs snapshot. > > Which suggests that the current CVS version won't preserve hardlinks? No. The issue is that asking to preserve hardlinks turns off incremental recursion mode, which was what gives the performance benefit. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
If those files don't have hardlinks (or you're not interested in preserving hardlinks), you can greatly improve performance by trying the 3.0.0 cvs snapshot. Which suggests that the current CVS version won't preserve hardlinks? -- Best regards, Charles -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
On Thu 02 Aug 2007, [EMAIL PROTECTED] wrote: > > I once tried to rsync around 100 GB (10 million files), but version > 2.6.6 needed too much RAM and was too slow. I believe 2.6.7 had a number of memory-saving options; each subsequent version will of course have other improvements. 2.6.9 is the current official release. If those files don't have hardlinks (or you're not interested in preserving hardlinks), you can greatly improve performance by trying the 3.0.0 cvs snapshot. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
