Which rsync version?
Hi all. I once tried to rsync around 100 GB (10 million files), but version 2.6.6 needed too much RAM and was too slow. Is one of the snapshots stable enough to try this again? Greetings Sven pgp2q99V2fn3K.pgp Description: PGP signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
Thu, 2 Aug 2007 16:52:53 +0200, paul wrote: > On Thu 02 Aug 2007, Sven wrote: > > > > I once tried to rsync around 100 GB (10 million files), but version > > 2.6.6 needed too much RAM and was too slow. > > I believe 2.6.7 had a number of memory-saving options; each subsequent > version will of course have other improvements. 2.6.9 is the current > official release. > > If those files don't have hardlinks (or you're not interested > in preserving hardlinks), No hardlinks. > you can greatly improve performance > by trying the 3.0.0 cvs snapshot. Yes, the cvs version from 20070803T08 started very modestly in terms of RAM, but after some minutes: > rsync -a --progress -v /c/ /data/c (source is mounted via nfs) ... ces/002495.ces 105851 100% 859.88kB/s0:00:00 (xfer#23, to-check=5762/13367) *** glibc detected *** rsync: munmap_chunk(): invalid pointer: 0x080a8cb0 *** === Backtrace: = /lib/libc.so.6[0x2a97e6e1] rsync[0x807c0f0] rsync[0x804b782] rsync[0x8056215] rsync[0x8056825] rsync[0x806053a] rsync[0x8060d89] rsync[0x8062108] rsync[0x807556f] rsync[0x80618c5] /lib/libc.so.6(__libc_start_main+0xdc)[0x2a92ff9c] rsync[0x804a6c1] === Memory map: 08048000-08092000 r-xp 00:14 389384 /linux/rsync-cvs/bin/rsync 08092000-08094000 rw-p 0004a000 00:14 389384 /linux/rsync-cvs/bin/rsync 08094000-0866a000 rw-p 08094000 00:00 0 [heap] 2a7c8000-2a7d2000 r-xp 08:02 2232639 /lib/libgcc_s.so.1 2a7d2000-2a7d4000 rw-p 9000 08:02 2232639 /lib/libgcc_s.so.1 2a7f8000-2a82d000 r--s 08:02 1577380 /var/run/nscd/passwd 2a84e000-2a89 rw-p 2a84e000 00:00 0 2a89-2a8c5000 r--s 08:02 1577476 /var/run/nscd/group 2a8e6000-2a919000 r--p 08:02 526055 /usr/lib/locale/en_US/LC_CTYPE 2a919000-2a91a000 rw-p 2a919000 00:00 0 2a91a000-2aa42000 r-xp 08:02 2232595 /lib/libc-2.5.so 2aa42000-2aa43000 r--p 00128000 08:02 2232595 /lib/libc-2.5.so 2aa43000-2aa45000 rw-p 00129000 08:02 2232595 /lib/libc-2.5.so 2aa45000-2aa48000 rw-p 2aa45000 00:00 0 2aa48000-2aa4f000 r-xp 08:02 480803 /usr/lib/libpopt.so.0.0.0 2aa4f000-2aa51000 rw-p 6000 08:02 480803 /usr/lib/libpopt.so.0.0.0 2aa6e000-2aa75000 r--s 08:02 542502 /usr/lib/gconv/gconv-modules.cache 2aa75000-2aa76000 rw-p 2aa75000 00:00 0 2aa76000-2aa91000 r-xp 08:02 2232588 /lib/ld-2.5.so 2aa91000-2aa93000 rw-p 0001a000 08:02 2232588 /lib/ld-2.5.so ffd49000-ffd54000 rw-p ffd49000 00:00 0 [stack] e000-f000 r-xp e000 00:00 0 rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken pipe (32) rsync: connection unexpectedly closed (2322 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.0cvs] [ERROR: 12] > Is this a known problem? Sven pgpQQdRiTXCbS.pgp Description: PGP signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
Fri, 3 Aug 2007 10:10:30 +0200, paul wrote: > On Fri 03 Aug 2007, Sven wrote: > > > > Yes, the cvs version from 20070803T08 started very modestly in terms of RAM, > > but after some minutes: > > > rsync -a --progress -v /c/ /data/c (source is mounted via nfs) > > ... > > ces/002495.ces > > 105851 100% 859.88kB/s0:00:00 (xfer#23, to-check=5762/13367) > > *** glibc detected *** rsync: munmap_chunk(): invalid pointer: 0x080a8cb0 > > *** > > === Backtrace: = > > /lib/libc.so.6[0x2a97e6e1] > > rsync[0x807c0f0] > > rsync[0x804b782] > > Do you have any means of translating this backtrace to functions and > source files with line numbers? E.g. as the gdb 'bt' command does, when > the code is compiled with debugging info? (-g in gcc). Yes, but the error looks somewhat different now (but is reproducable, at least): > gdb64 rsync GNU gdb 6.4 Copyright 2005 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i686-pc-linux-gnu"...Using host libthread_db library "/lib/libthread_db.so.1". (gdb) run -a --progress -v /cl/ /data/cl Starting program: /programs/linux/rsync-cvs/bin/rsync -a --progress -v /cl/ /data/cl warning: Lowest section in system-supplied DSO at 0xe000 is .hash at e0b4 sending incremental file list *** glibc detected *** /programs/linux/rsync-cvs/bin/rsync: double free or corruption (out): 0x2aa42180 *** === Backtrace: = /lib/libc.so.6[0x2a97b6e1] /lib/libc.so.6(cfree+0x89)[0x2a97cd79] /programs/linux/rsync-cvs/bin/rsync[0x807c0f0] /programs/linux/rsync-cvs/bin/rsync[0x804b782] /programs/linux/rsync-cvs/bin/rsync[0x8056215] /programs/linux/rsync-cvs/bin/rsync[0x8056825] /programs/linux/rsync-cvs/bin/rsync[0x806053a] /programs/linux/rsync-cvs/bin/rsync[0x8060d89] /programs/linux/rsync-cvs/bin/rsync[0x8062108] /programs/linux/rsync-cvs/bin/rsync[0x807556f] /programs/linux/rsync-cvs/bin/rsync[0x80618c5] /lib/libc.so.6(__libc_start_main+0xdc)[0x2a92cf9c] /programs/linux/rsync-cvs/bin/rsync[0x804a6c1] === Memory map: 08048000-08092000 r-xp 00:14 389384 /programs/linux/rsync-cvs/bin/rsync 08092000-08094000 rwxp 0004a000 00:14 389384 /programs/linux/rsync-cvs/bin/rsync 08094000-0810a000 rwxp 08094000 00:00 0 [heap] 2a60-2a621000 rwxp 2a60 00:00 0 2a621000-2a70 ---p 2a621000 00:00 0 2a732000-2a773000 rwxp 2a732000 00:00 0 2a7f5000-2a82a000 r-xs 08:02 1577380 /var/run/nscd/passwd 2a84b000-2a88d000 rwxp 2a84b000 00:00 0 2a88d000-2a8c2000 r-xs 08:02 1577476 /var/run/nscd/group 2a8e3000-2a916000 r-xp 08:02 526055 /usr/lib/locale/en_US/LC_CTYPE 2a916000-2a917000 rwxp 2a916000 00:00 0 2a917000-2aa3f000 r-xp 08:02 2232595 /lib/libc-2.5.so 2aa3f000-2aa4 r-xp 00128000 08:02 2232595 /lib/libc-2.5.so 2aa4-2aa42000 rwxp 00129000 08:02 2232595 /lib/libc-2.5.so 2aa42000-2aa45000 rwxp 2aa42000 00:00 0 2aa45000-2aa4c000 r-xp 08:02 480803 /usr/lib/libpopt.so.0.0.0 2aa4c000-2aa4e000 rwxp 6000 08:02 480803 /usr/lib/libpopt.so.0.0.0 2aa5f000-2aa69000 r-xp 08:02 2232639 /lib/libgcc_s.so.1 2aa69000-2aa6b000 rwxp 9000 08:02 2232639 /lib/libgcc_s.so.1 2aa6b000-2aa72000 r-xs 08:02 542502 /usr/lib/gconv/gconv-modules.cache 2aa72000-2aa73000 rwxp 2aa72000 00:00 0 2aa73000-2aa8e000 r-xp 08:02 2232588 /lib/ld-2.5.so 2aa8e000-2aa9 rwxp 0001a000 08:02 2232588 /lib/ld-2.5.so ffd85000-ffd8e000 rw-p ffd85000 00:00 0 [stack] e000-f000 r-xp e000 00:00 0 Program received signal SIGPIPE, Broken pipe. 0xe410 in __kernel_vsyscall () (gdb) bt #0 0xe410 in __kernel_vsyscall () #1 0x2a9c7903 in __write_nocancel () from /lib/libc.so.6 #2 0x0806ac22 in writefd_unbuffered (fd=7, buf=0xffd84c30 "ü\017", len=4) at io.c:1387 #3 0x0806b26b in mplex_write (fd=7, code=, buf=0x80ad6d8 "\030DdÓѺ \0019", len=4092, convert=0) at io.c:482 #4 0x0806b423 in io_flush (flush_it_all=0) at io.c:1440 #5 0x0806b58c in writefd (fd=, buf=0xffd86095 "qØÿhqØÿKÓ\004\b\a", len=0) at io.c:1474 #6 0x0806b70c in write_byte (f=7, c=32 ' ') at io.c:1584 #7 0x0804d34b in send_file_name (f=7, flist=, fname=, stp=0x0, flags=65540, filter_level=2) at flist.c:495 #8 0x0804dce2 in send_directory (f=7, flist=0x80b2
Re: Which rsync version?
Fri, 3 Aug 2007 11:48:02 +0200, paul wrote: > On Fri 03 Aug 2007, [EMAIL PROTECTED] wrote: > > > > I understand, but I have no idea how I can solve this problem. > > > > Is rsync starting a second rsync instance? > > Yes. > > http://www.delorie.com/gnu/docs/gdb/gdb_26.html may help. Thanks. Now gdb's output it might be more useful? (gdb) show follow-fork-mode Debugger response to a program call of fork or vfork is "parent". (gdb) set follow-fork-mode child (gdb) show follow-fork-mode Debugger response to a program call of fork or vfork is "child". (gdb) run -a --progress -v /cl/ /data/cl Starting program: /programs/linux/rsync-3.0.0-64/bin/rsync -a --progress -v /cl/ /data/cl sending incremental file list Program received signal SIGPIPE, Broken pipe. [Switching to process 18620] 0x2b89a0b4c280 in __write_nocancel () from /lib64/libc.so.6 (gdb) bt #0 0x2b89a0b4c280 in __write_nocancel () from /lib64/libc.so.6 #1 0x00420acc in writefd_unbuffered (fd=7, buf=0x66cb20 "8n&/schulorganisation,property=source.txt", len=2122) at io.c:1387 #2 0x0042126a in io_flush (flush_it_all=1) at io.c:1442 #3 0x00409c8c in read_ndx_and_attrs (f_in=0, iflag_ptr=0x7fff0a2284c8, type_ptr=0x7fff0a2284cf "", buf=0x7fff0a2263d0 "", len_ptr=0x7fff0a2284c4) at rsync.c:184 #4 0x0040f18e in recv_files (f_in=0, local_name=0x0) at receiver.c:371 #5 0x00416f39 in do_recv (f_in=0, f_out=1, local_name=0x0) at main.c:726 #6 0x004177cd in start_server (f_in=0, f_out=1, argc=1, argv=0x7fff0a229638) at main.c:884 #7 0x00418875 in child_main (argc=2122, argv=0x) at main.c:891 #8 0x0042af95 in local_child (argc=2, argv=0x7fff0a229630, f_in=0x7fff0a22b61c, f_out=0x7fff0a22b618, child_main=0x418860 ) at pipe.c:147 #9 0x0041817b in main (argc=2, argv=0x66c9b0) at main.c:445 (gdb) Sven pgpHrFeFfIDb8.pgp Description: PGP signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
Fri, 3 Aug 2007 15:06:51 -0400, hashproduct+rsync wrote: > No, that's just another SIGPIPE. There are a total of three rsync > processes, forked in the following pattern: > > sender > \_ generator > \_ receiver > > Your first backtrace was for the sender. Setting gdb to always follow > the child gave you the backtrace for the receiver. But the generator > is the one that is crashing and taking the others down with it. Thanks Matt. So here goes the backtrace from the third process, the generator: (gdb) catch fork Catchpoint 1 (fork) (gdb) run -a --progress -v /cl/ /data/cl Starting program: /programs/linux/rsync-3.0.0-64/bin/rsync -a --progress -v /cl/ /data/cl Catchpoint 1 (forked process 28707), 0x2b4167ab95bd in fork () from /lib64/libc.so.6 (gdb) set follow-fork-mode child (gdb) continue Continuing. sending incremental file list [Switching to process 28707] Catchpoint 1 (forked process 28708), 0x2b4167ab95bd in fork () from /lib64/libc.so.6 (gdb) set follow-fork-mode parent (gdb) continue Continuing. ./ Program received signal SIGSEGV, Segmentation fault. 0x2b4167a93ee5 in free () from /lib64/libc.so.6 (gdb) bt #0 0x2b4167a93ee5 in free () from /lib64/libc.so.6 #1 0x00431c9b in pool_free_old (p=0x66fb90, addr=) at lib/pool_alloc.c:266 #2 0x00404354 in flist_free (flist=0x730db0) at flist.c:2229 #3 0x0040da9c in check_for_finished_files (itemizing=1, code=FLOG, check_redo=1) at generator.c:1851 #4 0x0040df44 in generate_files (f_out=1, local_name=) at generator.c:1974 #5 0x00417007 in do_recv (f_in=0, f_out=1, local_name=0x0) at main.c:774 #6 0x004177cd in start_server (f_in=0, f_out=1, argc=1, argv=0x7fff4329d6a8) at main.c:884 #7 0x00418875 in child_main (argc=1768711524, argv=0x4) at main.c:891 #8 0x0042af95 in local_child (argc=2, argv=0x7fff4329d6a0, f_in=0x7fff4329f68c, f_out=0x7fff4329f688, child_main=0x418860 ) at pipe.c:147 #9 0x0041817b in main (argc=2, argv=0x66c9b0) at main.c:445 (gdb) > uname -a Linux ... 2.6.18.8-0.5-default #1 SMP Fri Jun 22 12:17:53 UTC 2007 x86_64 x86_64 x86_64 GNU/Linux -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re:
Sun, 12 Aug 2007 11:25:10 +0200, rekrutacja119 wrote: > Matt McCutchen wrote: > > On 8/4/07, Rekrutacja <[EMAIL PROTECTED]> wrote: > >> i'm trying to use latest CVS rsync, but i'm getting these errors: > >> > >> *** glibc detected *** rsync: munmap_chunk(): invalid pointer: > >> 0x0d757d58 *** > > > > This looks like the same problem Sven Hartrumpf recently encountered > > and described in this thread: > > > > http://lists.samba.org/archive/rsync/2007-August/018197.html > > > > Now that Sven has provided a useful backtrace for the generator, the > > ball is in Wayne's court to figure out what is causing the problem. > > > > Matt > > > > is there any chance that it's fixed or where i can see a progress changelog? I have found a work-around for me: I excluded the problematic directory; the problem seems to be related to file names with special (or many?) characters. I will try to isolate the problem during the next days. Maybe it is easier for you to find the problematic file names in your case? Ciao Sven pgpWyg55sISr2.pgp Description: PGP signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which rsync version?
Mon, 20 Aug 2007 22:11:12 -0700, wayned wrote: > On Fri, Aug 03, 2007 at 09:20:22PM +0200, Sven.Hartrumpf wrote: > > #0 0x2b4167a93ee5 in free () from /lib64/libc.so.6 > > #1 0x00431c9b in pool_free_old (p=0x66fb90, > > addr=) at lib/pool_alloc.c:266 > > I found and fixed a bug in pool_free_old() that could leave some freed > extents on the list of currently allocated extents (in one code path). > If that were to happen, rsync would try to double-free the memory. > > Hopefully the latest version in CVS (and the "nightly" tar file) will > avoid this crash now. Yes, it does! Many thanks. Sven pgp4UAsGfimoy.pgp Description: PGP signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Has anyone experience with rsync out of memory
Fri, 24 Aug 2007 12:00:01 +0200, mjacob wrote: > I calculated 100 bytes a file which lead me to about 350MB > needed for the process. However the process seems to stop > already at getting 16MB. > > Is there any restriction to processes on Unix system > that might influence this? Will set ulimit help? > > ulimit -a: > - > time(seconds)unlimited > file(blocks) 2097151 > data(kbytes) 131072 > stack(kbytes)32768 > memory(kbytes) 32768 This is only 32 MB - so you must raise this limit! pgptnZJ4NbF40.pgp Description: PGP signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
moderation?
Hi. I get the feeling it's time to turn this list into a moderated one ... Greetings Sven pgp4LjVHOUOLr.pgp Description: PGP signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: CVE-2007-4091
Thu, 30 Aug 2007 01:41:01 -0400, danno wrote: > http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-4091 > http://nvd.nist.gov/nvd.cfm?cvename=CVE-2007-4091 SUSE has provided fixed RPMs for SUSE 10.0 - 10.2 some days ago. But rsync's cvs sources do not contain the string "4091", yet. Ciao Sven pgpM8B8zc7WxO.pgp Description: PGP signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Recommended way for initial backup before rsync calls
Hi rsync'ers. What can you recommend to create the initial backup of a partition before calling rsync (under Linux, 100 GB, 10M files, only normal files and some soft links)? (Using rsync takes too long for this initial copy.) Here are some candidates: 1. star -copy -no-fsync -p -xdot -C $1 . $2 2. tar -C $1 -cf - . | tar -C $2 -xf - 3. cp -a $1 $2 4. cpio ... ... Any experiences? Sven pgplb5KECM971.pgp Description: PGP signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Recommended way for initial backup before rsync calls
Mon, 3 Sep 2007 20:00:06 -0400, hashproduct+rsync wrote: > On 9/3/07, Sven.Hartrumpf wrote: > > What can you recommend to create the initial backup of a > > partition before calling rsync > > (under Linux, 100 GB, 10M files, only normal files and > > some soft links)? > > If you're copying an entire partition, another option to consider is > copying the partition raw using dd. Yes, I will try this next time. > > (Using rsync takes too long for this initial copy.) > > I would be surprised if another tool were significantly faster than > rsync. If that is the case, it might be worth trying to understand > why so that rsync can be improved. Currently, I have no time and resources to investigate this :-( If noone else faces this problem, I would suspect that this problem is not caused by rsync, but by our local network. I gave up copying with rsync after 3 days and only 13 GB copied; now I use star for the initial copy ... > Are you using the current CVS > rsync with incremental recursion (which is critical when you're > transferring such a large number of files)? I use the current CVS version, which uses incremental recursion (by default?). Greetings Sven pgppSOof8ZBRC.pgp Description: PGP signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: security bugs (?)
Sat, 29 Sep 2007 10:55:32 +0200, lapo wrote: > Lapo Luchini wrote: > > As a Cygwin rsync package maintainer, the following security fixes have > > been brought to my attention: > > > > http://sources.gentoo.org/viewcvs.py/gentoo-x86/net-misc/rsync/files/rsync-2.6.9-stats-fix.patch > > http://sources.gentoo.org/viewcvs.py/gentoo-x86/net-misc/rsync/files/rsync-2.6.9-fname-obo.patch > > On a closer inspection, the first one doesn't really seem to regard > security... what about the other, aka CVE-2007-4091[1] and SA26493[2]? There is a thread under the subject "CVE-2007-4091" :-) in the archives of this list: http://lists.samba.org/archive/rsync/2007-August/thread.html Sven pgp2s7vXhlK5A.pgp Description: PGP signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: fixing user, group, and mtime with rsync?
Tue, 16 Sep 2008 17:35:40 +, matt wrote: > Rsync will preserve symlink mtimes using the lutimes system call if the > C library and kernel on the receiving machine both support it. Check > that "rsync --version" shows "symtimes", not "no symtimes". If the > kernel is Linux, it should be 2.6.22 or newer (IIRC). Thanks for the explanation. I rebuilt rsync using openSUSE 10.3 (and 11.0). Both use symtimes now. Before I configured on openSUSE 10.2 (= kernel 2.6.18) which was not enough as you said. Ciao Sven -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync many small files
Tue, 23 Sep 2008 06:49:33 -0400, magawake wrote: > At our lab we have storage with many small files. For example a > directory can contain over 15,000 files and each file averages about > 75k. I would like to sync this to another filesystem on a different > server but I am not sure if there is a rsync tuning flag I can use for > such a intensive job. I am using rsync 3.0.3. Also, I would like to > use little memory as possible. Synchronizing many files should be no problem with rsync versions >= 3 due to new incremental feature; I do this regularly with around 15 million mostly small files. But: The initial copy might take some days. If so, you might look around for a somewhat faster unconditional copy method. I prefer cpio here, but cloning tools might be better suited. Sven pgpsWP6em3k5A.pgp Description: PGP signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync many small files
Tue, 23 Sep 2008 13:25:10 +0200, paul+rsync wrote: > You may also want to look at csync2 http://oss.linbit.com/csync2/ > for keeping the copies synchronised, it's perhaps better suited than > rsync where you have such a large collection of files where there are > not that many updates going on (i.e. the searching for updates takes > significantly longer than transferring those updates). Thanks for the link! Reading the documentation I got the impression that for larger setups (with millions of files) one should use the inotify API for filling the hint database (see page 6 of the csync2 paper). Using this inotify idea would also allow more efficient syncing with an rsync script. Has anybody explored and documented this idea, with csync2 or rsync? Sven -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync 3
Thu, 25 Sep 2008 13:44:06 +0530, madhavanchari wrote: > We want to use the rsync 3 for incremental file transfer of our File system > from one box to another, however there are about a million files > to be copied, so just wanted to know by any one of your's previous > experience with rsync 3, will these many files get copied at first > time and incrementally in the subsequent run of the crontab. If you use one of the mailing list archives, you can get a wide impression on what people experienced. http://news.gmane.org/gmane.network.rsync.general/ Enter "million" in the "Search" field. My summary: it does work for million of files! :-) Sven -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync, what are the max transfer limitations ?
Tue, 18 Nov 2008 20:30:15 +0200, gourgi wrote: > rsync -avz rsync://192.168.0.X/disks/4backup /disks/mybackup What happens if you use only -a? pgpC9IcQtOSsl.pgp Description: PGP signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: file compression on target side
Mon, 19 Jan 2009 20:05:07 -0500, magawake wrote: > Using Redhat 4.5; I have been researching this for weeks and all signs > and wisemen (such as yourself) point to the Holy Grail -- ZFS! You could try FuseCompress: http://www.miio.net/fusecompress/ The author claims that he improved its speed recently. pgpfnQ5eFfaCr.pgp Description: PGP signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: find out the meaning of rsync error code
Tue, 20 Jan 2009 15:25:05 +0530, jignesh.shah1980 wrote: > Could anyone please tell me how to get the meaning of rsync error codes. > > In "rsync return code: 12" error, what 12 indicates? Please use the archive: http://www.mail-archive.com/rsync@lists.samba.org/msg22890.html pgpEalpvdwL2E.pgp Description: PGP signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync compression (-z) and timestamp
Tue, 27 Jan 2009 15:17:15 +0100, david.delama wrote: > Yes, I ran sync on both machines. The same results! :( sync is not enough. You will need to call a script dropcaches.sh that should contain (at least?): - # dropcaches.sh # deletes all file-io caches (pagecache, dentries and inodes) sync echo 3 > /proc/sys/vm/drop_caches - pgpU7j0fNs5PC.pgp Description: PGP signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync compression (-z) and timestamp
Tue, 27 Jan 2009 15:17:15 +0100, david.delama wrote: > I want to test it local, but how can I do it? > I don't see the amount of data rsync do compress. -v should do. Example: > rsync -z -v gcc-4.3.3.tar a gcc-4.3.3.tar sent 78,192,949 bytes received 31 bytes 3,191,550.20 bytes/sec total size is 405,166,080 speedup is 5.18 > ll gcc-4.3.3.tar a -rw-r--r-- 1 405166080 2009-01-27 16:48 a -rw-r--r-- 1 405166080 2009-01-26 11:02 gcc-4.3.3.tar You can even see the compression rate :-) Ciao Sven pgpbjN4G9zSev.pgp Description: PGP signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: how to migrate 40T data and 180M files
Hi. Tue, 11 Aug 2009 16:14:33 +0800, gaomingcn wrote: > The second question is about memory. > How much memory should I install to the linux box? The rsync FAQ > (http://rsync.samba.org/FAQ.html#4) says one file will use 100 bytes > to store relevant information, so 180M files will use about > 18G memory. That is true of rsync 2. rsync 3 uses an incremental mode that allows to transfer millions of files with 1 GB RAM or less. Sven -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: ssh failure due to local error
Tue, 18 Aug 2009 12:53:28 +0100, inpost wrote: > I repeat: I connect to the remote machine in a separate command terminal to > see > if the text file was transferred and there was no transfer. > > I repeated the command with the verbose command: > rsync -vt *.txt u...@remote.machine: > sent 39 bytes received 16 bytes 4.40 bytes/sec > total size is 24 speedup is 0.44 -v will also show the transferred files: so, your shell/rsync did not find any files that * match *.txt, * are in the directory were you called rsync, AND * need to be transferred Sven -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
