Re: rsyncing files that might change
At 23:09 31.10.2007 -0400, Matt McCutchen wrote: On Thu, 2007-11-01 at 10:35 +1000, Franc Carter wrote: If am rsyncing a file and I have the the following sequence of events happen in the same second 1. rsync starts 2. rsync sends some chunk of data to the other end 3 a local process modifies the chunk that has just been sent So, my question - is this case a race condition in which a subsequent run of rsync may miss that the file has been modified and hence skip it ? Unfortunately, yes. Shouldn't that be caught by the fact that the source file has a new (or at least different) time stamp now? bye Fabi -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsyncing files that might change
On 11/1/07, Fabian Cenedese [EMAIL PROTECTED] wrote: At 23:09 31.10.2007 -0400, Matt McCutchen wrote: On Thu, 2007-11-01 at 10:35 +1000, Franc Carter wrote: If am rsyncing a file and I have the the following sequence of events happen in the same second 1. rsync starts 2. rsync sends some chunk of data to the other end 3 a local process modifies the chunk that has just been sent So, my question - is this case a race condition in which a subsequent run of rsync may miss that the file has been modified and hence skip it ? Unfortunately, yes. Shouldn't that be caught by the fact that the source file has a new (or at least different) time stamp now? Sorry, I should have given a clearer example. All in one second 1. a process modifies the file and hence updates the source timesatmp 2. rsync starts 3. rsync sends the first chunk of data 4. a process modified the chunk that has been sent, but because it is in the same second there is no visible change to the timestamp bye Fabi -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Franc -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsyncing files that might change
On 11/1/07, Matt McCutchen [EMAIL PROTECTED] wrote: On Thu, 2007-11-01 at 10:35 +1000, Franc Carter wrote: If am rsyncing a file and I have the the following sequence of events happen in the same second 1. rsync starts 2. rsync sends some chunk of data to the other end 3 a local process modifies the chunk that has just been sent So, my question - is this case a race condition in which a subsequent run of rsync may miss that the file has been modified and hence skip it ? Unfortunately, yes. Ok, thanks. Matt -- Franc -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: multiple rsync sessions for one file?
On Wed 31 Oct 2007, Noam Birnbaum wrote: If one of these large files starts rsyncing at midnight, and the backup window ends before the file is complete (i.e. the rsync process stops), will it complete the rest of that file the following backup window? It will, if the --partial option is passed. The --partial-dir option may also be useful, depending on the requirements. If so, is there an easy way to configure rsync to stop execution at a specific time? No... Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync reports some files could not be transferred (code 23) when problems occur with password-file
Hi all, I think I just found a small bug in rsync... It has costed me about half an hour searching for a problem which actually didn't exist! If you use rsync like this: rsync --password-file=/tmp/non-existing-file /tmp/somefile [EMAIL PROTECTED]::tmp/ Rsync will ask for the password since the supplied file doesn't exist: rsync: could not open password file /tmp/non-existing-file: No such file or directory (2) Password: rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9] But actually, the file '/tmp/somefile' did get transferred to server 'bar'. When not supplying the bogus password file, rsync will again ask for the password, but not report the code 23. Bottom line: rsync reports a code 23 when there's a problem with the password file, even when the right password is supplied and files are transferred. Regards, -- Bas van Schaik -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Compression error?
Hi all, Sorry, I prefer bottom posting, please scroll down. Matt, Problem seems to have gone away. Unfortunately (for diagnosis problems anyway) the files in question are dynamic and updated nightly. Will keep an eye out to see if it reoccurs and try to capture. Regards /Steve Matt McCutchen wrote: On 7/19/07, WCSL [EMAIL PROTECTED] wrote: Error received when trying to use rsync -vz on the client inflate (token) returned -5 rsync error: error in rsync protocol data stream (code 12) at token.c(458) rsync: writefd_unbuffered failed to write 4092 bytes: phase unknown [generator]: Broken pipe (32) rsync error: error in rsync protocol data stream (code 12) at io.c(909) Seems to work fine without compression -z flag omitted. There have been occasional reports of -z getting corrupted in the past, but we've never gotten a reproducible case to investigate and find the underlying problem. If you run the command repeatedly, does that same error occur every time? Even better if it still happens with the same files on a different pair of machines. If so, please send the source and destination files prior to the run to Wayne Davison [EMAIL PROTECTED], the rsync maintainer, unless they're big and/or private in which case you should discuss other arrangements with him. Matt I actually have the same problem with transferring large amounts of files with the -z option, but the inflate fails on the same files every time. Example log snippet (receiving host, which I will call server): 2007/11/01 04:59:23 [17567] recv infinity [192.168.0.221] backups (infinity) tetra-servers/latest/xen_aphrodite/var/lib/mysql/otrs-valintra/article.MYD 11272960 2007/11/01 04:59:23 [17567] recv infinity [192.168.0.221] backups (infinity) tetra-servers/latest/xen_aphrodite/var/lib/mysql/otrs-valintra/article.MYI 2468864 2007/11/01 05:01:09 [17567] inflate (token) returned -5 2007/11/01 05:01:09 [17567] rsync error: error in rsync protocol data stream (code 12) at token.c(476) [receiver=2.6.9] 2007/11/01 05:01:09 [17567] rsync: connection unexpectedly closed (45884845 bytes received so far) [generator] 2007/11/01 05:01:09 [17567] rsync error: error in rsync protocol data stream (code 12) at io.c(453) [generator=2.6.9] Note that I think that the transfer of tetra-servers/latest/xen_aphrodite/var/lib/mysql/otrs-valintra/article.MYI did succeed, the inflate error occurs on the next file which isn't logged. I presumed that this next file is article_attachment.MYD which actually is quite large: about 1.4 gigabytes. This file exists on both client and server, so it should only be partially transferred. A few weeks ago I tried to remove the file on the server to force the client to transfer it entirely, that actually worked and fixed the problem. But now the problem is back and I felt obliged to report it to the list since there seem to be more people having the same problem. Assuming that it was indeed article_attachment.MYD which caused the inflate error, I decided to investigate that file. Client side, this is the output of ls -la article_attachment.MYD: -rw-rw 1 rsync nogroup 1125532216 2007-10-31 22:50 article_attachment.MYD And on the server: -rw-r- 1 rsync nogroup 1077162396 2007-11-01 13:14 article_attachment.MYD Of course I tried to transfer that file on its own in verbose-verbose-verbose mode and this is the result: $ rsync -atzvvv --compress-level=9 article_attachment.MYD --timeout=3600 [EMAIL PROTECTED]::tmp/rsync/ opening tcp connection to dust port 873 opening connection using --server -vvvlogDtprz --compress-level=9 --timeout=3600 . tmp/rsync/ Password: building file list ... [sender] make_file(article_attachment.MYD,*,2) done send_file_list done send_files starting send_files(0, article_attachment.MYD) send_files mapped article_attachment.MYD of size 1125532216 calling match_sums article_attachment.MYD article_attachment.MYD built hash table hash search b=32816 len=1125532216 match at 0 last_match=0 j=0 len=32816 n=0 (...) match at 627409104 last_match=627409104 j=19119 len=32816 n=0 (...) match at 1077119568 last_match=1077119568 j=32823 len=32816 n=0 rsync: writefd_unbuffered failed to write 16385 bytes [sender]: Connection reset by peer (104) inflate (token) returned -5 rsync: connection unexpectedly closed (229801 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(453) [sender=2.6.9] _exit_cleanup(code=12, file=io.c, line=453): about to call exit(12) The output was huge, my console got flooded by match at... lines, so I decided only to copy and paste three of them here. The values for 'len' and 'n' are constant, the value for 'j' increases by one every step and the other two values gradually increase also with greater steps. Relevant log snippet of the server side log: 2007/11/01 13:15:23 [10684] connect from infinity (192.168.0.221) 2007/11/01 13:15:28 [10684] rsync to tmp/rsync/
Re: Re: Script work fine on sheel, but no on a cron job
On 10/31/07, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: The script have line like rsync -avz --password-file=/usr/local/etc/rsync.pwd /etc/rsyncd.conf rsync://[EMAIL PROTECTED]:/etc_conf Your shell might have the RSYNC_PASSWORD environment variable preset, while the script does not. Is the password file chmoded 0640 and owned by root? Does the remote daemon log something like Access denied for [module_name] for [IP_address] ? -A -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsyncing files that might change
Franc Carter wrote: Unfortunately, yes. Shouldn't that be caught by the fact that the source file has a new (or at least different) time stamp now? Sorry, I should have given a clearer example. All in one second 1. a process modifies the file and hence updates the source timesatmp 2. rsync starts 3. rsync sends the first chunk of data 4. a process modified the chunk that has been sent, but because it is in the same second there is no visible change to the timestamp bye Fabi If this is a real problem, you might try a filesystem and OS version which supports microsecond or nanosecond timestamps. There's still no guarantee that changes are detected, but it's more likely. Oh, and I'm not sure if rsync reads the sub-second timestamps yet :-) -- Jamie -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: osx-create-time.diff
On Wed, Oct 31, 2007 at 09:36:39PM -0400, Robert DuToit wrote: patch -p1 patches/osx-create-time-1.diff patch -p1 patches/osx-create-time-2-.diff I'm not sure where you saw those filenames. There's only one file, which has been updated 2 times in CVS since its initial checkin. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: multiple rsync sessions for one file?
On Thu, 2007-11-01 at 11:30 +0100, Paul Slootman wrote: If so, is there an easy way to configure rsync to stop execution at a specific time? No... Well, there is the patch patches/time-limit.diff in the rsync source tree that adds options to make rsync stop at a certain time or after a certain time interval. You could compile and use your own copy of rsync with this patch. Or you could just have your script send a signal to the rsync process at the appropriate time, which will cause rsync to clean up and quit. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: multiple rsync sessions for one file?
Thanks folks! Matt, what kind of signal would my script have to send to make rsync clean up and quit? Best, noam Noam Birnbaum http://maccentricsolutions.com/ 877.luv.macs x89 Apple Certified Technical Coordinator Apple Certified Help Desk Specialist On Nov 1, 2007, at 12:30 PM, Matt McCutchen wrote: On Thu, 2007-11-01 at 11:30 +0100, Paul Slootman wrote: If so, is there an easy way to configure rsync to stop execution at a specific time? No... Well, there is the patch patches/time-limit.diff in the rsync source tree that adds options to make rsync stop at a certain time or after a certain time interval. You could compile and use your own copy of rsync with this patch. Or you could just have your script send a signal to the rsync process at the appropriate time, which will cause rsync to clean up and quit. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: multiple rsync sessions for one file?
On Thu, 2007-11-01 at 13:42 -0700, Noam Birnbaum wrote: Matt, what kind of signal would my script have to send to make rsync clean up and quit? SIGINT or SIGTERM should work. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
build/test problems with rsync-3.0.0pre4
Hi, I tried to build rsync-3.0.0pre4 on a list of different plattforms and got some problems with build or test suite. first, the good news: the build and tests worked fine on these plattforms Solaris 5.5.1, 5.6 and 5.8 (sparc) and 5.10 (x86 and sparc) SUSE Linux 9.0 (amd64), 10.1 (amd6) SLES10 (32 and 64 bit) RedHat 7.1 and 9.0 (32 bit) RedHat EL3 (ia64) problems showed up on HP-UX 11.00 (compile), 11.11 (parisc) and 11.23 (ia64) AIX 5.1 and 5.3 IRIX 6.5 MacOSX 10.4 (Darwin, ppc and x86) here is a small summary of the problems. feel free to ask for detailed logs etc. right now I'd be interested in patches/fixes esp. for the AIX problems --- HP-UX 11.00 compile problem: cc -Ae +DAportable -D_HPUX_SOURCE -I/usr/local/include -I. -I/soft/os/rsync/rsync-3.0.0pre4/rsync-3.0.0pre4 -g -DHAVE_CONFIG_H -I/soft/os/rsync/rsync-3.0.0pre4/rsync-3.0.0pre4/popt -D_LARGE_FILES -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/local/include -c /soft/os/rsync/rsync-3.0.0pre4/rsync-3.0.0pre4/acls.c -o acls.o cpp: /soft/os/rsync/rsync-3.0.0pre4/rsync-3.0.0pre4/lib/sysacls.h, line 142: error 4036: Can't open include file 'sys/aclv.h'. these acl header files are available: hpux1100 find /usr/include/ -name \*acl\* -print /usr/include/sys/acl.h /usr/include/acl.h /usr/include/acllib.h vs. HP-UX 11.11: hpux find /usr/include/ -name \*acl\* -print /usr/include/sys/acl.h /usr/include/sys/aclv.h /usr/include/acl.h /usr/include/acllib.h /usr/include/aclv.h --- HP-UX 11.11/11.23 test failures: - chown log follows Testing for symlinks using 'test -h' Running: /scr/viz-hpux/koenig/rsync-3.0.0pre4-1/ARENA/32/rsync --super -aHvv from/ to/ sending incremental file list created directory to delta-transmission disabled for local transfer or --whole-file ../ name1 rsync: failed to set permissions on /net/viz-hpux/fs1/scr/viz-hpux/koenig/rsync-3.0.0pre4-1/ARENA/32/testtmp/chown/to/.name1.a14598: Not owner (1) name2 rsync: failed to set permissions on /net/viz-hpux/fs1/scr/viz-hpux/koenig/rsync-3.0.0pre4-1/ARENA/32/testtmp/chown/to/.name2.a14598: Not owner (1) total: matches=0 hash_hits=0 false_alarms=0 data=40 sent 206 bytes received 54 bytes 520.00 bytes/sec total size is 40 speedup is 0.15 rsync error: some files could not be transferred (code 23) at /soft/os/rsync/rsync-3.0.0pre4/rsync-3.0.0pre4/main.c(1053) [sender=3.0.0pre4] - chown log ends FAILchown - dir-sgid log follows Testing for symlinks using 'test -h' sending incremental file list created directory /scr/viz-hpux/koenig/rsync-3.0.0pre4-1/ARENA/32/testtmp/dir-sgid/setgid-off/to delta-transmission disabled for local transfer or --whole-file file program dir/ dir/blah/ total: matches=0 hash_hits=0 false_alarms=0 data=16 sent 192 bytes received 58 bytes 500.00 bytes/sec total size is 16 speedup is 0.06 permissions: rwxrw-r-- on /scr/viz-hpux/koenig/rsync-3.0.0pre4-1/ARENA/32/testtmp/dir-sgid/setgid-off/to/dir should be: rwx-- failed test Target setgid-off - dir-sgid log ends FAILdir-sgid - overall results: 28 passed 2 failed 6 skipped --- AIX 5.1 and 5.3 test failures: - hands log follows Testing for symlinks using 'test -h' -n Test basic operation: Running: /scr/needa/koenig/rsync-3.0.0pre4-1/ARENA/32/rsync -av /scr/needa/koenig/rsync-3.0.0pre4-1/ARENA/32/testtmp/hands/from/ /scr/needa/koenig/rsync-3.0.0pre4-1/ARENA/32/testtmp/hands/to sending incremental file list ../ empty filelist nolf nolf-symlink - nolf text dir/ dir/text dir/subdir/ dir/subdir/foobar.baz dir/subdir/subsubdir/ dir/subdir/subsubdir/etc-ltr-list dir/subdir/subsubdir2/ dir/subdir/subsubdir2/bin-lt-list emptydir/ sent 1492996 bytes received 191 bytes 995458.00 bytes/sec total size is 1492170 speedup is 1.00 - check how the directory listings compare with diff: No differences encountered - check how the files compare with diff: - done. -n Test hard links: Running: /scr/needa/koenig/rsync-3.0.0pre4-1/ARENA/32/rsync -avH /scr/needa/koenig/rsync-3.0.0pre4-1/ARENA/32/testtmp/hands/from/ /scr/needa/koenig/rsync-3.0.0pre4-1/ARENA/32/testtmp/hands/to sending incremental file list rsync: connection unexpectedly closed (9 bytes received so far) [sender] rsync error: error allocating core memory buffers (code 22) at /soft/os/rsync/rsync-3.0.0pre4/rsync-3.0.0pre4/io.c(603) [sender=3.0.0pre4] - check how the directory listings compare with diff: *** /scr/needa/koenig/rsync-3.0.0pre4-1/ARENA/32/testtmp/hands/ls-from Wed Oct 31 11:19:29
