[Bug 13320] file contents cause rsync to fail (with certains args and dir structure)
https://bugzilla.samba.org/show_bug.cgi?id=13320 Wayne Davison changed: What|Removed |Added Resolution|--- |FIXED Status|NEW |RESOLVED --- Comment #5 from Wayne Davison --- This was fixed in commit 888ce058. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 13320] file contents cause rsync to fail (with certains args and dir structure)
https://bugzilla.samba.org/show_bug.cgi?id=13320 --- Comment #4 from Marcus Linsner --- Thanks for this fix Dave, worksforme! -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 13320] file contents cause rsync to fail (with certains args and dir structure)
https://bugzilla.samba.org/show_bug.cgi?id=13320 --- Comment #3 from Dave Gordon--- Problem introduced by commit f3873b3d88b61167b106e7b9227a20147f8f6197 Author: Wayne Davison Date: Mon Oct 10 11:49:50 2016 -0700 Support --sparse combined with --preallocate or --inplace. The new code tries to punch holes in the destination file using newer Linux fallocate features. It also supports a --whole-file + --sparse + --inplace copy on any filesystem by truncating the destination file. Root cause is that the new variable 'sparse_past_write' is not reset when switching files, thus leading to miscalculation of the current seek position in do_punch_hole(). Quick fix is to reset this variable to the current seek position on each call to write_sparse() - probably not the most efficient fix, but it seems to work. .Dave. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 13320] file contents cause rsync to fail (with certains args and dir structure)
https://bugzilla.samba.org/show_bug.cgi?id=13320 --- Comment #2 from Dave Gordon--- Created attachment 14018 --> https://bugzilla.samba.org/attachment.cgi?id=14018=edit Simple although probably suboptimal fix -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 13320] file contents cause rsync to fail (with certains args and dir structure)
https://bugzilla.samba.org/show_bug.cgi?id=13320 --- Comment #1 from Dave Gordon--- Bug will be triggered if: options include both --sparse and --preallocate, AND the second (or subsequent) file to be copied begins with one or more zero bytes. .Dave. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: file contents cause rsync to fail (with certains args and dir structure)
Problem was introduced with this commit: commit f3873b3d88b61167b106e7b9227a20147f8f6197 Author: Wayne Davison way...@samba.org Date: Mon Oct 10 11:49:50 2016 -0700 Support --sparse combined with --preallocate or --inplace. The new code tries to punch holes in the destination file using newer Linux fallocate features. It also supports a --whole-file + --sparse + --inplace copy on any filesystem by truncating the destination file. .Dave. Sent using Zoho Mail On Sun, 04 Mar 2018 23:17:23 +0100 Dave Gordon via rsync rsync@lists.samba.org wrote Quite strange at first sight that the failure should depend on the files containing NULs! But I've reproduced it on both Ubuntu and OpenSUSE with d73762e "Preparing for release of 3.1.3". The problem remains even if you drop the --checksum or --delay-updates options from the command line, but goes away if you don't use both "--sparse" and "--preallocate" together. So it looks like a bad interaction of these options, probably with the sparse handling being affected by the NULs in the files. HTH, .Dave. Sent using Zoho Mail On Sun, 04 Mar 2018 14:57:24 +0100 xftroxgpx via rsync rsync@lists.samba.org wrote -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html script to reproduce: #!/bin/bash #tested to fail as below: ArchLinux's rsync-3.1.3-1-x86_64.pkg.tar.xz #tested to fail as below: ArchLinux's rsync-3.1.3pre1-1-x86_64.pkg.tar.xz #tested to work ok : ArchLinux's rsync-3.1.2-8-x86_64.pkg.tar.xz if test "$1" == "clean"; then rm -vrf destdir sourcedir sourcedir2 sourcedir3 exit 0 fi echo '!! test 1:' mkdir -p destdir mkdir -p sourcedir/a #one \0 followed by a non-\0 (so, using a space) required: echo -ne '\0 ' sourcedir/a/b #non zero size file required: echo -ne 'c' sourcedir/c echo 'sourcedir' /tmp/filesfrom.lst.tmp rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate --sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/ #rsync: write failed on "/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir/a/b": No such file or directory (2) #rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3] # ^ this happens first and any subsequent times! echo '!! test 2:' mkdir -p sourcedir2/a #one \0 followed by a non-\0 (so, using an M) required: echo -ne '\0M' sourcedir2/a/b #one \0 followed by a non-\0 (so, using an M) required: echo -ne '\0M' sourcedir2/c #in order to see this error: (for file 'c') #non-zero file size required and it must be prefixed by '.' aka dot #echo -ne '1' sourcedir2/.d #XXX: ^ (un)comment, don't forget to ./go clean afterwards, then ./go #otherwise, the error is for file "a/b" echo 'sourcedir2' /tmp/filesfrom.lst.tmp rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate --sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/ #rsync: write failed on "/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir2/a/b": No such file or directory (2) #rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3pre1] echo '!! test 3:' #same as 2 but an extra file '.d' exists! mkdir -p sourcedir3/a #one \0 followed by a non-\0 (so, using an M) required: echo -ne '\0M' sourcedir3/a/b #one \0 followed by a non-\0 (so, using an M) required: echo -ne '\0M' sourcedir3/c #non-zero file size required and it must be prefixed by '.' aka dot echo -ne '1' sourcedir3/.d echo 'sourcedir3' /tmp/filesfrom.lst.tmp rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate --sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/ #rsync: write failed on "/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir2/c": No such file or directory (2) #rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3] A pristine copy of this script can be found here: https://github.com/xftroxgpx/a3/blob/37ebff3e0fe9d294aeec899a082dc2c51c486eb4/system/Z575/OSes/3archlinux/on_baremetal/filesystem_now/archlinux/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/go Just run ./go to start all 3 tests. All 3 tests should fail with: ArchLinux's rsync-3.1.3-1-x86_64.pkg.tar.xz ArchLinux's rsync-3.1.3pre1-1-x86_64.pkg.tar.xz All 3 tests will succeed with ArchLinux's rsync-3.1.2-8-x86_64.pkg.tar.xz (aka rsync version 3.1.2 protocol version 31) Cheers! Sent with ProtonMail Secure Email. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read:
Re: file contents cause rsync to fail (with certains args and dir structure)
Quite strange at first sight that the failure should depend on the files containing NULs! But I've reproduced it on both Ubuntu and OpenSUSE with d73762e "Preparing for release of 3.1.3". The problem remains even if you drop the --checksum or --delay-updates options from the command line, but goes away if you don't use both "--sparse" and "--preallocate" together. So it looks like a bad interaction of these options, probably with the sparse handling being affected by the NULs in the files. HTH, .Dave. Sent using Zoho Mail On Sun, 04 Mar 2018 14:57:24 +0100 xftroxgpx via rsync rsync@lists.samba.org wrote script to reproduce: #!/bin/bash #tested to fail as below: ArchLinux's rsync-3.1.3-1-x86_64.pkg.tar.xz #tested to fail as below: ArchLinux's rsync-3.1.3pre1-1-x86_64.pkg.tar.xz #tested to work ok : ArchLinux's rsync-3.1.2-8-x86_64.pkg.tar.xz if test "$1" == "clean"; then rm -vrf destdir sourcedir sourcedir2 sourcedir3 exit 0 fi echo '!! test 1:' mkdir -p destdir mkdir -p sourcedir/a #one \0 followed by a non-\0 (so, using a space) required: echo -ne '\0 ' sourcedir/a/b #non zero size file required: echo -ne 'c' sourcedir/c echo 'sourcedir' /tmp/filesfrom.lst.tmp rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate --sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/ #rsync: write failed on "/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir/a/b": No such file or directory (2) #rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3] # ^ this happens first and any subsequent times! echo '!! test 2:' mkdir -p sourcedir2/a #one \0 followed by a non-\0 (so, using an M) required: echo -ne '\0M' sourcedir2/a/b #one \0 followed by a non-\0 (so, using an M) required: echo -ne '\0M' sourcedir2/c #in order to see this error: (for file 'c') #non-zero file size required and it must be prefixed by '.' aka dot #echo -ne '1' sourcedir2/.d #XXX: ^ (un)comment, don't forget to ./go clean afterwards, then ./go #otherwise, the error is for file "a/b" echo 'sourcedir2' /tmp/filesfrom.lst.tmp rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate --sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/ #rsync: write failed on "/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir2/a/b": No such file or directory (2) #rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3pre1] echo '!! test 3:' #same as 2 but an extra file '.d' exists! mkdir -p sourcedir3/a #one \0 followed by a non-\0 (so, using an M) required: echo -ne '\0M' sourcedir3/a/b #one \0 followed by a non-\0 (so, using an M) required: echo -ne '\0M' sourcedir3/c #non-zero file size required and it must be prefixed by '.' aka dot echo -ne '1' sourcedir3/.d echo 'sourcedir3' /tmp/filesfrom.lst.tmp rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate --sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/ #rsync: write failed on "/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir2/c": No such file or directory (2) #rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3] A pristine copy of this script can be found here: https://github.com/xftroxgpx/a3/blob/37ebff3e0fe9d294aeec899a082dc2c51c486eb4/system/Z575/OSes/3archlinux/on_baremetal/filesystem_now/archlinux/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/go Just run ./go to start all 3 tests. All 3 tests should fail with: ArchLinux's rsync-3.1.3-1-x86_64.pkg.tar.xz ArchLinux's rsync-3.1.3pre1-1-x86_64.pkg.tar.xz All 3 tests will succeed with ArchLinux's rsync-3.1.2-8-x86_64.pkg.tar.xz (aka rsync version 3.1.2 protocol version 31) Cheers! Sent with ProtonMail Secure Email. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html