DO NOT REPLY [Bug 5075] Syncing with --iconv may yield protocol error
https://bugzilla.samba.org/show_bug.cgi?id=5075
--- Comment #2 from [EMAIL PROTECTED] 2007-11-12 00:17 CST ---
Sorry, I meant to add this too:
There is also an uninitialized variable in readfd_buffered that can cause
garbage to be read independently of the bug in mplex_write. Under case
MSG_DELETED, inbuf.pos is never initialized which can -- and will -- cause the
call to iconvbufs further down to process random data.
The fix is to initialize inbuf with a proper call to INIT_XBUF before
proceeding to do the conversion.
Diff below and attached.
--- 510,515
***
*** 1060,1066
int pos = 0;
INIT_CONST_XBUF(outbuf, line);
! inbuf.buf = ibuf;
while (msg_bytes) {
inbuf.len = msg_bytes > sizeof ibuf
--- 1069,1075
int pos = 0;
INIT_CONST_XBUF(outbuf, line);
! INIT_XBUF(inbuf, ibuf, 0, -1);
while (msg_bytes) {
inbuf.len = msg_bytes > sizeof ibuf
--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the QA contact for the bug, or are watching the QA contact.
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 5075] Syncing with --iconv may yield protocol error
https://bugzilla.samba.org/show_bug.cgi?id=5075 --- Comment #1 from [EMAIL PROTECTED] 2007-11-12 00:14 CST --- Created an attachment (id=2963) --> (https://bugzilla.samba.org/attachment.cgi?id=2963&action=view) Diffs to fix bug -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 5075] New: Syncing with --iconv may yield protocol error
https://bugzilla.samba.org/show_bug.cgi?id=5075
Summary: Syncing with --iconv may yield protocol error
Product: rsync
Version: 3.0.0
Platform: All
OS/Version: All
Status: NEW
Severity: major
Priority: P3
Component: core
AssignedTo: [EMAIL PROTECTED]
ReportedBy: [EMAIL PROTECTED]
QAContact: [EMAIL PROTECTED]
If you use --iconv to sync between two machines and a file with a name that
needs conversion is to be deleted, the rsync protocol may get out of sync and
yield a protocol error if the converted filename differs in length form the
unconverted name. This is because mplex_write in io.c will record and send the
message length *before* the message has been converted using iconvbuf. If the
converted message differs in length, the reading rsync process will get
confused and interpret part of the message data as tag & length.
Here is an example:
[21:45:27]lenux:~/Mirror$ rsync -a --progress --del -n --iconv=utf-8,utf-8-mac
Lennart/Auto/ sva:Mirror/Lennart/Auto/
sending incremental file list
Audi Cabrio 92 (CUB931)/D\#303\#244ck/
deleting Audi Cabrio 92 (CUB931)/D\#303\#244ck/www.stro.nu.url.URL\#001
unexpected tag -7 [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(1135)
[sender=3.0.0pre5]
(Note the garbage #\001 at the end of the filename.)
To fix this, mplex_write needs to convert the message data before sending its
length. Here is a possible rewrite:
/* Write an message to a multiplexed stream. If this fails, rsync exits. */
static void mplex_write(int fd, enum msgcode code, const char *buf, size_t len,
int convert)
{
char buffer[BIGPATHBUFLEN]; /* Oversized for use by iconv code. */
size_t n = len;
#ifdef ICONV_OPTION
if (convert && ic_send == (iconv_t)-1)
#endif
convert = 0;
#ifdef ICONV_OPTION
/* We need to convert buf before doing anything else so that we
* can include the (converted) byte length in the message header.
*/
if (convert) {
xbuf outbuf, inbuf;
INIT_XBUF(outbuf, buffer + 4, 0, sizeof(buffer) - 4);
INIT_XBUF(inbuf, (char*)buf, len, -1);
iconvbufs(ic_send, &inbuf, &outbuf,
ICB_INCLUDE_BAD | ICB_INCLUDE_INCOMPLETE);
if (inbuf.len > 0) {
rprintf(FERROR, "conversion buffer overflow in
mplex_write; "
"either enlarge 'buffer' or rewrite
code to allocate "
"buffer dynamically");
}
n = len = outbuf.len;
} else
#endif
if (n > sizeof(buffer) - 4)
n = 0;
else
memcpy(buffer + 4, buf, n);
SIVAL(buffer, 0, ((MPLEX_BASE + (int)code)<<24) + len);
defer_forwarding_keep = 1; /* defer_forwarding_messages++ on return */
writefd_unbuffered(fd, buffer, n+4);
defer_forwarding_keep = 0;
len -= n;
buf += n;
if (len)
writefd_unbuffered(fd, buf, len);
if (!--defer_forwarding_messages && !no_flush)
msg_flush();
}
Context diff based on 3.0.0pre5 follows:
*** io.c~ Sat Nov 3 00:20:05 2007
--- io.cSun Nov 11 21:59:57 2007
***
*** 468,485
char buffer[BIGPATHBUFLEN]; /* Oversized for use by iconv code. */
size_t n = len;
- SIVAL(buffer, 0, ((MPLEX_BASE + (int)code)<<24) + len);
-
#ifdef ICONV_OPTION
if (convert && ic_send == (iconv_t)-1)
#endif
convert = 0;
! if (convert || n > 1024 - 4) /* BIGPATHBUFLEN can handle 1024 bytes */
n = 0;
else
memcpy(buffer + 4, buf, n);
defer_forwarding_keep = 1; /* defer_forwarding_messages++ on return */
writefd_unbuffered(fd, buffer, n+4);
defer_forwarding_keep = 0;
--- 468,508
char buffer[BIGPATHBUFLEN]; /* Oversized for use by iconv code. */
size_t n = len;
#ifdef ICONV_OPTION
if (convert && ic_send == (iconv_t)-1)
#endif
convert = 0;
! #ifdef ICONV_OPTION
! /* We need to convert buf before doing anything else so that we
!* can include the (converted) byte length in the message header.
!*/
! if (convert) {
! xbuf outbuf, inbuf;
!
! INIT_XBUF(outbuf, buffer + 4, 0, sizeof(buffer) - 4);
! INIT_XBUF(inbuf, (char*)buf, len, -1);
!
! iconvbufs(ic_send, &inbuf, &outbuf,
! ICB_INCLUDE_BAD | ICB_INCLUDE_INCOMPLETE);
!
! if (inbuf.len > 0) {
! rprintf(FERROR, "conversion buffer overflow in
mplex_write; "
! "either enlarge 'buffer' or rewrite
code to allocate "
!
Re: xattrs: Permission denied?
On Nov 12, 2007, at 12:55 AM, Wayne Davison wrote: On Sun, Nov 11, 2007 at 03:09:08PM +0100, Wesley W. Terpstra wrote: There seem to be three distinct problems. Thanks for the detailed analysis. I'll be checking into this soon. I wanted to fix it, but every time I prepare a patch, it breaks somewhere else. I had the test-case I posted working, but then it would fail when doing fake-super (b/c the %stat gets set in two places then). How is this whole thing supposed to work? It seems to be doing some strange mix of pre- and post-order recursion. Wouldn't it be simpler to just to strict post-order [1]? ie: set/mkdir the directory 07xx, rsync all the contents, and then set_file_attrs at the end? Then there's no worry about touching up the mtime/permissions/etc. Also, the way it is now the file attributes get touched repeatedly, slowing it down by a non-negligible amount. [1] http://en.wikipedia.org/wiki/Tree_traversal -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: xattrs: Permission denied?
On Sun, Nov 11, 2007 at 03:09:08PM +0100, Wesley W. Terpstra wrote: > There seem to be three distinct problems. Thanks for the detailed analysis. I'll be checking into this soon. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 5071] Enhancement Request - Customizable Connection Timeout Option
https://bugzilla.samba.org/show_bug.cgi?id=5071 [EMAIL PROTECTED] changed: What|Removed |Added Status|NEW |ASSIGNED --- Comment #3 from [EMAIL PROTECTED] 2007-11-11 17:01 CST --- Yes, I snagged that patch but apparently neglected to check it in. There was a little tweaking I wanted to do to it before accepting it, which I worked on a bit this morning: http://rsync.samba.org/ftp/unpacked/rsync/patches/contimeout.diff Since it's so simple, I'm thinking of including it in the 3.0.0 release. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 5074] New: Push to a daemon with --only-write-batch modifies dest and then hangs
https://bugzilla.samba.org/show_bug.cgi?id=5074 Summary: Push to a daemon with --only-write-batch modifies dest and then hangs Product: rsync Version: 3.0.0 Platform: x86 OS/Version: Linux Status: NEW Severity: major Priority: P3 Component: core AssignedTo: [EMAIL PROTECTED] ReportedBy: [EMAIL PROTECTED] QAContact: [EMAIL PROTECTED] Using the latest development rsync, if I push files to a daemon with --only-write-batch, the daemon makes some of the changes to the destination and then hangs. The run should finish with no changes made to the destination. To reproduce, run the following in an empty directory: export PATH=/path/to/rsync/build/tree:$PATH wget http://mattmccutchen.net/rsync/instant-daemon.sh sh instant-daemon.sh module 3141 user (Choose a password and enter it twice) mkdir upload ln -s . upload/me echo foo >upload/file rsync -rl --only-write-batch=thebatch upload/ rsync://[EMAIL PROTECTED]:3141/module/ (Enter the password) (When the process hangs, press ^C) ls -la module (Note that the symlink was copied.) ./stop -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: dry-run bytest to be transferred?
On Sun, 2007-11-11 at 12:47 -0800, Max Kipness wrote: > Is there anyway on a dry-run to actually see per file how many bytes are to > be transferred? On a normal run I use --log-format=/%f/%l/%b/%i which shows > the total file size and the actual bytes transferred. > > Also, during a dry-run, is there anyway to show the Literal Data? > > Basically, I'm trying to calculate something before the actual run takes > place. You can't get this information on a dry run because a dry run skips file transfers altogether to save time. Instead, you could do a push with --only-write-batch ; this will give you accurate statistics without sending the file transfer data over the network or modifying the destination. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
dry-run bytest to be transferred?
Is there anyway on a dry-run to actually see per file how many bytes are to be transferred? On a normal run I use --log-format=/%f/%l/%b/%i which shows the total file size and the actual bytes transferred. Also, during a dry-run, is there anyway to show the Literal Data? Basically, I'm trying to calculate something before the actual run takes place. Dry-run is telling me which files have changed and why, but not telling me how much of the file is different. Thanks, Max -- View this message in context: http://www.nabble.com/dry-run-bytest-to-be-transferred--tf4787455.html#a13695755 Sent from the Samba - rsync mailing list archive at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Strange rsync error
Hello, Sometimes when I am using the command: rsync -avR -e rsh "/raid/system/Images/./GHOST.BAT" 192.168.192.2:/raid/system/Images/ --delete -z --progress --exclude-from=/usr/local/excludes --bwlimit=70 I get the following error: rsync: mkdir "/raid/system/Images" failed: File exists (17) rsync error: error in file IO (code 11) at main.c(529) [receiver=2.6.9] rsync: connection unexpectedly closed (8 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(453) [sender=2.6.9] Other times it just works fine. Any ideas what the above error means and why that happened? Version: rsync version 2.6.9 protocol version 29 Thanks for your time. -- Charalampos Pournaris [EMAIL PROTECTED] -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: xattrs: Permission denied?
I've been thus far unable to figure out why this breaks. There seem to be three distinct problems. Problem #1: receiver.c:451 calls set_file_attrs before the contents of a directory have been created. This resets the writable flag which was setup in generator.c to permit creating children. This causes files that are children of a read-only directory with extended attributes to fail. Problem #2: generator.c creates directories with mode file->mode. If the directory has no contents, the flags are never made writable. This causes the call in receiver.c to fail to create the extended attributes. Problem #3: generator.c also calls set_file_attrs early (before xattrs are available) which will reset the permissions on directories even if #2 is fixed. This is also the point at which the %stat xattr gets set. I think the solution to #2 is to add 0700 to non-root created directories and tweak the directory mode always. This will make setting the %stat attr in the early set_file_attrs succeed. #3 must then be taught to leave this mode tweak alone. #1 should perhaps only call the set_xattr code, not the whole set_file_attrs. After all, the earlier set_file_attrs already dealt with permissions. There also seems to be some race-condition between the generator and receiver. I noticed that if I placed sleep commands in the generator other things start to fail. I didn't investigate this further. I can't fix these problems on my own because I barely understand the correct order of the operations. Here is the test case that I'm using: mkdir y cd y touch file1 mkdir no-xattr mkdir no-xattr-stuff mkdir xattr mkdir xattr-stuff touch no-xattr-stuff/file2 touch xattr-stuff/file3 setfattr -n user.test1 -v foo . setfattr -n user.test2 -v bar xattr setfattr -n user.test3 -v baz xattr-stuff chmod -w * . cd .. rsync -aX y z If you can get this to work. Hats off to you! -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
