RFE to allow dry-run against read only target
I would like to request that rsync -n to an rsync server target that is read only be supported rather than generating the following error, ERROR: module is read only The motivation is to allow the generation of a list of files against an rsync target that will be used by another application. Since dry-run mode does not require write access it would be convenient if the “module is read only” error message was only thrown if/when a write attempt is made and not assume write access is needed. Thanks. -- Stuart Anderson ander...@ligo.caltech.edu http://www.ligo.caltech.edu/~anderson -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to exclude directories from source with --relative
On Fri, May 30, 2008 at 11:14:22AM -0700, Wayne Davison wrote: On Tue, May 27, 2008 at 10:43:00AM -0700, Stuart Anderson wrote: Nuts. I was afraid that was going to be the answer. Unfortunately, it violates the very helpful advise given above, i.e., run with -v and add exclude rules for files you don't want to transfer based on the output seen. That rule is not violated (the matching is correct), it is just that you can't try to exclude something in the middle of a relative path that you've specified. You can work around the issue by moving higher in the directory structure and using includes/excludes. For example, instead of: Wayne, Many thanks for you help on this. I now have something working by giving up on the --relative option and adding a longer list of alternating include/exclude rules. However, I would like to request that you consider increasing the scope of the filtering rules to include relative paths. I would like to be able to use shell globing to generate a long list of source directories but still allow rsync to filter out some of those. Put another way, I would like be able to specify --exclude abc/ and have the directory abc not be transferred under any circumstances unless it was first listed in an --include argument. P.S. The recent scalability and incremental scan improvements in 3.0 are working great for me, and are greatly appreciated. Thanks. -- Stuart Anderson [EMAIL PROTECTED] http://www.ligo.caltech.edu/~anderson -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to exclude directories from source with --relative
On Tue, May 27, 2008 at 03:18:03PM -0400, Matt McCutchen wrote: On Mon, 2008-05-26 at 23:40 -0700, Wayne Davison wrote: On Mon, May 26, 2008 at 09:46:04PM -0700, Stuart Anderson wrote: 3) Try excluding a/, this does not work: # rsync -avn --relative --exclude a/ /tmp/source/./a/? /tmp/destination That's because 'a' is an implied directory in the longer path that you told rsync to transfer. Implied directories can't be excluded, but their attributes can be omitted via --no-implied-dirs. If you don't want it in the transfer, don't specify it (and if you're using wild- cards, specify them in such as way as to avoid matching the dir you don't want). I can understand not being able to exclude a/ from the file list while still transferring stuff under it. Still, I should be able to exclude a/ and everything under it by passing --exclude='/a/***' . But this doesn't work in rsync 3.0.2: rsync excludes a/b but adds the implied dir a/ to the file list anyway. (Obviously, it worked in rsync 2.6.9, which allowed the user to exclude implied dirs.) I think rsync should check whether a source argument is excluded before adding any dirs it implies. This would be sufficient for my needs, and I didn't realize it is a change in behavior from 2.6.9 to 3.0.2. Thanks. -- Stuart Anderson [EMAIL PROTECTED] http://www.ligo.caltech.edu/~anderson -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to exclude directories from source with --relative
On Mon, May 26, 2008 at 11:40:31PM -0700, Wayne Davison wrote: On Mon, May 26, 2008 at 09:46:04PM -0700, Stuart Anderson wrote: # rsync -avn --relative --exclude b/ /tmp/source/./a/? /tmp/destination That excludes all b dirs. It's safer to exclude /a/b/ (anchored and fully specified). The simple rule is to exclude the names that rsync outputs via -v. 3) Try excluding a/, this does not work: # rsync -avn --relative --exclude a/ /tmp/source/./a/? /tmp/destination That's because 'a' is an implied directory in the longer path that you told rsync to transfer. Implied directories can't be excluded, but their attributes can be omitted via --no-implied-dirs. If you don't want it in the transfer, don't specify it (and if you're using wild- cards, specify them in such as way as to avoid matching the dir you don't want). Nuts. I was afraid that was going to be the answer. Unfortunately, it violates the very helpful advise given above, i.e., run with -v and add exclude rules for files you don't want to transfer based on the output seen. Is this an enhancement request you are willing to consider, i.e., allow filtering of implied directories? The real problem I want to solve has a pattern matching requirement that is too complicated for a glob that specifies the implied directories, and while the explicit list can be generated outside of rsync it would be much more convenient to use rsync's --exclude mechanism. Thanks. -- Stuart Anderson [EMAIL PROTECTED] http://www.ligo.caltech.edu/~anderson -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: error in rsync protocol data stream (code 12) at token.c(288)
While this does appear to work in 2.5.4, I suspect that is because of the 2.5.4 whole-file bug. With 2.5.5 it fails again, with the following error message: deflate on token returned 0 (16384 bytes left) rsync error: error in rsync protocol data stream (code 12) at token.c(288) According to sba: I did one quick test with an unmodified rsync-2.5.4, and it seems to have fixed this problem. However, someone else should verify this. According to Dave Dykstra: I did some playing with this. It takes a while because the file is so big. I don't really know what's going on, but I did find that deflate() is returning at line 493 of zlib/deflate.c just after a comment that says this is not an error situation. So if I change like 285 of token.c to delete the || tx_strm.avail_in != 0 it all seems to work properly. I'm not sure if that's an OK thing to do or not, though. Somebody who knows something about zlib would have to look at it. - Dave Dykstra On Wed, Feb 06, 2002 at 04:21:47PM -0800, Stuart Anderson wrote: The newer zlib-1.1.3 patch still has the problem. According to Dave Dykstra: I was able to reproduce this between a Solaris 2.7 and 2.8 machine with and without ssh, although I don't have any time to debug it today. Can somebody try applying Jos Backus latest zlib 1.1.3 upgrade patch to see if that helps? - Dave On Fri, Feb 01, 2002 at 02:01:03PM -0800, Stuart Anderson wrote: According to Dave Dykstra: On Fri, Feb 01, 2002 at 09:09:18AM -0800, Stuart Anderson wrote: According to sba: No I did not apply the patch, but I verified the same problem with rsync-2.4.8, is that sufficient? No, because rsync-2.4.8 include the same security patch that went into 2.5.2 last weekend. If you could try 2.4.6 or 2.5.1 that would tell us if it was some other problem introduced last weekend or not. The problem was introduced in version 2.4.4 and exists in all later versions up to and including 2.5.2. Whereas version 2.4.3 and 2.4.0 are both fine (the only earlier versions checked). The problem only exists for compression and large files. I encountered and tested the various rsync versions with the file, redhat/7.1/en/iso/i386/seawolf-i386-powertools.iso gotten from your nearest RedHat mirror, but it is more general than just this file. However, the problem is not reproducible with any of the 1018 small files from redhat/7.1/en/os/i386/RedHat/RPMS which range in size from 1810 Byte to 21541821 Byte. To reproduce: 1) rsync --archive a large file (ISO size?) to a remote machine 2) /bin/touch it on the destination machine 3) rsync --archive --compress again 4) Check for error message: deflate on token returned 0 (16384 bytes left) I don't know if it matters but I am running rsync between two Solaris 8 machines, and using ssh as the transport: ssh -V OpenSSH_2.9p2, SSH protocols 1.5/2.0, OpenSSL 0x0090601f -- Stuart Anderson [EMAIL PROTECTED] http://www.srl.caltech.edu/personnel/sba -- Stuart Anderson [EMAIL PROTECTED] http://www.srl.caltech.edu/personnel/sba -- Stuart Anderson [EMAIL PROTECTED] http://www.srl.caltech.edu/personnel/sba -- Stuart Anderson [EMAIL PROTECTED] http://www.srl.caltech.edu/personnel/sba -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: error in rsync protocol data stream (code 12) at token.c(288)
According to Dave Dykstra:
On Thu, Jan 31, 2002 at 03:26:16PM -0800, Stuart Anderson wrote:
I am getting the following error when mirroring part of the RedHat
distribution tree over a slow connection (~T1 speed). When running
over a faster network (100BaseT) the problem does not appear. Note,
the problem file a large 600MB ISO image, whereas other small files
appear to be fine.
rsync: open connection using /path/ssh remote.host /path/rsync --server
-vlHogDtprRz --timeout=600 --delete --force . /
rsync: building file list...
rsync: 90108 files to consider.
export/mirror/linux/redhat/7.1/en/iso/i386/
/export/mirror/linux/redhat/7.1/en/iso/i386/seawolf-i386-powertools.iso
deflate on token returned 0 (16384 bytes left)
rsync error: error in rsync protocol data stream (code 12) at token.c(288)
rsync finished
The file has in fact been mirrored correctly (md5sum is identical),
however, the file is left with today's date rather synchronizing the
file times as specified.
A repeat of the rsync command with the file already on the destination
machines results in the same error.
This is running between two Sun servers running Solaris 8 and using:
rsync version 2.5.2 protocol version 26
Copyright (C) 1996-2002 by Andrew Tridgell and others
http://rsync.samba.org/
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, no IPv6,
64-bit system inums, 64-bit internal inums
That's a different symptom than what we've seen before, but have you
applied the following patch that I posted on Tuesday? Rsync 2.5.2 is badly
broken without it.
- Dave Dykstra
--- match.c.O Tue Jan 29 15:31:37 2002
+++ match.c Tue Jan 29 15:31:54 2002
@@ -246,7 +246,7 @@
match. The 3 reads are caused by the
running match, the checksum update and the
literal send. */
- if (offset-last_match = CHUNK_SIZE+s-n
+ if (offset-last_match = CHUNK_SIZE+(int)s-n
(end-offset CHUNK_SIZE)) {
matched(f,s,buf,offset - s-n, -2);
}
No I did not apply the patch, but I verified the same problem with
rsync-2.4.8, is that sufficient?
I also determined that my comment above about working/not-working on
fast/slow network was misleading. The important discriminant is whether
compression was turned on or not: works without out it, fails with it
(my driver script was automatically selecting compression for slow
networks).
I also found on deja-news reports that this is a known and fixed problem
with zlib-1.1.2. Any chance of rsync upgrading to zlib-1.1.3?
--
Stuart Anderson [EMAIL PROTECTED] http://www.srl.caltech.edu/personnel/sba
Re: error in rsync protocol data stream (code 12) at token.c(288)
According to sba:
According to Dave Dykstra:
On Thu, Jan 31, 2002 at 03:26:16PM -0800, Stuart Anderson wrote:
I am getting the following error when mirroring part of the RedHat
distribution tree over a slow connection (~T1 speed). When running
over a faster network (100BaseT) the problem does not appear. Note,
the problem file a large 600MB ISO image, whereas other small files
appear to be fine.
rsync: open connection using /path/ssh remote.host /path/rsync --server
-vlHogDtprRz --timeout=600 --delete --force . /
rsync: building file list...
rsync: 90108 files to consider.
export/mirror/linux/redhat/7.1/en/iso/i386/
/export/mirror/linux/redhat/7.1/en/iso/i386/seawolf-i386-powertools.iso
deflate on token returned 0 (16384 bytes left)
rsync error: error in rsync protocol data stream (code 12) at token.c(288)
rsync finished
The file has in fact been mirrored correctly (md5sum is identical),
however, the file is left with today's date rather synchronizing the
file times as specified.
A repeat of the rsync command with the file already on the destination
machines results in the same error.
This is running between two Sun servers running Solaris 8 and using:
rsync version 2.5.2 protocol version 26
Copyright (C) 1996-2002 by Andrew Tridgell and others
http://rsync.samba.org/
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, no
IPv6,
64-bit system inums, 64-bit internal inums
That's a different symptom than what we've seen before, but have you
applied the following patch that I posted on Tuesday? Rsync 2.5.2 is badly
broken without it.
- Dave Dykstra
--- match.c.O Tue Jan 29 15:31:37 2002
+++ match.c Tue Jan 29 15:31:54 2002
@@ -246,7 +246,7 @@
match. The 3 reads are caused by the
running match, the checksum update and the
literal send. */
- if (offset-last_match = CHUNK_SIZE+s-n
+ if (offset-last_match = CHUNK_SIZE+(int)s-n
(end-offset CHUNK_SIZE)) {
matched(f,s,buf,offset - s-n, -2);
}
No I did not apply the patch, but I verified the same problem with
rsync-2.4.8, is that sufficient?
I also determined that my comment above about working/not-working on
fast/slow network was misleading. The important discriminant is whether
compression was turned on or not: works without out it, fails with it
(my driver script was automatically selecting compression for slow
networks).
I also found on deja-news reports that this is a known and fixed problem
with zlib-1.1.2. Any chance of rsync upgrading to zlib-1.1.3?
I have verified that the (int) typecast fix to match.c does not solve
this problem.
I have also determined that a good way to reproduce this problem is
to touch (change the timestamp) on some large iso files and then
attempt to re-rsync them with compression turned on, i.e., no need
to re-push the actual file over the network.
--
Stuart Anderson [EMAIL PROTECTED] http://www.srl.caltech.edu/personnel/sba
Re: error in rsync protocol data stream (code 12) at token.c(288)
According to Albert Chin: On Fri, Feb 01, 2002 at 08:51:17AM -0800, Stuart Anderson wrote: I also found on deja-news reports that this is a known and fixed problem with zlib-1.1.2. Any chance of rsync upgrading to zlib-1.1.3? Jos Backus posted a patch to do this a few days ago. Try applying his patch and reporting back to the list if it solves your problem. Unfortunately, this does not solve my problem, i.e., the following fix in zlib-1.1.3 is not the one that rsync is triggering: - fix an inflate input buffer bug that shows up on rare but persistent occasions (Mark) At this point the workaround is to turn off compression. -- Stuart Anderson [EMAIL PROTECTED] http://www.srl.caltech.edu/personnel/sba
Re: error in rsync protocol data stream (code 12) at token.c(288)
According to Dave Dykstra: On Fri, Feb 01, 2002 at 09:09:18AM -0800, Stuart Anderson wrote: According to sba: No I did not apply the patch, but I verified the same problem with rsync-2.4.8, is that sufficient? No, because rsync-2.4.8 include the same security patch that went into 2.5.2 last weekend. If you could try 2.4.6 or 2.5.1 that would tell us if it was some other problem introduced last weekend or not. The problem was introduced in version 2.4.4 and exists in all later versions up to and including 2.5.2. Whereas version 2.4.3 and 2.4.0 are both fine (the only earlier versions checked). The problem only exists for compression and large files. I encountered and tested the various rsync versions with the file, redhat/7.1/en/iso/i386/seawolf-i386-powertools.iso gotten from your nearest RedHat mirror, but it is more general than just this file. However, the problem is not reproducible with any of the 1018 small files from redhat/7.1/en/os/i386/RedHat/RPMS which range in size from 1810 Byte to 21541821 Byte. To reproduce: 1) rsync --archive a large file (ISO size?) to a remote machine 2) /bin/touch it on the destination machine 3) rsync --archive --compress again 4) Check for error message: deflate on token returned 0 (16384 bytes left) I don't know if it matters but I am running rsync between two Solaris 8 machines, and using ssh as the transport: ssh -V OpenSSH_2.9p2, SSH protocols 1.5/2.0, OpenSSL 0x0090601f -- Stuart Anderson [EMAIL PROTECTED] http://www.srl.caltech.edu/personnel/sba
