Re: exclude a pattern but only in the top level
On Wed 15 May 2013, Brian K. White wrote: > I did in the case when it was only one pattern, but that was just a > simplified example. > > The actual job involves too many include and exclude patterns to use > --include & --exclude, or even --include-from and --exclude-from, > because the patterns are generated on the fly by a script from > values supplied on the script commandline or hardcoded at the top, I think you're missing the point. --exclude-from=filename will do the same. -f ". filename" specifies a rule, and that rule tells rsync to read the file. --exclude-from=filename tells rsync directly to read the file. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Don't follow bind mounts?
Hello, The manual says that rsync treats bind mounts on UNIX (Linux) to the same filesystem as being on the same filesystem. I have a server with a pile of bind mounts to the same filesystem for some access control/ease of use for FTP users modifying websites. This makes my backups using rsync messy! Is there any way to stop rsync from following bind mounts to the same filesystem? Short of unmounting them all at backup time and remounting afterwards or explicitly excluding each one? -x -x seemed hopeful, but it didn't stop following the mount and I ended up with two copies (or more!) of the data on my backup server. Googling just takes me to the man page saying "no", in a roundabout sort of way. I found this : http://samba.2283325.n4.nabble.com/PATCH-One-more-x-to-not-traverse-cyclic-bind-mounts-td3216379.html Which never made it into rsync, it seems (didn't work?) and the script mnt-excl suggests that it is redundant for rsync >= 2.6.4. any examples of use of : awk '{print $2}' /proc/mounts | rsync -f 'merge,/- -' host:/dir /dest/ around with some details of its use for backups? Thank you Carl -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exclude a pattern but only in the top level
On 5/15/2013 8:35 AM, Paul Slootman wrote: On Wed 15 May 2013, Brian K. White wrote: rsync -avz /foo/ ${DEST}::root/foo/ This syntax does work in his case, and is easier to read, because it ends up using the exact same specification "/foo/" and "/foo/" for both source and dest, but the syntax I had was also correct. I no longer remember why I always do the way I posted, but I've been doing it that way for decades (if you count rcp before rsync). My reasoning is that if you're transferring a directory, make that obvious by specifying a trailing slash. But use whatever you're happy with, just don't complain if things get confusing (which it looked like was the problem). The -f ". filename" syntax is correct it's right in the manual and I've been using it for ages. Well, -f specifies a filter rule. You're using -f to specify a filter rule to tell rsync to merge in a file with additional filter rules, which is a bit of a roundabout way. Why not tell rsync directly to read a file with include/exclude rules by using --exclude-from I did in the case when it was only one pattern, but that was just a simplified example. The actual job involves too many include and exclude patterns to use --include & --exclude, or even --include-from and --exclude-from, because the patterns are generated on the fly by a script from values supplied on the script commandline or hardcoded at the top, and for each of those values several different include and exclude patterns are generated, and, the order they appear in the filter file matters, I need to write the filter file so that certain things get included before others, and all the includes & excludes aren't necessarily together in two contiguous blocks. -f ". filename" is actually the simplest and neatest way to express the job. -- bkw -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exclude a pattern but only in the top level
On Wed 15 May 2013, Brian K. White wrote: > > rsync -avz /foo/ ${DEST}::root/foo/ > > This syntax does work in his case, and is easier to read, because it > ends up using the exact same specification "/foo/" and "/foo/" for > both source and dest, but the syntax I had was also correct. I no > longer remember why I always do the way I posted, but I've been > doing it that way for decades (if you count rcp before rsync). My reasoning is that if you're transferring a directory, make that obvious by specifying a trailing slash. But use whatever you're happy with, just don't complain if things get confusing (which it looked like was the problem). > The -f ". filename" syntax is correct it's right in the manual and > I've been using it for ages. Well, -f specifies a filter rule. You're using -f to specify a filter rule to tell rsync to merge in a file with additional filter rules, which is a bit of a roundabout way. Why not tell rsync directly to read a file with include/exclude rules by using --exclude-from Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exclude a pattern but only in the top level
On 5/15/2013 5:17 AM, Paul Slootman wrote: On Wed 15 May 2013, Brian K. White wrote: Consider the following directory structure /foo/aaa/*/* /foo/bbb/*/* /foo/ccc/*/* I want to sync all of /foo, but exclude /foo/aaa rsync -avz /foo ${DEST}::root Firstly, I always recommend that with directory transfers you add a trailing slash to the source, so your command becomes: rsync -avz /foo/ ${DEST}::root/foo/ Then a filter would be: - /aaa/ + /* Note the leading slash, as your aaa directory is now in the root of the source. The second line is not really needed in this scenario IMHO To be honest I'm wondering about your usage of -f ". filter", I always do --exclude-from=filter > rsync -avz /foo/ ${DEST}::root/foo/ This syntax does work in his case, and is easier to read, because it ends up using the exact same specification "/foo/" and "/foo/" for both source and dest, but the syntax I had was also correct. I no longer remember why I always do the way I posted, but I've been doing it that way for decades (if you count rcp before rsync). It might be that the way I did it _always_ works as expected, where this way maybe doesn't work as expected in some cases, like maybe if the source is a file named /foo/aaa, so you use "/foo/aaa" the same in both the source and dest, but the dest machine happens to have a directory named /foo/aaa, then instead of having the dest directory converted to a file to match the source, the file from the source would be placed _within_ the directory on the dest, creating a file named /foo/aaa/aaa instead of /foo/aaa on dest. So, I'd rather, it's safer to do something the same way every time and know that it works the same way every time where possible. If you manually remove the trailing path component from the dest, then it always works the same way regardless if source is a file or dir and regardless if dest already has a file or dir of the same name. You follow the same exact rule every time and get the same exact result every time. No unpleasant surprises. The -f ". filename" syntax is correct it's right in the manual and I've been using it for ages. You're right the leading slash in the exclude pattern was the key in this case. -- bkw -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exclude a pattern but only in the top level
On Wed 15 May 2013, Brian K. White wrote: > Consider the following directory structure > > /foo/aaa/*/* > /foo/bbb/*/* > /foo/ccc/*/* > > I want to sync all of /foo, > but exclude /foo/aaa > rsync -avz /foo ${DEST}::root Firstly, I always recommend that with directory transfers you add a trailing slash to the source, so your command becomes: rsync -avz /foo/ ${DEST}::root/foo/ Then a filter would be: - /aaa/ + /* Note the leading slash, as your aaa directory is now in the root of the source. The second line is not really needed in this scenario IMHO To be honest I'm wondering about your usage of -f ". filter", I always do --exclude-from=filter Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exclude a pattern but only in the top level
On 5/15/2013 4:13 AM, Brian K. White wrote: Consider the following directory structure /foo/aaa/*/* /foo/bbb/*/* /foo/ccc/*/* I want to sync all of /foo, but exclude /foo/aaa but not exclude any other occurances of "aaa" or "foo/aaa" (be they files or dirs) that might occur within the other dirs /foo/bbb/* /foo/ccc/* etc I don't want to exclude /foo/bbb/aaa or /foo/ccc/111/aaa or /foo/ccc/111/foo/aaa etc... Destination is running rsync daemon and has a module named root that points to "/" such that normally, for the full /foo with no exclude, it's very simple, my rsync command would just be: rsync -avz /foo ${DEST}::root That works fine without the exclude. So, to that I want to add a filter file. (Well, I assume that's what I want) rsync -avz -f ". filter" /foo ${DEST}::root If I construct a filter file like this: - aaa + * or like this - aaa/ + * or like this - foo/aaa/ + * rsync hides the top level /foo/aaa but it also hides any other occurrences of the exclude pattern that occur anywhere within the job instead of just the top level one. How can I get it to exclude just the top-level directory "/foo/aaa" ? Never mind, I think I got it. I saw the special meaning of a leading slash in an include/exclude pattern in the man page. So: rsync -aznvvv --exclude '/pix/trinity/' /pix ${DEST}::root ... [sender] hiding directory pix/trinity because of pattern /pix/trinity/ ... [sender] make_file(pix/0/trinity,*,2) [sender] make_file(pix/0/pix/trinity,*,2) [sender] make_file(pix/0/pix/trinity/foo,*,2) [sender] make_file(pix/0/trinity/trinity,*,2) So it excluded /pix/trinity/ as desired, yet included other paths that had "/pix/trinity/" in them such as /pix/0/pix/trinity/foo" I only need the single pattern at the moment so --exclude is simpler than a filter file, but just for the record I verified a filter file like this worked the same too: - /pix/trinity/ Also please disregard the "+ *" in my previous examples. I know they are superfluous. They were in there only because I was starting with a copy of a different script that originally had "- *" at the bottom to include some patterns and exclude everything else. At the moment I need the opposite of that so when I reversed all the + & -, I ended up with a "+ *" at the bottom. -- bkw -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exclude a pattern but only in the top level
On 15.05.2013 04:13, Brian K. White wrote: > How can I get it to exclude just the top-level directory "/foo/aaa" ? With a '/' at the beginning you pin the pattern to the beginning. You don't use "/" at the end of directory names, so i'm not sure if the correct one is: /aaa or /foo/aaa With a '/' at the end (... source/ target/ ...) it's the first one, in your case it might be the other one. -- Matthias -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
exclude a pattern but only in the top level
Consider the following directory structure /foo/aaa/*/* /foo/bbb/*/* /foo/ccc/*/* I want to sync all of /foo, but exclude /foo/aaa but not exclude any other occurances of "aaa" or "foo/aaa" (be they files or dirs) that might occur within the other dirs /foo/bbb/* /foo/ccc/* etc I don't want to exclude /foo/bbb/aaa or /foo/ccc/111/aaa or /foo/ccc/111/foo/aaa etc... Destination is running rsync daemon and has a module named root that points to "/" such that normally, for the full /foo with no exclude, it's very simple, my rsync command would just be: rsync -avz /foo ${DEST}::root That works fine without the exclude. So, to that I want to add a filter file. (Well, I assume that's what I want) rsync -avz -f ". filter" /foo ${DEST}::root If I construct a filter file like this: - aaa + * or like this - aaa/ + * or like this - foo/aaa/ + * rsync hides the top level /foo/aaa but it also hides any other occurrences of the exclude pattern that occur anywhere within the job instead of just the top level one. How can I get it to exclude just the top-level directory "/foo/aaa" ? -- bkw -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html