Re: [Dorset] Exclude hidden folder in rsync
On 20/08/17 21:58, Ralph Corderoy wrote: Hi Tim, I want to compress my rsynced home folder, I want the video folder in the final compressed file but I don't want tar to try and compress it as it compresses the rest of the home folder, does that make sense? Yes, but it can't be done. tar(1) creates a tar file, and that's a simple catenation of a bit of metadata about the next file, and then the file's bytes, padded out to a nice round multiple of bytes for each file. Traditionally, you'd have either piped tar into a compression program, or produced foo.tar and then compressed it with a separate command. These days, tar has grown options for compressing, but they just do the pipe for you. Here's tar processing the `j' option and kicking off a bzip2(1) as the destination for write(2)s. $ strace -fe execve tar cjf /dev/null foo execve("/usr/bin/tar", ["tar", "cjf", "/dev/null", "foo"], 0x7ffe017683a8 /* 66 vars */) = 0 strace: Process 3162 attached strace: Process 3163 attached [pid 3163] execve("/bin/sh", ["/bin/sh", "-c", "bzip2"], 0x7ffe0deb68c0 /* 66 vars */) = 0 [pid 3163] execve("/usr/bin/bzip2", ["bzip2"], 0x73a950 /* 66 vars */) = 0 [pid 3163] +++ exited with 0 +++ [pid 3162] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3163, si_uid=1000, si_status=0, si_utime=0, si_stime=0} --- [pid 3162] +++ exited with 0 +++ --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3162, si_uid=1000, si_status=0, si_utime=0, si_stime=0} --- +++ exited with 0 +++ $ And the compression programs just see a stream of bytes, they've no idea that it's a tar file, and that you'd like the video directory to be skipped. That said, you could let the compressor attempt to do it anyway. It will burn some CPU, and may grow slightly in size. Or, exclude it from the compressed tar file and produce a second, video, uncompressed tar file. What do you do with these tar files then? Is rsync and tar the right combination? (I use tar for backups myself so I'm not against it per se.) Cheers, Ralph. Thanks for the input guys, I think I will leave the video folder in the tar. Rsync and tar just form a simple little backup for my home folder. It allows me to keep several copies of my home folder just encase something goes wrong. Tim -- Next meeting: Bournemouth, Tuesday, 2017-09-05 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Exclude hidden folder in rsync
> > I want to compress my rsynced home folder, I want the video > > folder in the final compressed file but I don't want tar to > > try and compress it as it compresses the rest of the home > > folder, does that make sense? > > Yes, but it can't be done. Unless you are prepared to accept two tar files as output. (One compressed, one uncompressed, with contents to be reunited upon untarring.) Patrick -- Next meeting: Bournemouth, Tuesday, 2017-09-05 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Exclude hidden folder in rsync
Hi Tim, > I want to compress my rsynced home folder, I want the video folder in > the final compressed file but I don't want tar to try and compress it > as it compresses the rest of the home folder, does that make sense? Yes, but it can't be done. tar(1) creates a tar file, and that's a simple catenation of a bit of metadata about the next file, and then the file's bytes, padded out to a nice round multiple of bytes for each file. Traditionally, you'd have either piped tar into a compression program, or produced foo.tar and then compressed it with a separate command. These days, tar has grown options for compressing, but they just do the pipe for you. Here's tar processing the `j' option and kicking off a bzip2(1) as the destination for write(2)s. $ strace -fe execve tar cjf /dev/null foo execve("/usr/bin/tar", ["tar", "cjf", "/dev/null", "foo"], 0x7ffe017683a8 /* 66 vars */) = 0 strace: Process 3162 attached strace: Process 3163 attached [pid 3163] execve("/bin/sh", ["/bin/sh", "-c", "bzip2"], 0x7ffe0deb68c0 /* 66 vars */) = 0 [pid 3163] execve("/usr/bin/bzip2", ["bzip2"], 0x73a950 /* 66 vars */) = 0 [pid 3163] +++ exited with 0 +++ [pid 3162] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3163, si_uid=1000, si_status=0, si_utime=0, si_stime=0} --- [pid 3162] +++ exited with 0 +++ --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3162, si_uid=1000, si_status=0, si_utime=0, si_stime=0} --- +++ exited with 0 +++ $ And the compression programs just see a stream of bytes, they've no idea that it's a tar file, and that you'd like the video directory to be skipped. That said, you could let the compressor attempt to do it anyway. It will burn some CPU, and may grow slightly in size. Or, exclude it from the compressed tar file and produce a second, video, uncompressed tar file. What do you do with these tar files then? Is rsync and tar the right combination? (I use tar for backups myself so I'm not against it per se.) Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2017-09-05 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Exclude hidden folder in rsync
On 20/08/17 16:12, Tim wrote: On 20/08/17 11:16, Ralph Corderoy wrote: Hi Tim, what is the correct phrase to stop rsync from copying the cache folder? Hard to say what you're doing wrong without an example. It works for me. $ find src | sort src src/bar src/bar/file1 src/.cache src/.cache/file1 src/file1 src/foo src/foo/.cache src/foo/.cache/file1 src/foo/file1 $ $ rsync -aciHAXS --exclude /.cache/ src/ dest cd+ ./ >f+ file1 cd+ bar/ >f+ bar/file1 cd+ foo/ >f+ foo/file1 cd+ foo/.cache/ >f+ foo/.cache/file1 $ $ >src/foo/file2 $ rsync -aciHAXS --exclude /.cache/ src/ dest .d..t.. foo/ >f+ foo/file2 $ $ find -name .cache | sorV ./dest/foo/.cache ./src/.cache ./src/foo/.cache $ Cheers, Ralph. Hi Ralph Thanks, from what you posted I managed to get it working, it is basically my home folder I am backing up but I am trying to safe space on my disk by simply syncing those folders in my home folder that I will want if I lost my home folder on my main disk. I don't really need .thumbnails and .cache which were two of the biggest space takers I did not need. To move this on slightly, once rsync has run I want compress the whole home folders (it copies it to another folder then compresses for storage), it is currently around 21gb but half of that is a video folder that I have which is 10gb in size, how do I exclude a folder from tar? I can see where you can choose files and patters *.ogg but the videos are various formats so it would be simpler to exclude the whole folder? is it as simple as --exclude=videofolder? Not sure I am reading the man page correctly Tim After reading something online I am going to clarify what I am trying to do, I want to compress my rsynced home folder, I want the video folder in the final compressed file but I don't want tar to try and compress it as it compresses the rest of the home folder, does that make sense? Tim -- Next meeting: Bournemouth, Tuesday, 2017-09-05 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Exclude hidden folder in rsync
On 20/08/17 11:16, Ralph Corderoy wrote: Hi Tim, what is the correct phrase to stop rsync from copying the cache folder? Hard to say what you're doing wrong without an example. It works for me. $ find src | sort src src/bar src/bar/file1 src/.cache src/.cache/file1 src/file1 src/foo src/foo/.cache src/foo/.cache/file1 src/foo/file1 $ $ rsync -aciHAXS --exclude /.cache/ src/ dest cd+ ./ >f+ file1 cd+ bar/ >f+ bar/file1 cd+ foo/ >f+ foo/file1 cd+ foo/.cache/ >f+ foo/.cache/file1 $ $ >src/foo/file2 $ rsync -aciHAXS --exclude /.cache/ src/ dest .d..t.. foo/ >f+ foo/file2 $ $ find -name .cache | sorV ./dest/foo/.cache ./src/.cache ./src/foo/.cache $ Cheers, Ralph. Hi Ralph Thanks, from what you posted I managed to get it working, it is basically my home folder I am backing up but I am trying to safe space on my disk by simply syncing those folders in my home folder that I will want if I lost my home folder on my main disk. I don't really need .thumbnails and .cache which were two of the biggest space takers I did not need. To move this on slightly, once rsync has run I want compress the whole home folders (it copies it to another folder then compresses for storage), it is currently around 21gb but half of that is a video folder that I have which is 10gb in size, how do I exclude a folder from tar? I can see where you can choose files and patters *.ogg but the videos are various formats so it would be simpler to exclude the whole folder? is it as simple as --exclude=videofolder? Not sure I am reading the man page correctly Tim -- Next meeting: Bournemouth, Tuesday, 2017-09-05 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Exclude hidden folder in rsync
Hi Tim, > what is the correct phrase to > stop rsync from copying the cache folder? Hard to say what you're doing wrong without an example. It works for me. $ find src | sort src src/bar src/bar/file1 src/.cache src/.cache/file1 src/file1 src/foo src/foo/.cache src/foo/.cache/file1 src/foo/file1 $ $ rsync -aciHAXS --exclude /.cache/ src/ dest cd+ ./ >f+ file1 cd+ bar/ >f+ bar/file1 cd+ foo/ >f+ foo/file1 cd+ foo/.cache/ >f+ foo/.cache/file1 $ $ >src/foo/file2 $ rsync -aciHAXS --exclude /.cache/ src/ dest .d..t.. foo/ >f+ foo/file2 $ $ find -name .cache | sorV ./dest/foo/.cache ./src/.cache ./src/foo/.cache $ Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2017-09-05 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
[Dorset] Exclude hidden folder in rsync
I have a hidden folder in my home folder called .cache . I want to block this ".cache" folder from being backed up (or the source and destination folders being synced if you prefer). I have tried many permutations but it is still synced when I run the script, what is the correct phrase to stop rsync from copying the cache folder? Tim -- Next meeting: Bournemouth, Tuesday, 2017-09-05 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR