Re: --detect-renamed --detect-moved and -b
Matt McCutchen wrote: On Wed, 2009-11-11 at 20:27 +0800, Thomas Gutzler wrote: I played around with the combination of rsync --detect-renamed --detect-moved and -b. You can't combine the --detect-renamed and --detect-moved options because they make use of the partial dir in incompatible ways. The last option on the command line takes priority. (Yes, the implementation of --detect-moved is a hack. No, I don't care to improve it myself.) I see. That's ok though, because I'm much more interested in the case that a user renames a directory which contained 100G of data than the case that a user renames 100G worth of files. Yes, the current design is that a deleted destination file is backed up even if it was also detected as a rename. In addition, your case tickles an awkward-to-fix bug in the --detect-renamed implementation: renames into new directories are not processed because their partial dirs are not available yet. See: http://lists.samba.org/archive/rsync/2007-December/019469.html Great. I'm running an --include='*/' --exclude='*' anyway, so I've got the directories. Any chance for a --exclude-moved-from-backup flag? Tom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Yet another include/exclude question
Thanks everyone for your help, I've got what I want. Wayne Davison wrote: On Sun, Nov 8, 2009 at 10:21 PM, Thomas Gutzler thomas.gutz...@gmail.com mailto:thomas.gutz...@gmail.com wrote: I thought --include=/this_dir/ --include=/this_dir/*** would do it, but it doesn't. The exclude * seems to overwrite the include matches: [sender] hiding file this_dir/foo because of pattern * Order is important. Whatever matches first, is what takes effect. Also, *** matches both the dir and its contents, so you could use: --include='/this_dir/***' --exclude=* It seems to be very picky about the order. Thanks for pointing that out. My first attempt has been --include=*/ --include=*.foo --include=*.bar --include=/this_dir/*** --exclude=* which did nothing than *.foo and *.bar. Shuffling it around, I found that --include=*/ --include=/this_dir/*** --include=*.foo --include=*.bar --exclude=* does what I want and it even makes sense. Cheers, Tom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Yet another include/exclude question
Matt McCutchen wrote: On Tue, 2009-11-10 at 09:45 +0800, Thomas Gutzler wrote: Thanks everyone for your help, I've got what I want. My first attempt has been --include=*/ --include=*.foo --include=*.bar --include=/this_dir/*** --exclude=* which did nothing than *.foo and *.bar. Shuffling it around, I found that --include=*/ --include=/this_dir/*** --include=*.foo --include=*.bar --exclude=* does what I want and it even makes sense. Those two commands should be equivalent. If you have a reproducible case in which they aren't, please share it and we can see if there's a bug. Going through my history, I found that I must have accidentally put a whitespace between '--include' and '=' which wasn't very obvious at the time thanks to the line wrapping of my terminal. I should have used an include file instead :) So no, no bugs. Tom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Yet another include/exclude question
What if I also want to include everything in /this_dir/ without running two instances and still being able to use --delete? I thought --include=/this_dir/ --include=/this_dir/*** would do it, but it doesn't. The exclude * seems to overwrite the include matches: [sender] hiding file this_dir/foo because of pattern * Tom Steven Monai wrote: Tony wrote: On Nov 6, 2009, at 12:43 AM, Thomas Gutzler wrote: Hi, How can I include only *.foo and *.bar files in an rsync? To make it a bit more difficult: Those files can be anywhere in the directory structure. I've been reading and trying for a while but the best I got is all *.foo and *.bar files in ./ but none of the ones inside any directories. There is an example in the man pages for *.c files that will do what you want. Try: --include=*/ --include=*.foo --exclude=* Yes. I would also strongly recommend the section of the man page that describes --prune-empty-dirs. Cheers, -SM -- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Yet another include/exclude question
Hi, How can I include only *.foo and *.bar files in an rsync? To make it a bit more difficult: Those files can be anywhere in the directory structure. I've been reading and trying for a while but the best I got is all *.foo and *.bar files in ./ but none of the ones inside any directories. Cheers, Tom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
exclude files from backup with rsync -b
Hi, is there a way to exclude files from being backed up when running rsync -b? Let's assume the following file tree: ./src/f1 ./src/temp/t1 ./dest/src/f1 ./dest/src/temp/t1 Now, I'm modifying both files f1 and t1 and run rsync -a --relative -b --backup-dir=/backup/ src/ dest/ Both ./dest/src/f1 and ./dest/src/temp/t1 should have been updated but I only want /backup/ to contain the old src/f1 and NOT the old temp/t1. How? Tom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Feature I'd love to see: --move-dest
Hi, This seems to be a reappearing request. Interestingly enough, I wanted the exact same thing a while ago and nobody was able to suggest a quick solution without implementing a new feature. I didn't have the time to read into the rsync source code, so I decided to write a little perl script, which works similar to rsnapshot. It - keeps one exact copy of the source which is updated every time - stores the changes between the current and the last run in an extra directory which is labelled interval_date_time - supports different intervals by merging several more frequent ones into one less frequent (7 daily - 1 weekly) - does not use hard links At the moment, I run it about 5 times a day on a 3TB file system (1.2M files) with about 400MB changes each time and it takes around half an hour. I haven't put an amazing amount of work into it because it was never supposed to be used by anyone else than myself; it's not foolproof but I am willing to share it around and improve it if there is interest. Cheers, Tom Josh Hanson wrote: David, You make some good points. However, I still think a --move-dest feature would be handy. First of all, --link-dest is nice on platforms and file systems that support hard links, but not all platforms and file systems do, and some that do have tagged them on as an afterthought but don't support them very well, or very efficiently. Second, even on systems that do support it, for frequent backups of large numbers of files which may change only rarely, the size of all the hard links may not be negligible. Finally, it's always nice, given the option, to avoid hard links and keep a one-to-one relationship between the files I see (i.e. the links) and the actual allocated regions on the disk. Hard links make it impossible to know just how much space each backup takes up. If my backup disk is filling up, I'd like to be able to see which files are taking the most space; I might decide to manually delete older copies of large files, and I won't want to go through and delete every single link to the file. -- Josh -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of David Overton Sent: Monday, December 08, 2008 5:35 PM To: Josh Hanson Cc: rsync@lists.samba.org Subject: Re: Feature I'd love to see: --move-dest Hi, I'm curious as to why you need a new option rather than just using --link-dest. Both --link-dest and --copy-dest already allow the two very common restor scenarios that you mention and --link-dest uses hardly any extra disk space than your --move-dest suggestion. You also get the added advantage that each previous backup continues to look like a full snapshot. David 2008/12/9 Josh Hanson [EMAIL PROTECTED]: Looking over rsync's --compare-dest, --copy-dest, and --link-dest options, there's one thing I really wish I could do, and I've been looking for a few years now for a program that could do it: Instead of copying or hard-linking from the compare directory to the target directory, I'd like to move the files. That is, whenever the file already exists in the compare directory, it is moved forward into the target directory; but if the existing file is different or doesn't exist, then a new file is made in the target directory. The result would be that, after several backups (each to a new directory, perhaps named with the current date) the most recent directory would be a snapshot of the source, and each previous directory would have previous versions of files or copies of deleted files. (This is effectively the opposite result of --compare-dest, which gives you a snapshot of the first version of the files, with each successive directory storing only the changed files.) This makes two very common restore scenarios very simple: restoring the entire directory tree to the most recent version after a drive failure, and restoring a previous version of a single file after it was accidentally over-written, even if the modified version has since been backed up. I looked over the rsync source to see if I could add this functionality myself, but I'm really not a very good programmer and it's definitely beyond me. To someone familiar with the source, however, I have to imagine it would be a fairly straightforward feature to add... just add replace the copy operation in --copy-dest or the link operation in --link-dest with a move operation. -- Josh -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Thomas Gutzler Optical+Biomedical Engineering Laboratory The University of Western Australia http://obel.ee.uwa.edu.au/ phone: +61 8 6488 3916 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https
Re: rsync hangs during --remove-source-files
Wayne Davison wrote: On Fri, Oct 17, 2008 at 03:34:03PM +0800, Thomas Gutzler wrote: I had another rsync hang. I've attached a text file with the stack traces you asked for and some other info. [...] So, the strange thing is that the receiver is waiting around for more files to process, which raises the question of why the generator doesn't know that the files it initiated are all done (or make me wonder if some pipe data didn't get delivered, but that is less likely). Did your transfer report any errors? No, there are no errors in the log file expect the one when I killed the process. I've just switched back to rsync version 2.6.9 - see if that works. Happy to test/debug stuff, you just have to tell me what to do. Tom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync hangs during --remove-source-files
Hi, I had another rsync hang. I've attached a text file with the stack traces you asked for and some other info. There were three rsync processes running at the time. I haven't heard anything from Wayne yet - maybe this generates more interest. Cheers, Tom Matt McCutchen wrote: On Mon, 2008-10-13 at 21:19 +0800, Thomas Gutzler wrote: I'm using the following command to move files from a to b: rsync -a --remove-source-files --ignore-existing /path/to/dir1/ /path/to/dir2 logfile dir2 isn't empty so I only move files from dir1 to dir2 that don't exist already. Every so often rsync stops in the middle and doesn't continue with the result that I have to kill it. Any suggestions how to debug this so that I can find out what's going on? Unfortunately I haven't found a way to reproduce it. Every time I kill and execute the same command again, it works. The next time rsync hangs, before you kill it, attach gdb to each of the three rsync processes (gdb /PATH/TO/rsync PID) and get a stack trace (bt). Wayne should have more suggestions for what information would be helpful for debugging. Matt [EMAIL PROTECTED]:~# ps ax | grep rsync 5249 ?S 0:00 sh -c /usr/local/rsync3/bin/rsync -a --remove-source-files --ignore-existing /home/backup/snapshots/frequent_2008-10-09_18-00-01/ /home/backup/snapshots/daily_2008-10-17_05-00-01 /home/backup/logs/frequent_2008-10-17_12-00-01.log 21 5250 ?S 0:02 /usr/local/rsync3/bin/rsync -a --remove-source-files --ignore-existing /home/backup/snapshots/frequent_2008-10-09_18-00-01/ /home/backup/snapshots/daily_2008-10-17_05-00-01 5251 ?S 0:00 /usr/local/rsync3/bin/rsync -a --remove-source-files --ignore-existing /home/backup/snapshots/frequent_2008-10-09_18-00-01/ /home/backup/snapshots/daily_2008-10-17_05-00-01 5252 ?S 0:02 /usr/local/rsync3/bin/rsync -a --remove-source-files --ignore-existing /home/backup/snapshots/frequent_2008-10-09_18-00-01/ /home/backup/snapshots/daily_2008-10-17_05-00-01 [EMAIL PROTECTED]:~# lsof | grep rsync | grep backup COMMAND PIDUSER FD TYPE DEVICESIZE NODE NAME rsync 5250root cwd DIR 254,04096 15290437 /home/backup/snapshots/frequent_2008-10-09_18-00-01 rsync 5250root1w REG 254,0 1258828 76689435 /home/backup/logs/frequent_2008-10-17_12-00-01.log rsync 5250root2w REG 254,0 1258828 76689435 /home/backup/logs/frequent_2008-10-17_12-00-01.log rsync 5251root cwd DIR 254,04096 15294467 /home/backup/snapshots/daily_2008-10-17_05-00-01 rsync 5251root2w REG 254,0 1258828 76689435 /home/backup/logs/frequent_2008-10-17_12-00-01.log rsync 5252root cwd DIR 254,04096 15294467 /home/backup/snapshots/daily_2008-10-17_05-00-01 rsync 5252root2w REG 254,0 1258828 76689435 /home/backup/logs/frequent_2008-10-17_12-00-01.log [EMAIL PROTECTED]:/home/backup/logs# tail -2 frequent_2008-10-17_12-00-01.log touch_up_dirs: home/io/obel/acuratolo/.winnt/profile/Cookies (20) set modtime of home/io/obel/acuratolo/.winnt/profile/Cookies to (1223546878) Thu Oct 9 18:07:58 2008 [EMAIL PROTECTED]:/home/backup/snapshots# ll -d daily_2008-10-17_05-00-01/home/io/obel/acuratolo/.winnt/profile/Cookies frequent_2008-10-09_18-00-01/home/io/obel/acuratolo/.winnt/profile/Cookies drwx-- 2 1161 1097 4096 2008-10-09 18:07 daily_2008-10-17_05-00-01/home/io/obel/acuratolo/.winnt/profile/Cookies/ drwx-- 2 1161 1097 4096 2008-10-17 12:15 frequent_2008-10-09_18-00-01/home/io/obel/acuratolo/.winnt/profile/Cookies/ [EMAIL PROTECTED]:~# gdb /usr/local/rsync3/bin/rsync 5250 GNU gdb 6.4.90-debian Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as i486-linux-gnu...Using host libthread_db library /lib/tls/i686/cmov/libthread_db.so.1. Attaching to program: /usr/local/rsync3/bin/rsync, process 5250 Reading symbols from /lib/tls/i686/cmov/libc.so.6...done. Loaded symbols for /lib/tls/i686/cmov/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/tls/i686/cmov/libnss_compat.so.2...done. Loaded symbols for /lib/tls/i686/cmov/libnss_compat.so.2 Reading symbols from /lib/tls/i686/cmov/libnsl.so.1...done. Loaded symbols for /lib/tls/i686/cmov/libnsl.so.1 Reading symbols from /lib/tls/i686/cmov/libnss_nis.so.2...done. Loaded symbols for /lib/tls/i686/cmov/libnss_nis.so.2 Reading symbols from /lib/tls/i686/cmov/libnss_files.so.2...done. Loaded symbols for /lib/tls/i686/cmov
rsync hangs during --remove-source-files
Hi, I'm using the following command to move files from a to b: rsync -a --remove-source-files --ignore-existing /path/to/dir1/ /path/to/dir2 logfile dir2 isn't empty so I only move files from dir1 to dir2 that don't exist already. Every so often rsync stops in the middle and doesn't continue with the result that I have to kill it. The last time it happened, rsync had successfully moved 1320 files, there were 2943 files in dir2 and 1101 files in dir1. lsof showed no open files inside dir1 or dir2, the log file was opened 4 times, dir1 itself showed up and dir2 twice. This is on a dual core CPU with rsync version 3.0.4 protocol version 30. The log file shows no errors. Last 4 lines say: renaming dir/res/.svn/text-base/.somefile.svn-base.mDBdUb to dir/res/.svn/text-base/somefile.svn-base sender removed dir/.svn/text-base/somefile.svn-base touch_up_dirs: dir/.svn/prop-base (84) set modtime of dir/res/.svn/prop-base to (1223212308) Sun Oct 5 21:11:48 2008 Any suggestions how to debug this so that I can find out what's going on? Unfortunately I haven't found a way to reproduce it. Every time I kill and execute the same command again, it works. Cheers, Tom -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
