rsync seems to overwhelm a failing hard disk
Hello rsync people Today I was recovering data from a beginning-to-fail external USB hard disk. I started with my usual 'rsync -av --ignore-errors source dest', and that was fine until it got to the first I/O errors. It paused but continued after the first couple of errors, but then the disk started buzzing and rsync gave error messages for every file (I'm afraid I didn't note the exact text of the messages). After that, every run of rsync (trying to --exclude the faulty files and folders) started the disk buzzing again, and rsync couldn't copy anything. The curious thing was that after replugging the disk, I was able to manually drag-and-drop a lot more files from the disk via the Debian desktop. Trying rsync again quickly caused the buzzing problem again. In other words, a simple file copy was able to read the disk when rsync gave errors. Is that to be expected? Is there a way to make rsync more gentle on a fragile disk? cheers Chris -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync equivalent of 'cp -al' ?
Hello rsync people I thought I knew how to use rsync, but I can't work out how to use it to do the equivalent of cp -al dir1 dir2 where dir1 and dir2 are both local and on the same disk. In other words I want to make dir2 a copy of dir1, with every file hard-linked to its counterpart in dir1. Why not just use cp? Because I want to be able to do it as a user who has sudo permission to run rsync but not cp. cheers Chris -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync equivalent of 'cp -al' ?
On 18/06/13 15:02, Kevin Korb wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 rsync -vai --lin-dest=/path/to/source/ /path/to/source/ /path/to/target/ Note that if you try it with relative paths the link-dest will be relative to the source not . Thank you Kevin. I'd forgotten that --link-dest needed a relative path. This works for me: rsync -Haxv --stats --link-dest ../dir1 dir1/ dir2 The / on the end of the second dir1 is, of course, essential. cheers Chris On 06/18/13 09:39, Chris Dennis wrote: Hello rsync people I thought I knew how to use rsync, but I can't work out how to use it to do the equivalent of cp -al dir1 dir2 where dir1 and dir2 are both local and on the same disk. In other words I want to make dir2 a copy of dir1, with every file hard-linked to its counterpart in dir1. Why not just use cp? Because I want to be able to do it as a user who has sudo permission to run rsync but not cp. cheers Chris - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.20 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHAaIgACgkQVKC1jlbQAQcKLQCfSaHopk/J2bhPCUFuwR3KSIMM daQAnjpVXhgeRz0bvb2/ttKVNZHAmuq6 =5f7j -END PGP SIGNATURE- -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync equivalent of 'cp -al' ?
On 18/06/13 16:53, Kevin Korb wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The -H there isn't needed and could only cause increased memory usage. I realise that --link-dest implies hard links between directories, but I use -H as well to maintain any hard links within the source directory. Does that make sense? cheers Chris On 06/18/13 11:49, Chris Dennis wrote: On 18/06/13 15:02, Kevin Korb wrote: rsync -vai --lin-dest=/path/to/source/ /path/to/source/ /path/to/target/ Note that if you try it with relative paths the link-dest will be relative to the source not . Thank you Kevin. I'd forgotten that --link-dest needed a relative path. This works for me: rsync -Haxv --stats --link-dest ../dir1 dir1/ dir2 The / on the end of the second dir1 is, of course, essential. cheers Chris -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync equivalent of 'cp -al' ?
On 18/06/13 17:35, Kevin Korb wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 If you make a link to a link you make a link to all of its links. The effect is the same. Good point -- I hadn't thought of it like that. Thanks for the tip. cheers Chris On 06/18/13 12:27, Chris Dennis wrote: On 18/06/13 16:53, Kevin Korb wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The -H there isn't needed and could only cause increased memory usage. I realise that --link-dest implies hard links between directories, but I use -H as well to maintain any hard links within the source directory. Does that make sense? cheers Chris On 06/18/13 11:49, Chris Dennis wrote: On 18/06/13 15:02, Kevin Korb wrote: rsync -vai --lin-dest=/path/to/source/ /path/to/source/ /path/to/target/ Note that if you try it with relative paths the link-dest will be relative to the source not . Thank you Kevin. I'd forgotten that --link-dest needed a relative path. This works for me: rsync -Haxv --stats --link-dest ../dir1 dir1/ dir2 The / on the end of the second dir1 is, of course, essential. cheers Chris - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.20 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHAjG4ACgkQVKC1jlbQAQcvVgCgks/5JRXax+OLMVH3chMOqge8 vMsAoJYat6BZWtDdDGeg2kTQvr0gT5Rx =xaMU -END PGP SIGNATURE- -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: speedup is always 0.99
On 22/02/13 21:23, Grant wrote: I'm syncing from a USB disk to my hard disk like this: rsync -vr /path/to/usb/disk/dir/ /path/to/hard/disk/dir/ But the speedup is always 0.99 which I think means it is just copying the files each time instead of syncing them. What could be wrong? - Grant If either of the disks is formatted with FAT, then you'll need --modify-window=1 -- see the man page or http://serverfault.com/questions/54949/how-can-i-use-rsync-with-a-fat-file-system cheers Chris -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--list-only ordering
Hello rsync people I've noticed an apparent inconsistency in the ordering of output from the --list-only option. For example: $ ls d1 d2 d2-x d3 f1 f2 f2-x f3 $ rsync --list-only . drwxr-xr-x4096 2012/12/17 15:18:05 . -rw-r--r-- 0 2012/12/17 15:17:52 f1 -rw-r--r-- 0 2012/12/17 15:17:52 f2 -rw-r--r-- 0 2012/12/17 15:17:52 f2-x -rw-r--r-- 0 2012/12/17 15:17:52 f3 drwxr-xr-x4096 2012/12/17 15:17:40 d1 drwxr-xr-x4096 2012/12/17 15:17:40 d2-x drwxr-xr-x4096 2012/12/17 15:17:40 d2 drwxr-xr-x4096 2012/12/17 15:17:40 d3 Note that the files are listed in a 'sensible' order, with f2-x coming after f2. But for the directories, d2-x comes before d2. Is this a bug or a feature? (I'm running rsync 3.0.9 on 64-bit Arch Linux.) And more importantly, can I rely on the output staying the same in the future? cheers Chris -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --list-only ordering
On 17/12/12 17:09, Paul Slootman wrote: On Mon 17 Dec 2012, Chris Dennis wrote: I've noticed an apparent inconsistency in the ordering of output from the --list-only option. For example: $ ls d1 d2 d2-x d3 f1 f2 f2-x f3 $ rsync --list-only . drwxr-xr-x4096 2012/12/17 15:18:05 . -rw-r--r-- 0 2012/12/17 15:17:52 f1 -rw-r--r-- 0 2012/12/17 15:17:52 f2 -rw-r--r-- 0 2012/12/17 15:17:52 f2-x -rw-r--r-- 0 2012/12/17 15:17:52 f3 drwxr-xr-x4096 2012/12/17 15:17:40 d1 drwxr-xr-x4096 2012/12/17 15:17:40 d2-x drwxr-xr-x4096 2012/12/17 15:17:40 d2 drwxr-xr-x4096 2012/12/17 15:17:40 d3 Note that the files are listed in a 'sensible' order, with f2-x coming after f2. But for the directories, d2-x comes before d2. I wouldn't be surprised if rsync appends a slash to the directory names internally; '-' sorts before '/' in ASCII. That would explain it. I just hope that the behaviour doesn't change. Thanks for your help. Chris -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Brandysnap -- a new rsync-based snapshot management script
On 04/07/11 02:24, Henri Shustak wrote: Hi Chris, https://github.com/StarsoftAnalysis/brandysnap I am involved with the LBackup project. Would you be okay with a link being generated to the Brandy on github page from the LBackup alternativeshttp://www.lbackup.org/alternatives page? Yes, that would be great. Depending upon the license you release Brandysnap under, perhaps in the future Brandysnap functionality could be incorporated into the LBackup project? It's GPL'd -- share and enjoy! cheers Chris -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Brandysnap -- a new rsync-based snapshot management script
Brandysnap is an rsync-based script with a difference. Unlike dirvish, it does not assign importance to snapshots when they are created. All snapshots are created equal, and then they are managed so that the required number of old snapshots is maintained. Unlike rsnapshot, brandysnap does not get its snapshots out of sync if runs are missed, and it is very easy to configure. The link between brandysnap and cron is very simple: just make sure brandysnap is run often enough to create the first level of snapshots. There is no need for separate runs to rotate the snapshot names at the required intervals. Defining which snapshots to keep is simple and intuitive. For example 4d7,7w3,4m11,3y means 'keep 4 snapshots a day for the last 7 days, 7 a week for the 3 weeks before that, then 4 a month for 11 months, then 3 a year forever'. This example is referred to in the explanations below. Such specifications can be as simple or as complex as you like. Brandysnap is flexible. It can be ultra-cautious, refusing to delete old snapshots unless there are enough to completely fill the requested schedule. Or it can be more relaxed, and apply the rules more simply. The periods 'day', 'week', 'month', etc. can be aligned on real calendar periods, or they can be considered as simple spans of time working backwards from 'now'. Brandysnap is effectively stateless: it does not keep any sort of list or database of information about previous runs. It uses the existing snapshots as the basis for deciding what needs to be done. The snapshots it creates are simple directories, which can be accessed without any special tools. Brandysnap is a Perl script designed to run on Linux. If there is demand, I hope to make it work on other operating systems too. It uses rsync (http://rsync.samba.org/) to do all the hard work, including the network and authentication stuff. Most of the cleverness in brandysnap is in deciding which snapshots to keep. As time passes, existing snapshots move into the realm of later specs, which require fewer snapshots to be kept. For example, after a week of keeping 4 snapshots a day, only 7 snapshots out of the existing 28 may still be required. Brandysnap chooses 7 that are spread out across the week to give the most useful set for future recovery. But, and this is the important bit, if runs have been missed for any reason, brandysnap will skip over incomplete periods. For example, if the destination is an external USB drive and someone forgot to plug it in, there will be snapshots missing. Brandysnap does not assume that a '4d7' spec has been completed after 7 days: it only does so when it has found 7 days each with at least 4 snapshots. Only then does it start looking for weeks with at least 7 snapshots. If you don't intend to create snapshots at weekends, specify something like 4d5,7w3,4m11,3y and brandysnap will automatically skip over the weekend days. There are options to fine-tune the way brandysnap behaves. The output includes an analysis of the existing snapshots and the way snapshots have been chosen to be kept or deleted. At present (3 July 2011), brandysnap is still being developed and should be considered 'experimental'. Do not use it for anything important (yet). Brandysnap can be downloaded from GitHub at https://github.com/StarsoftAnalysis/brandysnap where you can follow its development and read more details. Hopefully it will prove useful. I look forward to hearing comments and ideas. cheers Chris -- Chris Denniscgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: keep 2 dirs in sync
On 01/07/11 17:44, Paul Slootman wrote: On Fri 01 Jul 2011, Michael Makuch wrote: I have two hosts (my portable and my desktop). I work on both hosts at different times and so I keep a few dirs sync'd between the two. I have a docs dir where I may be modifying files, adding files, renaming files and deleting files on *either* host. You may want to investigate unison, which is designed for such usage. Or syncany -- www.syncany.org cheers Chris -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync 3.0.7 hangs with unreadable hard-links files
Hello rsync list I've stumbled across a fairly obscure situation in which rsync hangs -- it just waits until I press ctrl-C. Here's a transcript. Note that directory foo contains two small files that are hard-linked together, and are unreadable. -- $ uname -a Linux ferox 2.6.35-28-generic-pae #50-Ubuntu SMP Fri Mar 18 20:43:15 UTC 2011 i686 GNU/Linux $ rsync --version rsync version 3.0.7 protocol version 30 Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, iconv, symtimes rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public Licence for details. $ ls -dl foo drwxr-xr-x 2 chris chris 4096 2011-06-09 23:23 foo $ ls -il foo total 8 15958048 --w--- 2 chris chris 193 2011-05-25 19:19 bd 15958048 --w--- 2 chris chris 193 2011-05-25 19:19 bd2 $ rsync -avv --hard-links foo /tmp sending incremental file list delta-transmission disabled for local transfer or --whole-file rsync: send_files failed to open /home/chris/Dropbox/perlwork/brandysnap/foo/bd2: Permission denied (13) ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(543) [sender=3.0.7] -- If this is not a known bug, then hopefully this information will be of use. regards Chris -- Chris Dennis cgden...@btinternet.com Fordingbridge, Hampshire, UK -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
