Re: Disabling "quick check"
Hi: I just wanted to thank you for your help. I was able to get a comprehensive list using the checksum switch. To summarize I recovered my data with: -acvzIi --no-o --no-g --no-p -Clint On Wed, Oct 28, 2015 at 10:22 AM, Kevin Korb <k...@sanitarium.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > if you see >f it is doing something to the file. At least a > delta-xfer. If it was just a metadata change it would show cf. If > you see an >fc without a t then that is an example where rsync found a > file that didn't match even though the timestamps did. That isn't > supposed to happen very often. > > On 10/28/2015 01:19 PM, Clint Olsen wrote: > > Ok, thank you for this extra info. I have experienced exactly what > > you described. The rsync dry run is _still_ running after being > > started at 1:30am PST :) > > > > But it is finding the right files to update. Most of the entries > > are: > > > >> fc > > > > Which is what I want. > > > > So, just because I see: > > > >> f > > > > at the beginning... > > > > That doesn't necessarily mean that the file is going to get updated > > at the destination? In other words, you're saying that just doing > > the delta transfer is more time efficient and rsync won't touch the > > file unless _some_ relevant change is observed? It just concerned > > me because the file list was extensive, and I definitely don't want > > anything copied unless for example the checksums don't match. > > > > Thanks, > > > > -Clint > > > > > > On Wed, Oct 28, 2015 at 5:41 AM, Kevin Korb <k...@sanitarium.net > > <mailto:k...@sanitarium.net>> wrote: > > > > --checksum generally takes a lot longer than --size-only. A delta > > transfer generally goes quicker than a checksum. However, if you > > want to make a list of what is corrupt a checksumming utility that > > is less stupid than rsync can be useful. I say that because > > rsync's --checksum is entirely unintelligent. It will checksum > > every single file on both ends including files that only exist on > > one end and files that shouldn't match (the size is wrong). At the > > end of --checksum you will still have to do the delta xfers that > > you would do without it. > > > > OTOH, you are using --dry-run. If you are trying to generate a > > report about what files are corrupted then only --checksum an do > > that. It will just do it in the dumbest/slowest way possible. > > > > On 10/28/2015 02:08 AM, Clint Olsen wrote: > >> What about -c? It seems I'm getting a lot of spurious file > >> transfer candidates when using: > > > >> -avvznIi --no-o --no-g --no-p > > > >> It's showing transfers (receive) for many files I know haven't > >> been tampered with. > > > >> Thanks, > > > >> -Clint > > > >> On Tue, Oct 27, 2015 at 7:53 PM, Kevin Korb <k...@sanitarium.net > >> <mailto:k...@sanitarium.net> <mailto:k...@sanitarium.net > >> <mailto:k...@sanitarium.net>>> wrote: > > > >> That is correct. Rsync will re-copy (delta xfer unless > >> --wholefile) all the files. You can always --dry-run if you > >> aren't sure. > > > >> On 10/27/2015 10:07 PM, Clint Olsen wrote: > >>> Hi: > > > >>> I've been using rsync to create backups for a few years. A few > >>> months ago I started experiencing sector errors. I ended up > >>> replacing the drive and copying the drive data. It turns out > >>> that due to the default behavior of rsync "quick check", some > >>> of the files were modified without altering the modification > >>> time or size, so these files are still clean in the backup. I > >>> would like to recover these files, but I need to defeat the > >>> quick check in order to do this. It looks like using: > > > >>> -I, --ignore-times don't skip files that match size > >>> and time > > > >>> will work. I just want to confirm that this covers it. The > >>> long-form of the switch doesn't mention size, so I was > >>> concerned this was the right way to accomplish this. If there > >>> are any other switches that would be useful in the context of > >>> potentially corrupt files, please feel free to chime in... > > > >>> Thanks, > > > >>> -Clint > > > > > > > > > > > >> -- Please use
Re: Disabling "quick check"
What about -c? It seems I'm getting a lot of spurious file transfer candidates when using: -avvznIi --no-o --no-g --no-p It's showing transfers (receive) for many files I know haven't been tampered with. Thanks, -Clint On Tue, Oct 27, 2015 at 7:53 PM, Kevin Korb <k...@sanitarium.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > That is correct. Rsync will re-copy (delta xfer unless --wholefile) > all the files. You can always --dry-run if you aren't sure. > > On 10/27/2015 10:07 PM, Clint Olsen wrote: > > Hi: > > > > I've been using rsync to create backups for a few years. A few > > months ago I started experiencing sector errors. I ended up > > replacing the drive and copying the drive data. It turns out that > > due to the default behavior of rsync "quick check", some of the > > files were modified without altering the modification time or size, > > so these files are still clean in the backup. I would like to > > recover these files, but I need to defeat the quick check in order > > to do this. It looks like using: > > > > -I, --ignore-times don't skip files that match size and > > time > > > > will work. I just want to confirm that this covers it. The > > long-form of the switch doesn't mention size, so I was concerned > > this was the right way to accomplish this. If there are any other > > switches that would be useful in the context of potentially corrupt > > files, please feel free to chime in... > > > > Thanks, > > > > -Clint > > > > > > > > - -- > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > - -*~ > Kevin Korb Phone:(407) 252-6853 > Systems Administrator Internet: > FutureQuest, Inc. ke...@futurequest.net (work) > Orlando, Floridak...@sanitarium.net (personal) > Web page: http://www.sanitarium.net/ > PGP public key available on web site. > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > - -*~ > -BEGIN PGP SIGNATURE- > Version: GnuPG v2 > > iEYEARECAAYFAlYwOLkACgkQVKC1jlbQAQcvhgCgsD1NcaUijxcovEAdBNnCvqeB > KZsAmwY8f73FoUY5cen2OmA7JlzE2Nml > =auhS > -END PGP SIGNATURE- > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html > -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Disabling "quick check"
Ok, thank you for this extra info. I have experienced exactly what you described. The rsync dry run is _still_ running after being started at 1:30am PST :) But it is finding the right files to update. Most of the entries are: >fc Which is what I want. So, just because I see: >f at the beginning... That doesn't necessarily mean that the file is going to get updated at the destination? In other words, you're saying that just doing the delta transfer is more time efficient and rsync won't touch the file unless _some_ relevant change is observed? It just concerned me because the file list was extensive, and I definitely don't want anything copied unless for example the checksums don't match. Thanks, -Clint On Wed, Oct 28, 2015 at 5:41 AM, Kevin Korb <k...@sanitarium.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > - --checksum generally takes a lot longer than --size-only. A delta > transfer generally goes quicker than a checksum. However, if you want > to make a list of what is corrupt a checksumming utility that is less > stupid than rsync can be useful. I say that because rsync's > - --checksum is entirely unintelligent. It will checksum every single > file on both ends including files that only exist on one end and files > that shouldn't match (the size is wrong). At the end of --checksum > you will still have to do the delta xfers that you would do without it. > > OTOH, you are using --dry-run. If you are trying to generate a report > about what files are corrupted then only --checksum an do that. It > will just do it in the dumbest/slowest way possible. > > On 10/28/2015 02:08 AM, Clint Olsen wrote: > > What about -c? It seems I'm getting a lot of spurious file > > transfer candidates when using: > > > > -avvznIi --no-o --no-g --no-p > > > > It's showing transfers (receive) for many files I know haven't > > been tampered with. > > > > Thanks, > > > > -Clint > > > > On Tue, Oct 27, 2015 at 7:53 PM, Kevin Korb <k...@sanitarium.net > > <mailto:k...@sanitarium.net>> wrote: > > > > That is correct. Rsync will re-copy (delta xfer unless > > --wholefile) all the files. You can always --dry-run if you aren't > > sure. > > > > On 10/27/2015 10:07 PM, Clint Olsen wrote: > >> Hi: > > > >> I've been using rsync to create backups for a few years. A few > >> months ago I started experiencing sector errors. I ended up > >> replacing the drive and copying the drive data. It turns out > >> that due to the default behavior of rsync "quick check", some of > >> the files were modified without altering the modification time or > >> size, so these files are still clean in the backup. I would like > >> to recover these files, but I need to defeat the quick check in > >> order to do this. It looks like using: > > > >> -I, --ignore-times don't skip files that match size and > >> time > > > >> will work. I just want to confirm that this covers it. The > >> long-form of the switch doesn't mention size, so I was concerned > >> this was the right way to accomplish this. If there are any > >> other switches that would be useful in the context of potentially > >> corrupt files, please feel free to chime in... > > > >> Thanks, > > > >> -Clint > > > > > > > > > > > > -- Please use reply-all for most replies to avoid omitting the > > mailing list. To unsubscribe or change options: > > https://lists.samba.org/mailman/listinfo/rsync Before posting, > > read: http://www.catb.org/~esr/faqs/smart-questions.html > > > > > > > > > > - -- > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > - -*~ > Kevin Korb Phone:(407) 252-6853 > Systems Administrator Internet: > FutureQuest, Inc. ke...@futurequest.net (work) > Orlando, Floridak...@sanitarium.net (personal) > Web page: http://www.sanitarium.net/ > PGP public key available on web site. > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > - -*~ > -BEGIN PGP SIGNATURE- > Version: GnuPG v2 > > iEYEARECAAYFAlYwwnoACgkQVKC1jlbQAQfHMACfXgY160iVto4Sm5tN7gv+QL39 > 8DsAn3hvPbx3kunv1DIlC9BRXdSVD9fn > =xDso > -END PGP SIGNATURE- > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html > -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Disabling "quick check"
Hi: I've been using rsync to create backups for a few years. A few months ago I started experiencing sector errors. I ended up replacing the drive and copying the drive data. It turns out that due to the default behavior of rsync "quick check", some of the files were modified without altering the modification time or size, so these files are still clean in the backup. I would like to recover these files, but I need to defeat the quick check in order to do this. It looks like using: -I, --ignore-times don't skip files that match size and time will work. I just want to confirm that this covers it. The long-form of the switch doesn't mention size, so I was concerned this was the right way to accomplish this. If there are any other switches that would be useful in the context of potentially corrupt files, please feel free to chime in... Thanks, -Clint -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --link-dest won't link even if existing file is out of date
Not to mention the fact that ZFS requires considerable hardware resources (CPU memory) to perform well. It also requires you to learn a whole new terminology to wrap your head around it. It's certainly not a trivial swap to say the least... Thanks, -Clint On Mon, Apr 6, 2015 at 9:12 AM, Ken Chase rsync-list-m...@sizone.org wrote: This has been a consideration. But it pains me that a tiny change/addition to the rsync option set would save much time and space for other legit use cases. We know rsync very well, we dont know ZFS very well (licensing kept the tech out of our linux-centric operations). We've been using it but we're not experts yet. Thanks for the suggestion. /kc On Mon, Apr 06, 2015 at 12:07:05PM -0400, Kevin Korb said: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Since you are in an environment with millions of files I highly recommend that you move to ZFS storage and use ZFS's subvolume snapshots instead of --link-dest. It is much more space efficient, rsync run time efficient, and the old backups can be deleted in seconds. Rsync doesn't have to understand anything about ZFS. You just rsync to the same directory every time and have ZFS do a snapshot on that directory between runs. On 04/06/2015 01:51 AM, Ken Chase wrote: Feature request: allow --link-dest dir to be linked to even if file exists in target. This statement from the man page is adhered to too strongly IMHO: This option works best when copying into an empty destination hierarchy, as rsync treats existing files as definitive (so it never looks in the link-dest dirs when a destination file already exists). I was suprised by this behaviour as generally the scheme is to be efficient/save space with rsync. When the file is out of date but exists in the --l-d target, it would be great if it could be removed and linked. If an option was supplied to request this behaviour, I'd actually throw some money at making it happen. (And a further option to retain a copy if inode permissions/ownership would otherwise be changed.) Reasoning: I backup many servers with --link-dest that have filesystems of 10+M files on them. I do not delete old backups - which take 60min per tree or more just so rsync can recreate them all in an empty target dir when 1% of files change per day (takes 3-5 hrs per backup!). Instead, I cycle them in with mv $olddate $today then rsync --del --link-dest over them - takes 30-60 min depending. (Yes, some malleability of permissions risk there, mostly interested in contents tho). Problem is, if a file exists AT ALL, even out of date, a new copy is put overtop of it per the above man page decree. Thus much more disk space is used. Running this scheme with moving old backups to be written overtop of accumulates many copies of the exact same file over time. Running pax -rpl over the copies before rsyncing to them works (and saves much space!), but takes a very long time as it traverses and compares 2 large backup trees thrashing the same device (in the order of 3-5x the rsync's time, 3-5 hrs for pax - hardlink(1) is far worse, I suspect a some non-linear algorithm therein - it ran 3-5x slower than pax again). I have detailed an example of this scenario at http://unix.stackexchange.com/questions/193308/rsyncs-link-dest-option-does-not-link-identical-files-if-an-old-file-exists which also indicates --delete-before and --whole-file do not help at all. /kc - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2 iEYEARECAAYFAlUirykACgkQVKC1jlbQAQc83ACfa7lawkyPFyO9kDE/D8aztql0 AkAAoIQ970yTCHB1ypScQ8ILIQR6zphl =ktEg -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Ken Chase - ken att heavycomputing.ca Toronto Canada Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read:
Re: Link-dest thinks file is newly created, but it isn't
I suspect that I've found the problem. It appears that the permissions for these directories have been turned off at some point for these directories. This might be causing the problem with --link-dest. When it tried to stat the file on the backup it couldn't and therefore did not exist. Although, you would expect if you ignored owner, group, and perms that it would just default to those permissions on the target volume. d-+ 1 nancy None 0 Jul 8 2013 0313BookshelvesCSD drwx--+ 1 nancy None 0 Dec 27 2013 0413SpringChorusCSD d-+ 1 nancy None 0 Jul 8 2013 0513CapeCodCSD -rwx-- 1 nancy None 114309469 Nov 13 2013 0612BonAppetitCSD.zip -rwx-- 1 nancy None 85356572 Nov 13 2013 0612BonAppetitCSD_QDDL.zip -rwx-- 1 nancy None 5893538 Nov 13 2013 0612BonAppetitCSD_Templates.zip d-+ 1 nancy None 0 Jul 8 2013 0613HopesCSD drwx--+ 1 nancy None 0 Dec 27 2013 0913TakeWingCSD drwx--+ 1 nancy None 0 Dec 27 2013 1013LockKeyCSD Thanks, -Clint On Sat, Jan 10, 2015 at 6:19 PM, Kevin Korb k...@sanitarium.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 If it seeing the files as new then I agree that stat won't help. It might have explained some other itemize output. Since it is seeing the files as new then they must not be where it is looking. Meaning that your link-dest parameter must not be appropriate for your target. On 01/10/2015 08:49 PM, Clint Olsen wrote: On Sat Jan 10 2015 at 5:21:33 AM Kevin Korb k...@sanitarium.net mailto:k...@sanitarium.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 What does --itemize-changes say about that file? Try using the stat command on the various copies of it to see what is different about them. In my original message, I stated I used --itemize-changes, and I reported the following: f+ nancy/Documents/SCRAPPING/__Digital/Club Scrap/0313BookshelvesCSD/__0313BookshelvesCSD.zip Are there other debug switches I should consider? I don't see why stat'ing the file will matter considering that rsync thinks this file is new. When I run with a command like the following: /usr/bin/rsync -avz /--itemize-changes /--dry-run --no-o --no-g --no-p --exclude-from=/cygdrive/c/__Users/clint/.rsync/exclude --delete --delete-excluded --link-dest=../latest /cygdrive/c/Users/clint /cygdrive/c/Users/nancy rsync@nas::NetBackup/sith/__2015-01-09T11_58_16 Rsync comes back and tells me this: f+ nancy/Documents/SCRAPPING/__Digital/Club Scrap/0313BookshelvesCSD/__0313BookshelvesCSD.zip Thanks, -Clint - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2 iEYEARECAAYFAlSx3cgACgkQVKC1jlbQAQf0UACg2XQS42aTwLPv0xbEmQQalbDL TEQAoOTSbhtM4p3wj/nrxROoy85O1GRa =lB3o -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Link-dest thinks file is newly created, but it isn't
On Sat Jan 10 2015 at 5:21:33 AM Kevin Korb k...@sanitarium.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 What does --itemize-changes say about that file? Try using the stat command on the various copies of it to see what is different about them. In my original message, I stated I used --itemize-changes, and I reported the following: f+ nancy/Documents/SCRAPPING/Digital/Club Scrap/0313BookshelvesCSD/0313BookshelvesCSD.zip Are there other debug switches I should consider? I don't see why stat'ing the file will matter considering that rsync thinks this file is new. When I run with a command like the following: /usr/bin/rsync -avz *--itemize-changes *--dry-run --no-o --no-g --no-p --exclude-from=/cygdrive/c/Users/clint/.rsync/exclude --delete --delete-excluded --link-dest=../latest /cygdrive/c/Users/clint /cygdrive/c/Users/nancy rsync@nas::NetBackup/sith/2015-01-09T11_58_16 Rsync comes back and tells me this: f+ nancy/Documents/SCRAPPING/Digital/Club Scrap/0313BookshelvesCSD/0313BookshelvesCSD.zip Thanks, -Clint -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Link-dest thinks file is newly created, but it isn't
Hi: I've been using rsync for a couple years now. Unfortunately, I've made some changes on both ends, so it's unclear what could be the culprit. I make extensive use of --link-dest to provide a cheap Time Machine-like backup for a Windows machine. Source: Windows 7 running Cygwin (CYGWIN_NT-6.1 sith 1.7.33-2(0.280/5/3) 2014-11-13 15:47 x86_64 Cygwin) Destination: Synology 1512+ (Linux NAS 3.2.40 #5021 SMP Wed Dec 17 18:30:52 CST 2014 x86_64 GNU/Linux synology_cedarview_1512+) When I run with a command like the following: /usr/bin/rsync -avz --itemize-changes --dry-run --no-o --no-g --no-p --exclude-from=/cygdrive/c/Users/clint/.rsync/exclude --delete --delete-excluded --link-dest=../latest /cygdrive/c/Users/clint /cygdrive/c/Users/nancy rsync@nas::NetBackup/sith/2015-01-09T11_58_16 Rsync comes back and tells me this: f+ nancy/Documents/SCRAPPING/Digital/Club Scrap/0313BookshelvesCSD/0313BookshelvesCSD.zip However, the file is clearly there. Here's a bit of history of the file (with inodes listed): 301113395 -rwx-- 329 rsync users 205949262 Jun 26 2013 2014-12-29T02_00_02/nancy/Documents/SCRAPPING/Digital/Club Scrap/0313BookshelvesCSD/0313BookshelvesCSD.zip 44883985 -rwx-- 1 rsync users 205949262 Jun 26 2013 2015-01-05T10_17_47/nancy/Documents/SCRAPPING/Digital/Club Scrap/0313BookshelvesCSD/0313BookshelvesCSD.zip 24985621 -rwx-- 1 rsync users 205949262 Jun 26 2013 2015-01-06T02_00_02/nancy/Documents/SCRAPPING/Digital/Club Scrap/0313BookshelvesCSD/0313BookshelvesCSD.zip 13205531 -rwx-- 1 rsync users 205949262 Jun 26 2013 2015-01-07T02_00_01/nancy/Documents/SCRAPPING/Digital/Club Scrap/0313BookshelvesCSD/0313BookshelvesCSD.zip 56131615 -rwx-- 1 rsync users 205949262 Jun 26 2013 2015-01-08T02_00_02/nancy/Documents/SCRAPPING/Digital/Club Scrap/0313BookshelvesCSD/0313BookshelvesCSD.zip 30498850 -rwx-- 1 rsync users 205949262 Jun 26 2013 2015-01-09T02_00_02/nancy/Documents/SCRAPPING/Digital/Club Scrap/0313BookshelvesCSD/0313BookshelvesCSD.zip 30498850 -rwx-- 1 rsync users 205949262 Jun 26 2013 latest/nancy/Documents/SCRAPPING/Digital/Club Scrap/0313BookshelvesCSD/0313BookshelvesCSD.zip I'm really perplexed as to how rsync thinks this file is new. The checksums seem to match as well. Thanks, -Clint -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Rsync like Time Machine
I've been very interested in these discussions and uses of rsync as a clone of Time Machine. A couple of things have been keeping me from a fully automated solution. I'd like to eliminate the need for Samba/NFS mounts of any kind, because they have proven to be unreliable for me and under some operating environments (Cygwin) it breaks --link-dest. In most of the articles I've read, a target date directory is created with some sort of latest symlink for the --link-dest parameter. I can accomplish those tasks via remote ssh commands, but I was hoping there was a better way. For example, is there any circumstance where you can coax rsync into creating a target directory that's not there already? % rsync source user@nas::module/somedir-exists/newdir-with-date So, newdir-with-date doesn't exist (yet). I would like to have rsync create it for me. Is that even possible? Thanks, -Clint -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync like Time Machine
Ok, that is helpful. As you can guess based on my question, it would be nice if all the automation can be done on the client side rather than having some specialized scripting on the receiving side to manage directories and symlinks etc. Thanks, -Clint On Fri, Jul 27, 2012 at 12:55 PM, Greg Deback (rsync) greg.deb+rs...@gmail.com wrote: Hi, As for the destination directory and the backup directory (--backup-dir), rsync will create the missing subdirectory (one level below the existing dir only), so yes for /somedir-exists/newdir-with-date, no for /somedir-exists/newdir-with-year/newdir-with-month on january 1st... But if you want this dir to be a symlink, you can't. Greg On Fri, Jul 27, 2012 at 7:16 PM, Clint Olsen clint.ol...@gmail.comwrote: I've been very interested in these discussions and uses of rsync as a clone of Time Machine. A couple of things have been keeping me from a fully automated solution. I'd like to eliminate the need for Samba/NFS mounts of any kind, because they have proven to be unreliable for me and under some operating environments (Cygwin) it breaks --link-dest. In most of the articles I've read, a target date directory is created with some sort of latest symlink for the --link-dest parameter. I can accomplish those tasks via remote ssh commands, but I was hoping there was a better way. For example, is there any circumstance where you can coax rsync into creating a target directory that's not there already? % rsync source user@nas::module/somedir-exists/newdir-with-date So, newdir-with-date doesn't exist (yet). I would like to have rsync create it for me. Is that even possible? Thanks, -Clint -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Rsync doesn't seem to be able to preserve permission bits/flags in Cygwin
Hi: I'm using the following arguments to rsync when backing up, and yet the flags are not preserved when I copy: % rsync -aP options deleted Nancy@macki /cygdrive/c/Documents and Settings/Nancy/My Documents $ ls -al z1.jpg -rwxr-xr-x+ 1 Nancy None 118258 Aug 29 2011 z1.jpg Nancy@macki /cygdrive/f/macki/back-2012-06-04T16:57:38/Nancy/My Documents $ ls -al z1.jpg -rw-r--r-- 1 Nancy None 118258 Aug 29 2011 z1.jpg I was wondering if this is common with Cygwin. The NAS is connected using CIFS. Thanks, -Clint -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Fwd: --link-dest does not appear to be linking on Cygwin
Hi: I have attempted to following some instructions to use --link-dest in order to preserve space for multiple backups. I'm using rsync on Cygwin with a NAS (ext4) which does support hard-links on the filesystem. I've written a short program that does attempt to create a hard-link on this NAS from Cygwin and it does look to be working. If I run ls -li on the NAS the inodes are the same. However, on Cygwin the inodes returned are not. I assumed that what's important is that the NAS handles the hard-links correctly. I checked the archives and it looks like a common mistake is inconsistent usage of the trailing backslash. My usage looks something like: % rsync -aP --delete --delete-excluded --exclude-from=/home/Nancy/.rsync/exclude --link-dest=/cygdrive/f/macki/current/ Nancy /cygdrive/f/macki/incomplete-back-2012-06-05T13:53:51 I'm running this from /cygwin/c/Documents and Settings/ directory, and the directory under the symlink current is Nancy. Is there a way to tell if rsync thinks it's found a match and attempting to link? Thanks, -Clint -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Fwd: --link-dest does not appear to be linking on Cygwin
Hi:
I will try the itemize switch as you suggested, thanks...
I have tested this on both the Windows box and the NAS. The inode
numbers come back differently on Cygwin but the NAS sees them as the
same:
Nancy@macki /cygdrive/f/macki/test
$ ls -ali
total 1048576
3297336581512553257 drwxr-xr-x 1 Nancy None 0 Jun 5 11:37 .
16963795924520120485 drwxr-xr-x 1 Nancy None 0 Jun 5 14:07 ..
5321094878451953619 -rw-r--r-- 1 Nancy None 4 Jun 5 10:57 foo
9211568837569774878 -rw-r--r-- 1 Nancy None 4 Jun 5 10:57 foo1
9211568837569774879 -rw-r--r-- 1 Nancy None 4 Jun 5 10:57 foo2
3595967447105090765 -rw-r--r-- 1 Nancy None 4 Jun 5 10:57 foolink
NAS cd /volume1/Backup/macki/test/
NAS ls -ali
16678913 drwxrwxrwx2 nancyusers 4096 Jun 5 11:37 .
8814593 drwxrwxrwx5 nancyusers 4096 Jun 5 14:07 ..
16678915 -rwxrwxrwx4 nancyusers4 Jun 5 10:57 foo
16678915 -rwxrwxrwx4 nancyusers4 Jun 5 10:57 foo1
16678915 -rwxrwxrwx4 nancyusers4 Jun 5 10:57 foo2
16678915 -rwxrwxrwx4 nancyusers4 Jun 5 10:57 foolink
The final file foolink was created using a C program ala:
$ cat link.c
#include unistd.h
#include stdio.h
int main(void)
{
if (link(/cygdrive/f/macki/test/foo,
/cygdrive/f/macki/test/foolink) == 0) {
printf(Link was successful\n);
} else {
printf(Link was not successful\n);
}
return 0;
}
Thanks,
-Clint
On Tue, Jun 5, 2012 at 2:06 PM, Kevin Korb k...@sanitarium.net wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
- --itemize-changes would be the first place to start. Also, see if you
can make a hard link on your NAS. Just because the underlying ext4
supports it doesn't mean that your network mount does. Try something
like:
cd /cygdrive/f/macki
touch testfile
ln testfile testfile2
ls -ls testfile testfile2
They should both have the same inode number.
On 06/05/12 17:02, Clint Olsen wrote:
Hi:
I have attempted to following some instructions to use --link-dest
in order to preserve space for multiple backups. I'm using rsync
on Cygwin with a NAS (ext4) which does support hard-links on the
filesystem. I've written a short program that does attempt to
create a hard-link on this NAS from Cygwin and it does look to be
working. If I run ls -li on the NAS the inodes are the same.
However, on Cygwin the inodes returned are not. I assumed that
what's important is that the NAS handles the hard-links correctly.
I checked the archives and it looks like a common mistake is
inconsistent usage of the trailing backslash. My usage looks
something like:
% rsync -aP --delete --delete-excluded
--exclude-from=/home/Nancy/.rsync/exclude
--link-dest=/cygdrive/f/macki/current/ Nancy
/cygdrive/f/macki/incomplete-back-2012-06-05T13:53:51
I'm running this from /cygwin/c/Documents and Settings/
directory, and the directory under the symlink current is
Nancy.
Is there a way to tell if rsync thinks it's found a match and
attempting to link?
Thanks,
-Clint
- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Florida k...@sanitarium.net (personal)
Web page: http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk/OdL4ACgkQVKC1jlbQAQe9EgCgqYVuiaHi0MTD6MZjISt539aF
7SMAoOJu8y508ZGMySbqgkch6Mellg3g
=DBkr
-END PGP SIGNATURE-
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Fwd: --link-dest does not appear to be linking on Cygwin
Ok, itemize changes prints out some info for each file, and it looks like:
cf...p. Nancy/Favorites/Links/desktop.ini
cf...p. Nancy/Favorites/Links/latin music.url
cd...p. Nancy/Favorites/Microsoft Websites/
I did a quick search on this switch and someone was bitching that the
fields were not easy to decipher w/o reading the source code :)
'P' would appear to be permissions. I do get a couple of
warnings/errors about not being able to preserve all attributes during
the copy, but the permissions should not be changing...
Thanks,
-Clint
On Tue, Jun 5, 2012 at 2:10 PM, Clint Olsen clint.ol...@gmail.com wrote:
Hi:
I will try the itemize switch as you suggested, thanks...
I have tested this on both the Windows box and the NAS. The inode
numbers come back differently on Cygwin but the NAS sees them as the
same:
Nancy@macki /cygdrive/f/macki/test
$ ls -ali
total 1048576
3297336581512553257 drwxr-xr-x 1 Nancy None 0 Jun 5 11:37 .
16963795924520120485 drwxr-xr-x 1 Nancy None 0 Jun 5 14:07 ..
5321094878451953619 -rw-r--r-- 1 Nancy None 4 Jun 5 10:57 foo
9211568837569774878 -rw-r--r-- 1 Nancy None 4 Jun 5 10:57 foo1
9211568837569774879 -rw-r--r-- 1 Nancy None 4 Jun 5 10:57 foo2
3595967447105090765 -rw-r--r-- 1 Nancy None 4 Jun 5 10:57 foolink
NAS cd /volume1/Backup/macki/test/
NAS ls -ali
16678913 drwxrwxrwx 2 nancy users 4096 Jun 5 11:37 .
8814593 drwxrwxrwx 5 nancy users 4096 Jun 5 14:07 ..
16678915 -rwxrwxrwx 4 nancy users 4 Jun 5 10:57 foo
16678915 -rwxrwxrwx 4 nancy users 4 Jun 5 10:57 foo1
16678915 -rwxrwxrwx 4 nancy users 4 Jun 5 10:57 foo2
16678915 -rwxrwxrwx 4 nancy users 4 Jun 5 10:57 foolink
The final file foolink was created using a C program ala:
$ cat link.c
#include unistd.h
#include stdio.h
int main(void)
{
if (link(/cygdrive/f/macki/test/foo,
/cygdrive/f/macki/test/foolink) == 0) {
printf(Link was successful\n);
} else {
printf(Link was not successful\n);
}
return 0;
}
Thanks,
-Clint
On Tue, Jun 5, 2012 at 2:06 PM, Kevin Korb k...@sanitarium.net wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
- --itemize-changes would be the first place to start. Also, see if you
can make a hard link on your NAS. Just because the underlying ext4
supports it doesn't mean that your network mount does. Try something
like:
cd /cygdrive/f/macki
touch testfile
ln testfile testfile2
ls -ls testfile testfile2
They should both have the same inode number.
On 06/05/12 17:02, Clint Olsen wrote:
Hi:
I have attempted to following some instructions to use --link-dest
in order to preserve space for multiple backups. I'm using rsync
on Cygwin with a NAS (ext4) which does support hard-links on the
filesystem. I've written a short program that does attempt to
create a hard-link on this NAS from Cygwin and it does look to be
working. If I run ls -li on the NAS the inodes are the same.
However, on Cygwin the inodes returned are not. I assumed that
what's important is that the NAS handles the hard-links correctly.
I checked the archives and it looks like a common mistake is
inconsistent usage of the trailing backslash. My usage looks
something like:
% rsync -aP --delete --delete-excluded
--exclude-from=/home/Nancy/.rsync/exclude
--link-dest=/cygdrive/f/macki/current/ Nancy
/cygdrive/f/macki/incomplete-back-2012-06-05T13:53:51
I'm running this from /cygwin/c/Documents and Settings/
directory, and the directory under the symlink current is
Nancy.
Is there a way to tell if rsync thinks it's found a match and
attempting to link?
Thanks,
-Clint
- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Florida k...@sanitarium.net (personal)
Web page: http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk/OdL4ACgkQVKC1jlbQAQe9EgCgqYVuiaHi0MTD6MZjISt539aF
7SMAoOJu8y508ZGMySbqgkch6Mellg3g
=DBkr
-END PGP SIGNATURE-
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
