Re: cut-off time for rsync ?
> >I used to rsync a /home with thousands of home directories every > >night, although only a hundred or so would be used on a typical day, > >and many of them have not been used for ages. This became too large a > >burden on the poor old destination server, so I switched to a script > >that uses "find -ctime -7" on the source to select recently used homes > >first, and then rsyncs only those. (A week being a more than good > >enough safety margin in case something goes wrong occasionally.) > > Doing it this way you can't delete files that have disappeared or been > renamed. > > >Is there a smarter way to do this, using rsync only ? I would like to > >use rsync with a cut-off time, saying "if a file is older than this, > >don't even bother checking it on the destination server (and the same > >for directories -- but without ending a recursive traversal)". Now > >I am traversing some directories twice on the source server to lighten > >the burden on the destination server (first find, then rsync). > > I would split up the tree into several sub trees and snyc them > normally, like /home/a* etc. You can then distribute the calls > over several days. If that is still too much then maybe to the > find call but then sync the whole user's home instead of just > the found files. As I did say in my original mail, but apparently did not emphasize sufficiently, rsyncing complete homes if anything changed in them is actually what I do; so files that have been deleted or renamed are handled correctly. Anyway, the first paragraph was just to provide some context: my real question is: can you specify a cut-off time using rsync only, meaning that files are ignored and directories are considered up to date on the destination server if they have not been touched for x days on the source ? Dirk van Deun -- Ceterum censeo Redmond delendum -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: cut-off time for rsync ?
> If your goal is to reduce storage, and scanning inodes doesnt matter, > use --link-dest for targets. However, that'll keep a backup for every > time that you run it, by link-desting yesterday's copy. The goal was not to reduce storage, it was to reduce work. A full rsync takes more than the whole night, and the destination server is almost unusable for anything else when it is doing its rsyncs. I am sorry if this was unclear. I just want to give rsync a hint that comparing files and directories that are older than one week on the source side is a waste of time and effort, as the rsync is done every day, so they can safely be assumed to be in sync already. Dirk van Deun -- Ceterum censeo Redmond delendum -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: cut-off time for rsync ?
> The goal was not to reduce storage, it was to reduce work. A full > rsync takes more than the whole night, and the destination server is > almost unusable for anything else when it is doing its rsyncs. I > am sorry if this was unclear. I just want to give rsync a hint that > comparing files and directories that are older than one week on > the source side is a waste of time and effort, as the rsync is done > every day, so they can safely be assumed to be in sync already. I thought something rang a bell ... >From the man page : >-I, --ignore-times > Normally rsync will skip any files that are already the > same size and have the same modification time-stamp. > This option turns off this "quick check" behavior, > causing all files to be updated. As I read this, the default is to look at the file size/timestamp and if they match then do nothing as they are assumed to be identical. So unless you have specified this, then files which have already been copied should be ignored - the check should be quite low in CPU, at least compared to the "cost" of generating a file checksum etc. AFAIK there is no option to completely ignore files by timestamp - at least not within rsync itself. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: cut-off time for rsync ?
What is taking time, scanning inodes on the destination, or recopying the entire backup because of either source read speed, target write speed or a slow interconnect between them? Do you keep a full new backup every day, or are you just overwriting the target directory? /kc On Wed, Jul 01, 2015 at 10:06:57AM +0200, Dirk van Deun said: >> If your goal is to reduce storage, and scanning inodes doesnt matter, >> use --link-dest for targets. However, that'll keep a backup for every >> time that you run it, by link-desting yesterday's copy. > >The goal was not to reduce storage, it was to reduce work. A full >rsync takes more than the whole night, and the destination server is >almost unusable for anything else when it is doing its rsyncs. I >am sorry if this was unclear. I just want to give rsync a hint that >comparing files and directories that are older than one week on >the source side is a waste of time and effort, as the rsync is done >every day, so they can safely be assumed to be in sync already. > >Dirk van Deun >-- >Ceterum censeo Redmond delendum -- Ken Chase - k...@heavycomputing.ca skype:kenchase23 +1 416 897 6284 Toronto Canada Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
