Re: Take2: rsync over ssh with --link-dest
Hi, rsync -avz --delete --numeric-ids --exclude-from=/backup/exclude-file.txt \ --link-dest=/backup/seg/serverA/xxx.xxx.228/cdrive/20091109 \ rsync://serverA/cdrive/20091110/ \ /backup/seg/serverA/xxx.xxx.228/cdrive/20091110 Your daemon setup looks exactly right. I don't understand why you consider it more complicated than the previous setup. Previously you had one SSH authorized key for each backup point with the appropriate Well, no, I didn't have that because it doesn't work. I've been using empty passphrase keys with command= for many years, but never really had a requirement to override the path in this way. In contrast, the forced-server-command approach would clobber the 20091110/ part, preventing you from setting up the --link-dest properly. Is it clear now? Yes, and hopefully I've clarified what I'm doing now too. :-) It seems to me that you would have had to do the same thing under the forced-server-command setup. No? Yep, exactly :-) In any case, sed is overkill to vary the date. You can just use a shell variable: DATE=$(date +%Y%m%d) rsync ... /backup/seg/serverA/xxx.xxx.228/cdrive/$DATE It's part of a larger backup script with its own configuration file, actually. Thanks so much. Best regards, Alex -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Take2: rsync over ssh with --link-dest
On Tue, 2009-11-10 at 22:47 -0500, Alex wrote: If the daemon solution is looking fundamentally more complicated than the set of forced server commands you previously had, you're probably going about it the wrong way and I could help if you post the rsyncd.conf file. Okay, great, it's included below. use chroot = no syslog facility = local5 [cdrive] list = true path = /backup/servers/xxx.xxx.228/cdrive comment = 228-cdrive read only = true gid = root uid = root [more modules like this...] The client side command looks something like this: rsync -avz --delete --numeric-ids --exclude-from=/backup/exclude-file.txt \ --link-dest=/backup/seg/serverA/xxx.xxx.228/cdrive/20091109 \ rsync://serverA/cdrive/20091110/ \ /backup/seg/serverA/xxx.xxx.228/cdrive/20091110 Your daemon setup looks exactly right. I don't understand why you consider it more complicated than the previous setup. Previously you had one SSH authorized key for each backup point with the appropriate path forced; now the daemon configuration contains the same information. The daemon configuration looks more verbose, but that could be helped by moving the parameters that are the same for all modules (list, ...) to the global section at the top, or you could even generate the configuration file with M4. The client command is the same except for the specification of a module name instead of an SSH key. Am I missing something? Realize that the ability to set a module's path in the rsyncd.conf file and have the client refer to the module by a well-known name gives you the same functionality, and actually more, because the client can specify a subdirectory path within the module. I'm not sure I understand fully, but this may be what I'm already doing. An example would be helpful. You are indeed doing this. Your source is: rsync://serverA/cdrive/20091110/ The rsync://serverA/cdrive/ is mapped to /backup/servers/xxx.xxx.228/cdrive according to the configuration file, resulting in: /backup/servers/xxx.xxx.228/cdrive/20091110/ In contrast, the forced-server-command approach would clobber the 20091110/ part, preventing you from setting up the --link-dest properly. Is it clear now? To make it even more complicated, on the receiving (client) side, I have to pass the backup script I use through sed to rewrite the date which gets appended to the source path, to update every day so I can use the --link-dest rsync argument to create a backup of only the changes from day-to-day, and hard link the rest from the reference location, which is usually the previous day. It seems to me that you would have had to do the same thing under the forced-server-command setup. No? In any case, sed is overkill to vary the date. You can just use a shell variable: DATE=$(date +%Y%m%d) rsync ... /backup/seg/serverA/xxx.xxx.228/cdrive/$DATE -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Take2: rsync over ssh with --link-dest
Hi, Overriding the server command line is generally a bad idea for the reasons you went on to note. Instead, use an rsync daemon. You can even have the daemon invoked over ssh with a forced command, see: https://bugzilla.samba.org/show_bug.cgi?id=4163#c4 That looks like the way to go. I've got it set up and running for a few days now, and it's working okay. I'm a bit disappointed that I can't do it by overriding the server command line, but it's okay. Thanks again, Alex -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Take2: rsync over ssh with --link-dest
I'm a bit disappointed that I can't do it by overriding the server command line Why? Does it have some advantage over the use of an rsync daemon (I can't imagine what)? One can't really expect overriding the server arguments not to lead to the inconsistent behavior you saw. Simplicity. I'd like to be able to have the server side feed me all data beginning with the path I specify, such /backup/serverA, and have the client side be able to --include=/usr/bin to be able to 'download' only the data from /backup/serverA/usr/bin and exclude everything else. Instead, I have about eight sections of my rsyncd.conf to do this; one for each directory tree that I want to back up to a different location, primarily based on dates, to have multiple incrementals. Thanks, Alex -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Take2: rsync over ssh with --link-dest
On Mon, 2009-11-09 at 00:28 -0500, Alex wrote: I'm a bit disappointed that I can't do it by overriding the server command line Why? Does it have some advantage over the use of an rsync daemon (I can't imagine what)? One can't really expect overriding the server arguments not to lead to the inconsistent behavior you saw. Simplicity. I'd like to be able to have the server side feed me all data beginning with the path I specify, such /backup/serverA, In other words, you were specifying a bogus source path and counting on sshd to override it with the real one. Clever. I didn't catch that the first time. Realize that the ability to set a module's path in the rsyncd.conf file and have the client refer to the module by a well-known name gives you the same functionality, and actually more, because the client can specify a subdirectory path within the module. and have the client side be able to --include=/usr/bin to be able to 'download' only the data from /backup/serverA/usr/bin and exclude everything else. That --include means exclude everything else is a common misconception; see the man page. But I imagine the issue will become moot if you have the client specify a subdirectory of the module instead of using filters. Instead, I have about eight sections of my rsyncd.conf to do this; one for each directory tree that I want to back up to a different location, primarily based on dates, to have multiple incrementals. If the daemon solution is looking fundamentally more complicated than the set of forced server commands you previously had, you're probably going about it the wrong way and I could help if you post the rsyncd.conf file. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
