Re: Using rsync as an incremental backup
I'm using rsync to do an incremental backup of my desktop here, to a remote server as follows: #/usr/bin/bash old=$(date -d 'now - 1 week' +%Y-%m-%d) new=$(date +%Y-%m-%d) rsync -avP --delete --link-dest=../$dir /home/bakers bak...@perturb.org:/home/bakers/backup/$new/ This is actually working GREAT! The only problem is that sometimes the cronjob won't complete (internet is down, something like that). When it tries to run the next week it does --link-dest against a dir that doesn't exist. It happily complies and transfers EVERY file because there is no source to hardlink from. I'd really like rsync to exit and throw an error if the --link-dest isn't present. I can't find anything in the man page about any fancy --list-dest options. Am I missing it? Hopefully, Scott's suggestion will solve this issue for you. You may also be interested in the way LBackup works for hard-linked incremental backups. It uses a numbering system rather than a date system. If you still want to have the date named entries within the file system, it is just a matter of enabling a post-action which keeps a date-named symbolic links directory to these backup snapshots up to date. Details on the LBackup post and pre actions are available from the following URLs : - http://www.lbackup.org/pre_and_post_actions (basic intro on LBackup pre/post actions) - http://bit.ly/lIMfAo (source code for the example post action script mentioned) This is all just another way of approaching this issue which you may find of interest. The one issue is that the script (source code link above) will need to be run on a system which has access to the file system which stores the snapshots. It would require modification if you wanted to run it on a remote system. Hope this helps :^) - This email is protected by LBackup, an open source backup solution. http://www.lbackup.org -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Using rsync as an incremental backup
Scott Baker (sc...@perturb.org) wrote on 23 June 2011 15:30: I'm using rsync to do an incremental backup of my desktop here, to a remote server as follows: #/usr/bin/bash old=$(date -d 'now - 1 week' +%Y-%m-%d) new=$(date +%Y-%m-%d) rsync -avP --delete --link-dest=../$dir /home/bakers bak...@perturb.org:/home/bakers/backup/$new/ This is actually working GREAT! The only problem is that sometimes the cronjob won't complete (internet is down, something like that). When it tries to run the next week it does --link-dest against a dir that doesn't exist. It happily complies and transfers EVERY file because there is no source to hardlink from. I'd really like rsync to exit and throw an error if the --link-dest isn't present. I can't find anything in the man page about any fancy --list-dest options. Am I missing it? You should handle it in your script. You can pass several directories with link-dest and rsync will search for the file in all of them, in the order you gave. If it doesn't find one for hardlinking it pulls the file through the net. We do it here in our backup script to avoid useless copying, both when the backup machine starts the process and when the source of the files starts it. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Using rsync as an incremental backup
We handle it via 2 scripts. One script (on the server side) does daily rotations mid-morning on successfully completed rsyncs from the night before. (i.e. removes daily.7, moves daily.6 to daily.7, etc... - if the rsync was successful) The client side always sends to daily.0 and uses the daily.1 folder as the link-dest. To make the rsync as completed successfully, the client script parses the log and uses sftp to send a summary file to the daily.0 folder. The server side script checks the content of that summary file to see if it should nuke daily.7 and proceed with the rotation. I'm using a heavily modified version of the rsyncsnapshot scripts with automated email reporting from both client and server sides. On 6/24/2011 10:38 AM, Carlos Carvalho wrote: Scott Baker (sc...@perturb.org) wrote on 23 June 2011 15:30: I'm using rsync to do an incremental backup of my desktop here, to a remote server as follows: #/usr/bin/bash old=$(date -d 'now - 1 week' +%Y-%m-%d) new=$(date +%Y-%m-%d) rsync -avP --delete --link-dest=../$dir /home/bakers bak...@perturb.org:/home/bakers/backup/$new/ This is actually working GREAT! The only problem is that sometimes the cronjob won't complete (internet is down, something like that). When it tries to run the next week it does --link-dest against a dir that doesn't exist. It happily complies and transfers EVERY file because there is no source to hardlink from. I'd really like rsync to exit and throw an error if the --link-dest isn't present. I can't find anything in the man page about any fancy --list-dest options. Am I missing it? You should handle it in your script. You can pass several directories with link-dest and rsync will search for the file in all of them, in the order you gave. If it doesn't find one for hardlinking it pulls the file through the net. We do it here in our backup script to avoid useless copying, both when the backup machine starts the process and when the source of the files starts it. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Using rsync as an incremental backup
In 4e03c170.2020...@perturb.org, on 06/23/11 at 03:42 PM, Scott Baker sc...@perturb.org said: Hi, If it were local that would be easy. This is remote (via SSH), so I'd have to login before I run rsync to verify that directory is that. I was hoping I could save that step since rsync is already doing all that already. Your script should be able to use rsync to check if the link-dest directory exists with something like rsync --dirs bak...@perturb.org:/home/bakers/backup/$old/ However, I suspect that the multiple link-dest directories suggested by others will be a simplier solution. Steven -- -- Steven Levine stev...@earthlink.net eCS/Warp/DIY etc. www.scoug.com www.ecomstation.com -- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Using rsync as an incremental backup
I'm using rsync to do an incremental backup of my desktop here, to a remote server as follows: #/usr/bin/bash old=$(date -d 'now - 1 week' +%Y-%m-%d) new=$(date +%Y-%m-%d) rsync -avP --delete --link-dest=../$dir /home/bakers bak...@perturb.org:/home/bakers/backup/$new/ This is actually working GREAT! The only problem is that sometimes the cronjob won't complete (internet is down, something like that). When it tries to run the next week it does --link-dest against a dir that doesn't exist. It happily complies and transfers EVERY file because there is no source to hardlink from. I'd really like rsync to exit and throw an error if the --link-dest isn't present. I can't find anything in the man page about any fancy --list-dest options. Am I missing it? - Scott -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Using rsync as an incremental backup
On 06/23/2011 03:40 PM, Lancashire, Pete wrote: pretty easy to put a wrapper around the script or add it .. If it were local that would be easy. This is remote (via SSH), so I'd have to login before I run rsync to verify that directory is that. I was hoping I could save that step since rsync is already doing all that already. A --link-dest-required flag would be great. That way rsync doesn't copy a bunch of files since there is no hard link source. - Scott -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
