Re: Send/Receive howto and script for others to use (was Re: Is anyone using btrfs send/receive)
On 2014/03/22 11:11 PM, Marc MERLIN wrote: Please consider adding a blank line between quotes, it makes them just a bit more readable :) Np. On Sat, Mar 22, 2014 at 11:02:24PM +0200, Brendan Hide wrote: - it doesn't create writeable snapshots on the destination in case you want to use the copy as a live filesystem One of the issues with doing writeable snapshots by default is that the backup is (ever so slightly) less safe from fat-finger syndrome. If I want a writeable snapshot, I'll make it from the read-only snapshot, thereby reducing the chances of accidentally tainting or deleting data in the backup. I actually *did* accidentally delete my entire filesystem (hence the paranoid umounts). But, of course, my script *first* created read-only snapshots from which recovery took only a few minutes. ;) The writeable snapshot I create is on top of the read only one used by btrfs receive. So, I can play with it, but it won't upset/break anything for the backup. The historical snapshots I keep give me cheap backups to go back to do get a file I may have deleted 3 days ago and want back now even though my btrfs send/receive runs hourly. Ah. In that case my comment is moot. I could add support for something like this but I'm unlikely to use it. [snip] - Your comments say shlock isn't safe and that's documented. I don't see that in the man page http://manpages.ubuntu.com/manpages/trusty/man1/shlock.1.html That man page looks newer than the one I last looked at - specifically the part saying improved by Berend Reitsma to solve a race condition. The previous documentation on shlock indicated it was safe for hourly crons - but not in the case where a cron might be executed twice simultaneously. Shlock was recommended by a colleague until I realised this potential issue, thus my template doesn't use it. I should update the comment with some updated information. It's not super important, it was more my curiosity. If a simple lock program in C isn't atomic, what's the point of it? I never looked at the source code, but maybe I should... Likely the INN devs needed something outside of a shell environment. Based on the man page, shlock should be atomic now. I'd love to have details on this if I shouldn't be using it - Is set -o noclobber; echo $$ $lockfile really atomic and safer than shlock? If so, great, although I would then wonder why shlock even exists :) The part that brings about an atomic lock is noclobber, which sets it so that we are not allowed to clobber/overwrite an existing file. Thus, if the file exists, the command fails. If it successfully creates the new file, the command returns true. I understand how it's supposed to work, I just wondered if it was really atomic as it should be since there would be no reason for shlock to even exist with that line of code you wrote. When I originally came across the feature I wasn't sure it would work and did extensive testing: For example, spawn 30 000 processes, each of which tried to take the lock. After the machine became responsive again ;) only 1 lock ever turned out to have succeeded. Since then its been in production use across various scripts on hundreds of servers. My guess (see above) is that the INN devs couldn't or didn't want to use it. The original page where I learned about noclobber: http://www.davidpashley.com/articles/writing-robust-shell-scripts/ Thanks for the info Marc No problem - and thanks. :) -- __ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Send/Receive howto and script for others to use (was Re: Is anyone using btrfs send/receive)
On 2014/03/21 07:29 PM, Marc MERLIN wrote:
On Wed, Jan 08, 2014 at 12:02:06AM -0800, Marc MERLIN wrote:
On Tue, Jan 07, 2014 at 10:53:29AM +, Hugo Mills wrote:
You need to move /mnt/btrfs_pool2/tmp_read_only_new to a different
name as well. The send stream contains the name of the subvolume it
wants to create, so it's trying to create a subvolume called
tmp_read_only_new in /mnt/btrfs_pool2, and there's one there already.
Aaah, got it, thanks.
I'll contribute back a shell script that makes this much easier when I'm
done with it.
I've now spent enough time on this and have enough testing to release it.
http://marc.merlins.org/perso/btrfs/post_2014-03-22_Btrfs-Tips_-Doing-Fast-Incremental-Backups-With-Btrfs-Send-and-Receive.html
which points to
http://marc.merlins.org/linux/scripts/btrfs-subvolume-backup
Just for the archives, here is a working copy (the blog post above explains
why it keeps multiple snapshots after the fact, and creates writeable ones
on the destination, but basically this allows you to automatically boot from
the latest snapshot (a writeable snapshot copy pointed to by a symlink tha
tis kept up to date)):
#!/bin/bash
# By Marc MERLIN marc_s...@merlins.org
# License: GPL-2 or BSD at your choice.
# Source: http://marc.merlins.org/linux/scripts/
# $Id: btrfs-subvolume-backup 958 2014-03-16 00:23:28Z svnuser $
# cron jobs might not have /sbin in their path.
export PATH=$PATH:/sbin
set -o nounset
set -o errexit
set -o pipefail
# From https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
# bash shortcut for `basename $0`
PROG=${0##*/}
lock=/var/run/$PROG
usage() {
cat EOF
Usage:
cd /mnt/source_btrfs_pool
$PROG [--init] [--keep|-k num] [--dest hostname] volume_name
/mnt/backup_btrfs_pool
Options:
--init: Print this help message and exit.
--keep num: Keep the last snapshots for local backups (5 by default)
--dest hostname: If present, ssh to that machine to make the copy.
This will snapshot volume_name in a btrfs pool, and send the diff
between it and the previous snapshot (volume_name.last) to another btrfs
pool (on other drives)
If your backup destination is another machine, you'll need to add a few
ssh commands this script
The num sanpshots to keep is to give snapshots you can recover data from
and they get deleted after num runs. Set to 0 to disable (one snapshot will
be kept since it's required for the next diff to be computed).
EOF
exit 0
}
die () {
msg=${1:-}
# don't loop on ERR
trap '' ERR
rm $lock
echo $msg 2
echo 2
# This is a fancy shell core dumper
if echo $msg | grep -q 'Error line .* with status'; then
line=`echo $msg | sed 's/.*Error line \(.*\) with status.*/\1/'`
echo DIE: Code dump: 2
nl -ba $0 | grep -3 \b$line\b 2
fi
exit 1
}
# Trap errors for logging before we die (so that they can be picked up
# by the log checker)
trap 'die Error line $LINENO with status $?' ERR
init=
# Keep the last 5 snapshots by default
keep=5
TEMP=$(getopt --longoptions help,usage,init,keep:,dest: -o h,k:,d: -- $@) ||
usage
dest=localhost
ssh=
# getopt quotes arguments with ' We use eval to get rid of that
eval set -- $TEMP
while :
do
case $1 in
-h|--help|--usage)
usage
shift
;;
--keep|-k)
shift
keep=$1
shift
;;
--dest|-d)
shift
dest=$1
ssh=ssh $dest
shift
;;
--init)
init=1
shift
;;
--)
shift
break
;;
*)
echo Internal error!
exit 1
;;
esac
done
[[ $keep 1 ]] die Must keep at least one snapshot for things to work ($keep
given)
DATE=$(date '+%Y%m%d_%H:%M:%S')
[[ $# != 2 ]] usage
vol=$1
dest_pool=$2
# shlock (from inn) does the right thing and grabs a lock for a dead process
# (it checks the PID in the lock file and if it's not there, it
# updates the PID with the value given to -p)
if ! shlock -p $$ -f $lock; then
echo $lock held for $PROG, quitting 2
exit
fi
if [[ -z $init ]]; then
test -e ${vol}_last \
|| die Cannot sync $vol, ${vol}_last missing. Try --init?
src_snap=$(readlink -e ${vol}_last)
fi
src_newsnap=${vol}_ro.$DATE
src_newsnaprw=${vol}_rw.$DATE
$ssh test -d $dest_pool/ || die ABORT: $dest_pool not a directory (on $dest)
btrfs subvolume snapshot -r $vol $src_newsnap
# There is currently an issue that the snapshots to be used with btrfs send
# must be physically on the disk, or you may receive a stale NFS file handle
# error. This is accomplished by sync after the snapshot
sync
if [[ -n $init ]]; then
btrfs send $src_newsnap | $ssh btrfs receive $dest_pool/
else
btrfs send -p $src_snap
Re: Send/Receive howto and script for others to use (was Re: Is anyone using btrfs send/receive)
On 2014/03/22 09:44 PM, Brendan Hide wrote: On 2014/03/21 07:29 PM, Marc MERLIN wrote: Hi, Marc Feel free to use ideas from my own script. Some aspects in my script are more mature and others are frankly pathetic. ;) There are also quite a lot of TODOs throughout my script that aren't likely to get the urgent attention they deserve. It has been slowly evolving over the last two weeks. http://swiftspirit.co.za/scripts/btrfs-snd-rcv-backup I forgot to include some notes: The script depends on a config file at /etc/btrfs-backup/paths.conf which is supposed to contain the paths as well as some parameters. At present the file consists solely of the following paths as these are all separate subvolumes in my test system: / /home /usr /var The snapshot names on source and backup are formatted as below. This way, daylight savings doesn't need any special treatment: __2014-03-17-23h00m01s+0200 __2014-03-18-23h00m01s+0200 __2014-03-19-23h00m01s+0200 __2014-03-20-23h00m02s+0200 __2014-03-21-23h00m01s+0200 _home_2014-03-17-23h00m01s+0200 _home_2014-03-18-23h00m01s+0200 _home_2014-03-19-23h00m01s+0200 _home_2014-03-20-23h00m02s+0200 _home_2014-03-21-23h00m01s+0200 _usr_2014-03-17-23h00m01s+0200 _usr_2014-03-18-23h00m01s+0200 _usr_2014-03-19-23h00m01s+0200 _usr_2014-03-20-23h00m02s+0200 _usr_2014-03-21-23h00m01s+0200 _var_2014-03-17-23h00m01s+0200 _var_2014-03-18-23h00m01s+0200 _var_2014-03-19-23h00m01s+0200 _var_2014-03-20-23h00m02s+0200 _var_2014-03-21-23h00m01s+0200 -- __ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Send/Receive howto and script for others to use (was Re: Is anyone using btrfs send/receive)
On Sat, Mar 22, 2014 at 09:44:05PM +0200, Brendan Hide wrote: Hi, Marc Feel free to use ideas from my own script. Some aspects in my script are more mature and others are frankly pathetic. ;) There are also quite a lot of TODOs throughout my script that aren't likely to get the urgent attention they deserve. It has been slowly evolving over the last two weeks. http://swiftspirit.co.za/scripts/btrfs-snd-rcv-backup I figured I likely wasn't the only one working on a script like this :) From a quick read, it looks even more complex than mine :) but - it doesn't do ssh to a destination for a remote backup - it doesn't seem to keep a list of configurable snapshots not necessary for send/restore but useful for getting historical data - it doesn't seem to use a symlink to keep track of the last complete snapshot on the source and destination, and does more work to compensate when recovering from an incomplete backup/restore. - it doesn't create writeable snapshots on the destination in case you want to use the copy as a live filesystem Things I noticed: - I don't use ionice, maybe I should. Did you find that it actually made a difference with send/receive? - Your comments say shlock isn't safe and that's documented. I don't see that in the man page http://manpages.ubuntu.com/manpages/trusty/man1/shlock.1.html I'd love to have details on this if I shouldn't be using it - Is set -o noclobber; echo $$ $lockfile really atomic and safer than shlock? If so, great, although I would then wonder why shlock even exists :) Thanks, Marc -- A mouse is a device used to point at the xterm you want to type in - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Send/Receive howto and script for others to use (was Re: Is anyone using btrfs send/receive)
On 2014/03/22 10:00 PM, Marc MERLIN wrote: On Sat, Mar 22, 2014 at 09:44:05PM +0200, Brendan Hide wrote: Hi, Marc Feel free to use ideas from my own script. Some aspects in my script are more mature and others are frankly pathetic. ;) There are also quite a lot of TODOs throughout my script that aren't likely to get the urgent attention they deserve. It has been slowly evolving over the last two weeks. http://swiftspirit.co.za/scripts/btrfs-snd-rcv-backup I figured I likely wasn't the only one working on a script like this :) From a quick read, it looks even more complex than mine :) but Well ... I did say some things are pathetic my side. ;) I also use a template (its about 3 years old now) when I make a new script, hence the options such as being able to ignore the mutex checks and also having a random delay at start. These obviously add some unnecessary complexity. - it doesn't do ssh to a destination for a remote backup There should be a TODO for this on my side. Presently in testing I'm only using it for device-local backup to a separate disk and not to a proper remote backup. - it doesn't seem to keep a list of configurable snapshots not necessary for send/restore but useful for getting historical data I'm not sure what this is useful for. :-/ If related, I plan on creating a separate script to move snapshots around into _var_daily_$date, _var_weekly_$date, etc. - it doesn't seem to use a symlink to keep track of the last complete snapshot on the source and destination, and does more work to compensate when recovering from an incomplete backup/restore. Yes, a symlink would make this smoother. - it doesn't create writeable snapshots on the destination in case you want to use the copy as a live filesystem One of the issues with doing writeable snapshots by default is that the backup is (ever so slightly) less safe from fat-finger syndrome. If I want a writeable snapshot, I'll make it from the read-only snapshot, thereby reducing the chances of accidentally tainting or deleting data in the backup. I actually *did* accidentally delete my entire filesystem (hence the paranoid umounts). But, of course, my script *first* created read-only snapshots from which recovery took only a few minutes. ;) Things I noticed: - I don't use ionice, maybe I should. Did you find that it actually made a difference with send/receive? This is just a habit I've developed over time in all my scripts. I figure that if I'm using the machine at the time and the snapshot has a large churn, I'd prefer the ionice. That said, the main test system is a desktop which is likely to have much less churn than a server. In the last two weeks the longest daily incremental backup took about 5 minutes to complete, while it typically takes about 30 seconds only. - Your comments say shlock isn't safe and that's documented. I don't see that in the man page http://manpages.ubuntu.com/manpages/trusty/man1/shlock.1.html That man page looks newer than the one I last looked at - specifically the part saying improved by Berend Reitsma to solve a race condition. The previous documentation on shlock indicated it was safe for hourly crons - but not in the case where a cron might be executed twice simultaneously. Shlock was recommended by a colleague until I realised this potential issue, thus my template doesn't use it. I should update the comment with some updated information. My only two worries then would be a) if it is outdated on other distros and b) that it appears that it is not installed by default. On my Arch desktop it seems to be available with inn[1] (Usenet server and related software) and nowhere else. It seems the same on Ubuntu (Google pointed me to inn2-dev). Do you have INN installed? If not, where did you get shlock from? I'd love to have details on this if I shouldn't be using it - Is set -o noclobber; echo $$ $lockfile really atomic and safer than shlock? If so, great, although I would then wonder why shlock even exists :) The part that brings about an atomic lock is noclobber, which sets it so that we are not allowed to clobber/overwrite an existing file. Thus, if the file exists, the command fails. If it successfully creates the new file, the command returns true. I'd consider changing this mostly for the fact that depending on INN is a very big dependency. There are other options as well, though I don't think they're as portable as noclobber. Thanks, Marc Thanks for your input. It has already given me some direction. :) [1] https://www.archlinux.org/packages/community/x86_64/inn/files/ -- __ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Send/Receive howto and script for others to use (was Re: Is anyone using btrfs send/receive)
Please consider adding a blank line between quotes, it makes them just a bit more readable :) On Sat, Mar 22, 2014 at 11:02:24PM +0200, Brendan Hide wrote: - it doesn't create writeable snapshots on the destination in case you want to use the copy as a live filesystem One of the issues with doing writeable snapshots by default is that the backup is (ever so slightly) less safe from fat-finger syndrome. If I want a writeable snapshot, I'll make it from the read-only snapshot, thereby reducing the chances of accidentally tainting or deleting data in the backup. I actually *did* accidentally delete my entire filesystem (hence the paranoid umounts). But, of course, my script *first* created read-only snapshots from which recovery took only a few minutes. ;) The writeable snapshot I create is on top of the read only one used by btrfs receive. So, I can play with it, but it won't upset/break anything for the backup. The historical snapshots I keep give me cheap backups to go back to do get a file I may have deleted 3 days ago and want back now even though my btrfs send/receive runs hourly. Things I noticed: - I don't use ionice, maybe I should. Did you find that it actually made a difference with send/receive? This is just a habit I've developed over time in all my scripts. I figure that if I'm using the machine at the time and the snapshot has a large churn, I'd prefer the ionice. That said, the main test system is a desktop which is likely to have much less churn than a server. In the last two weeks the longest daily incremental backup took about 5 minutes to complete, while it typically takes about 30 seconds only. I'll consider that if I see too much load without ionice, thanks. - Your comments say shlock isn't safe and that's documented. I don't see that in the man page http://manpages.ubuntu.com/manpages/trusty/man1/shlock.1.html That man page looks newer than the one I last looked at - specifically the part saying improved by Berend Reitsma to solve a race condition. The previous documentation on shlock indicated it was safe for hourly crons - but not in the case where a cron might be executed twice simultaneously. Shlock was recommended by a colleague until I realised this potential issue, thus my template doesn't use it. I should update the comment with some updated information. It's not super important, it was more my curiosity. If a simple lock program in C isn't atomic, what's the point of it? I never looked at the source code, but maybe I should... I'd love to have details on this if I shouldn't be using it - Is set -o noclobber; echo $$ $lockfile really atomic and safer than shlock? If so, great, although I would then wonder why shlock even exists :) The part that brings about an atomic lock is noclobber, which sets it so that we are not allowed to clobber/overwrite an existing file. Thus, if the file exists, the command fails. If it successfully creates the new file, the command returns true. I understand how it's supposed to work, I just wondered if it was really atomic as it should be since there would be no reason for shlock to even exist with that line of code you wrote. I'd consider changing this mostly for the fact that depending on INN is a very big dependency. There are other options as well, though I don't think they're as portable as noclobber. I totally agree, bringing INN to get shlock sucks a bit. I have shlock stashed in /usr/local/bin so that I don't have to :) Thanks for the info Marc -- A mouse is a device used to point at the xterm you want to type in - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
