> > In the old tape days, I used to force a level-0 dump when I wanted to do > > that. This was a pain and never very satisfying. So I wrote a perl script > > that will extract the most recent full dump for every disk partition out > > of the vtapes. > > What a neat idea, and one that works best with vtape as the daily backup. > > Would you consider writing up your experiences and techniques?
Sure, although maybe the script itself says it best? :-) I'll append it below. > I'm guessing that for recovery from the archived dumps you would not > use amanda's indexing features. Yes, that's right. The script builds a table-of-contents file for each dump and puts that on the disk as well (this is the part which assumes my dumps are done with gtar). There might be a smarter way to extract that info from amanda's existing indices instead. I also liked someone's idea of copying amanda's indices to the off-site media and having them avaiable for the restore. If someone would like to extend this to have that ability, that would be neat. > Have you had to do any recoveries from the archived dumps rather > than the vtapes? I've done a lot of test restores, and they work fine (using "dd bs=32k skip=1 ..."). I haven't had a call to use the offsites to restore anything under pressure yet. If I never ever get to use my offsite backups and this is all a waste of time, I'll be a happy man :-) Mark -- Mark Costlow | Southwest Cyberport | Fax: +1-505-232-7975 [EMAIL PROTECTED] | Web: www.swcp.com | Voice: +1-505-232-7992 "Education is never a waste" - Viscount du Valmont --------------------- amoffsite ------------------------------------------ #!/usr/local/bin/perl # # $Id: amoffsite,v 1.3 2005/01/12 06:31:02 cheeks Exp $ # # Program: amoffsite # Author: Mark Costlow <[EMAIL PROTECTED]> # Date: Jan, 2005 # # This program prepares an offsite dump. It finds the most recent # level-0 dump for each disk partition in an Amanda config. These are in the # "virtual tapes" of the large RAID that we use for nightly backups. It # copies those files to the "offsite" disk. # # The idea is that someone will run this script once a month to copy the # offsite dumps to a disk, then pull that disk out of the disk array and take # it home with them. If something happens to our disk array, or if we need to # restore a file older than what we have on-site, the offsite disk(s) should # save us. # # # This is the first working version of this program -- there's plenty of room # for improvement. If you have suggestions or fixes, please send them to # [EMAIL PROTECTED] # # As with the rest of amanda, THIS SOFTWARE IS BEING MADE AVAILABLE ``AS-IS''. # It might work for what you want but it might also delete every backup you # ever made, cause your computer to melt, and your hair to catch fire. # # # Usage: offsite-dump configname # # configname is the name of an amanda config. # $| = 1; # No STDOUT buffer $gzip = "/usr/bin/gzip"; $tar = "/usr/bin/gtar"; require 'getopts.pl'; Getopts('hzivd:'); $defroot = "/amanda/offsite"; $offroot = $opt_d ? $opt_d : $defroot; $docomp = $opt_z; $doidx = $opt_i; $verbose = $opt_v; $cat = $docomp ? "/usr/bin/zcat" : "/bin/cat"; $config = shift; if ($config eq '' || $opt_h) { print "Usage: amoffsite [-d dir] configname\n\n"; print "configname is the name of an amanda config.\n\n"; print "-h This help message.\n"; print "-d dir Use dir instead of default target directory [$defroot].\n"; print "-z Compress the dump files with gzip.\n"; print "-i Generate an index for each dump file.\n"; print "-v Be verbose about progress.\n"; exit 1; } %disklist = &read_disklist($config); %tapelist = &find_tapes($config, \%disklist); $vtape_dir = &get_vtape_dir($config); %slots = &find_slots($vtape_dir, \%tapelist); ©_files($config, $vtape_dir, $offroot, \%tapelist, \%slots); exit 0; sub read_disklist { local($config) = @_; local(@lines, $line, $host, $disk, $lnum, $dspec, %dlist); print "Reading disk list ..." if $verbose; $/ = ""; # paragraph input mode open(CF, "amadmin $config disklist |") || die "'amadmin $config disklist: $!\n"; while (<CF>) { @lines = split(/\n/, $_); $host = $disk = ''; foreach $line (@lines) { if ($line =~ /host (\S+):$/) { $host = $1; } elsif ($line =~ /disk (\S+):$/) { $disk = $1; } elsif ($line =~ /^line (\d+)/) { $lnum = $1; } } if ($host eq '' || $disk eq '') { print STDERR "ERROR processing line '$lnum'. host=$host disk=$disk\n"; exit 1; } $dspec = "${host}:${disk}"; $dlist{$dspec} = 1; } close(CF); print " done\n" if $verbose; $/ = "\n"; # back to line input mode return %dlist; } # $dlist is a hashref sub find_tapes { local($config,$dlist) = @_; local(@lines, %tlist, $line, $host, $disk, $dspec, $tspec); local($level, $tape, $fnum, $status, $h, $d, $date); local(%zdates, @dtmp, $nd); $nd = scalar(keys %$dlist); print "Finding tapes for $nd disks ..." if $verbose; foreach $dspec (sort keys %$dlist) { print "." if $verbose; ($host,$disk) = split(/:/, $dspec); %zdates = (); open(AM, "amadmin $config find $host $disk |") || die "'amadmin $config find $host $disk: $!\n"; while (<AM>) { ($date, $h, $d, $level, $tape, $fnum, $status) = split; next if ($date !~ /^\d\d\d\d-\d\d-\d\d/); next if ($status ne 'OK'); next if ($level ne '0'); # If we're still here, then this is a successful level-0 dump $tspec = "${tape}:${fnum}"; $zdates{$date} = $tspec; } close(AM); if (scalar(keys %zdates) == 0) { print STDERR "ERROR: no level-0 for $dspec -- Continue? [n] "; chomp($ans = <STDIN>); if ($ans !~ /y/i) { exit 1; } else { $tlist{$dspec} = 'SKIP'; next; } } # Sort the list of level-0 dates, and then take the last one, which # should be the most recent. @dtmp = sort keys %zdates; $date = pop(@dtmp); $tlist{$dspec} = $zdates{$date}; $dumpdate{$dspec} = $date; } print " done\n" if $verbose; # Return the modified disklist return %tlist; } # $tlist is a hashref sub find_slots { local($tdir, $tlist) = @_; local(%slots, $slot, $tape, $tspec, $fnum); local($files, @files, $nf, $sdir); chdir ($tdir) || die "Can't cd to $tdir: $!\n"; foreach $tspec (sort values %$tlist) { next if ($tspec eq 'SKIP'); ($tape,$fnum) = split(/:/, $tspec); next if (defined $slots{$tape}); # There should be a file called slotN/00000.TAPENAME. This tells us what # directory to find the dump files in for this "tape". $files = `find . -name 00000.$tape -print`; @files = split(/\n/, $files); $nf = scalar(@files); if ($nf != 1) { print STDERR "ERROR: wanted exactly 1 tape with label $tape, but found $nf. Abort.\n"; exit 1; } $sdir = $files[0]; $sdir =~ s|^\./||; $sdir =~ s|/.*||; $slots{$tape} = $sdir; } return %slots; } sub get_vtape_dir { local($config) = @_; local($str, $file, $dir); chomp($str = `amgetconf $config tapedev`); ($file, $dir) = split(/:/, $str); if ($file ne 'file') { print STDERR "ERROR: failed to find a 'file' tape device in $config config.\n"; exit 1; } return $dir; } # # The %tlist and %slots hashes are passed by reference to this function # sub copy_files { local($config, $src_root, $dst_root, $tlist, $slots) = @_; local(@disks, $ndisks, $dnum, $ddir, $dfile, $sfile); local($now,$datestamp,$target_dir); local($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst); # Make a target directory named after the config and today's date $now = time(); ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($now); $datestamp = sprintf("%04d%02d%02d", $year+1900, $mon+1, $mday); $target_dir = "${dst_root}/${config}-${datestamp}"; if (-e "$target_dir") { print "$target_dir already exists. Continue? [n] "; chomp($ans = <STDIN>); if ($ans !~ /y/i) { exit 1; } } else { mkdir($target_dir,0700); if (! -e "$target_dir") { print STDERR "ERROR: Failed to create $target_dir -- abort.\n"; exit 1; } } &init_dumpdate_file($target_dir); @disks = sort keys %$tlist; $ndisks = scalar(@disks); $dnum = 0; foreach $disk (@disks) { $dnum++; if ($tlist->{$disk} eq 'SKIP') { print "Skipping $dnum/$ndisks $disk.\n"; next; } ($host, $part) = split(/:/, $disk); $part =~ s|/|_|g; ($tape, $fnum) = split(/:/, $tlist->{$disk}); $slot = $slots->{$tape}; &add_dumpdate($target_dir,$disk,$dumpdate{$disk}); $sfile = sprintf("%s/%s/%05d.%s.%s.0", $src_root, $slot, $fnum, $host, $part); $ddir = "${target_dir}/${host}"; if (! -d "$ddir") { mkdir($ddir,0700); } $dfile = "${ddir}/${host}.${part}.0"; # printf("%5s %s\n", &getsize($sfile), $sfile); $size_spec = &getsize($sfile); print "Copying $size_spec file $dnum/$ndisks -> ${host}.${part}.0 "; # system ("cp $sfile $dfile"); if ($docomp) { $dfile = "${dfile}.gz"; system ("$gzip -v < $sfile > $dfile"); } else { system ("cp $sfile $dfile"); print "\n"; } if ($doidx) { $ifile = "${ddir}/${host}.${part}.0.INDEX.gz"; print "Creating index file $ifile ..."; system ("$cat $dfile | dd bs=32k skip=1 | $tar tvf - | $gzip > $ifile"); print " done.\n"; } } } # sub getsize { local($file) = @_; local($b, $k, $m, $g, $t); local($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size, $atime,$mtime,$ctime,$blksize,$blocks) = stat($file); $b = $size; if ($b > 1024) { $k = $b / 1024; } else { return sprintf("%.2f B", $b); } if ($k > 1024) { $m = $k / 1024; } else { return sprintf("%.2f KB", $k); } if ($m > 1024) { $g = $m / 1024; } else { return sprintf("%.2f MB", $m); } if ($g > 1024) { $t = $g / 1024; } else { return sprintf("%.2f GB", $g); } # If we get this far, the file is over 1 TB. Wow. return sprintf("%.2f TB", $t); } sub getbytes { local($file) = @_; local($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size, $atime,$mtime,$ctime,$blksize,$blocks) = stat($file); return $size; } sub init_dumpdate_file { local($dir) = @_; open (DDF, "> $dir/dumpdates.txt") || die "Can't open $dir/dumpdates.txt: $!\n"; close(DDF); } sub add_dumpdate { local($dir,$disk,$date) = @_; open (DDF, ">> $dir/dumpdates.txt") || die "Can't open $dir/dumpdates.txt: $!\n"; print DDF "$date\t$disk\n"; close(DDF); }