-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, as lots of other people I had serious issues transferring the pool to a remote site. Rsync needs days to sync the hardlinks. dump/restore doesn't work properly (or dumps too much data, don't remember). drbd needs a proxy through small lines (has to be purhcased). Full image backup is too much data for the line or needs manual intervention (bring the USB drive manually to remote office).
So I created a small script which will backup my pool to an remote office through the line. It'll split the image in 512MB chunks, compress them and store them on the USB drive. After the backup is done, I'll use rsync to transfer the relatively small chunks to remote office. What doese it do so? Reads the disk image in 512MB chunk and copies them to ramdisk /dev/shm (you'll need enough ram though!). It'll calculate the md5sum of the chunk and compare to the existing one. If they differ, it'll gzip the chunk and copy it to the USB and update the md5. If md5 are same, it'll just remove the chunk from ramdisk. Disadvantage is the script has to read the entire device every time is runs. Another disadvantge might be the sizing for the chunks. If they are to large a single bit change in a block will force the transfer of the whole chunk. Too small chunks might reduce performance (not tested!). It scales nearly linear with thesize of disk. If no data has changed it'll need double time on double size. It's not perfect at the moment, but it works. My initial backup now runs for 3 hrs and has transferred 100G (compressed to 94G). Mostly waiting for gzip to finish- so if you prefer, just skip compression. Any comments and suggestions are welcome. #!/bin/bash date SRC=/dev/sdb1 DST=/srv/backuppc TMP_FILE=/dev/shm/tmp DST_FILE=$DST/backuppc.gz.iso service backuppc stop mount /srv || exit if [ ! -e /srv/backuppc ] ; then mkdir /srv/backuppc; fi rm -fr /dev/shm/* sync rm -f $DST/backuppc.log i=0 loop=`losetup -f` bs=$[64*1024] # blocksize 64k fs=$[1024*1024*512] # filesize 512M count=$[$fs/$bs] # block count per file (16384) devlen=`df | grep $SRC| awk '{print $2}'` # #of 1k blocks umount /var/lib/BackupPC devlen=$[$devlen*1024] # #of bytes devbsblocks=$[$devlen/$bs] # #of 64k blocks # In case device does not perfectly fit into 64 blocks if [ $[$devbsblocks*$bs] -ne $devlen ] ; then devbsblocks=$[$devbsblocks+1] ; fi files=$[$devbsblocks/$count] # #of blocks per dest file if [ $[$files*$count] -ne $devbsblocks ] ; then files=$[$files+1] ; fi while [ $i -le $files ] ; do losetup -o $[$i*$count*$bs] $loop $SRC free=$[`df /dev/shm | grep shm|awk '{print $4}'`*1024] while [ $free -lt $fs ] ; do sleep 30s free=$[`df /dev/shm | grep shm|awk '{print $4}'`*1024] done #dd if=$loop bs=$bs count=$count | tee $TMP_FILE.$i | md5sum > $TMP_FILE.$i.md5 dd if=$loop bs=$bs count=$count of=$TMP_FILE.$i md5sum $TMP_FILE.$i > $TMP_FILE.$i.md5 losetup -d $loop if [ -e $DST_FILE.$i.md5 ] ; then echo "diff $DST_FILE.$i.md5 $TMP_FILE.$i.md5" diff $DST_FILE.$i.md5 $TMP_FILE.$i.md5 if [ $? -ne 0 ] ; then echo "$i is different" >> $DST/backuppc.log ( cat $TMP_FILE.$i | gzip -2 > $DST_FILE.$i ; rm -f $TMP_FILE.$i ) & mv $TMP_FILE.$i.md5 $DST_FILE.$i.md5 else rm -f $TMP_FILE.$i $TMP_FILE.$i.md5 fi else ( cat $TMP_FILE.$i | gzip -2 > $DST_FILE.$i ; rm -f $TMP_FILE.$i ) & mv $TMP_FILE.$i.md5 $DST_FILE.$i.md5 fi i=$[$i+1] done umount /srv service backuppc start date Greetings Christian -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with CentOS - http://enigmail.mozdev.org/ iD8DBQFLD8Ih0XNIYlAXmzsRAs9MAJsFi2H/pxWQOBQohPcXRXtXYUqjagCfUM3c VHc9dbKXsVZP+d1s9KPn6xI= =E6yw -----END PGP SIGNATURE----- ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/