On 15/05/12 09:34, Andreas Piening wrote: > There are already two USB-disks that are swapped every few days and I > copy nightly images on the currently connected drive. Some weeks ago, > the office had a water-pipe break. The water has been stopped early > enough so there was no damage, but my customer assigned me to create a > external backup solution. He wants me to be able to completely restore > the system including the virtual machines if someone breaks into the > office and steals the hardware ore something like that. If this > happens, I need at least one day to buy new hardware. But the point is > that I need to be able to restore the system afterwards to a working > state just like it was one day before the disaster happens. > > As I understand the rsync functionality the algorithm is able to do > in-file incremental updates. My problem is just that I can't figure > out what prevents it from working in my case... >
Currently, I handle the situation as follows: 1) File level backup with backuppc, this covers the non-disaster/disaster scenario where somebody inadvertently trashes your database, or deletes a "important" file, etc... need to restore to yesterday, or last week, etc but only a small number of files, not all of them. Some applications require pre-processing to be done, eg, MS SQL, so I use custom scripts that will export those files using whatever "backup procedure" is required for the proprietary application. Then depending on the output: a) If it is compressed, especially compressed "ascii format files", then I will de-compress them (and continue with (b) b) If it is a large file (over 100M) then I will split the file into smaller chunks of about 20M to 50M each (just based on the quantity) c) I will keep multiple folders (1 for each work day (5), or 1 for each day of the week (7)), and let backuppc backup all these folders. This helps ensure that every version of the daily backup makes it into backuppc, and also allows for backuppc to run during the local backup/split process, because it will always get a complete backup copy within the past 24 hours d) I then use Xymon monitoring to ensure that these local backup files have been modified within the past 26 hours (to alert me if the local pre-processing script has failed/etc). No point of backups if the files are never updated. 2) Disk level backups a) shutdown the virtual machine (for VM's) b) split the VM disk into chunks in another folder (local) c) copy the chunks to a remote machine with rsync d) the remote machine examines the chunks to ensure it has a full up to date copy, and then re-assembles the chunks into a disk image, preserving the previous disk image as .old BTW, the reason to split large files into smaller chunks are: 1) if a chunk has no changes, backuppc gets to take advantage of pooling 2) backuppc doesn't handle large files with small changes very well (performance wise in my experience), it seems to handle a totally new file, or small files with changes perfectly. 3) If the rsync fails, it can be re-started, completed chunks are compared and skipped much more quickly (you could use partial) 4) You don't need to wait for the remote rsync to make a temp copy of the large file before starting to compare/transfer the file (you could use inplace) Some rsync options are available (inplace and partial) but I feel the chunk method is still more efficient from a time, bandwidth and disk I/O points of view. I originally used LVM2 snapshots to backup the live VM's, but had two issues with this: 1) Potentially the image I've backed up is corrupted since it is "point in time" and therefore not "cleanly unmounted" 2) I had dreadful disk I/O performance issues (the VM, snapshot space, and destination temp copy were ALL stored on the same RAID10 array, this might have been better if a different disk was used for the destination, but either way, I found the customer was happy to shutdown the machine for a backup window each night. I hope some of the above helps... Regards, Adam -- Adam Goryachev Website Managers www.websitemanagers.com.au ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/