Hello everyone, trying to come up with a way for efficiently synchronising a BackupPC archive on one server with a remote and encrypted offsite backup, the following problems arise: * As often pointed out on this list, filesystem-level synchronisation is extremely cpu and memory-intensive. Not actually impossible, but depending on the scale of your backups, it is maybe not a practical solution. In our case of a 350GiB pool containing 4 million directories and 20 miilion inodes, simply locally copying the whole pool using cp/rsync/xfsdump/whatever thrashes, gets killed by OOM or at least takes days, longer than i find reasonable for a remote synchronisation run. * Furthermore, we want our offsite backup to be encrypted - in the ideal case using a secret key that is at no single moment ever known at the remote location - there should only be encrypted files sent to it, and stored there. Doing this encryption at the file level given such massive amount of small files, is a very serious additional overhead. * The alternative to file-level synchronisation is (block)device-level synchronisation. Many possibilities exist here, including ZFS send/receive (if you use ZFS), using snapshots (eg. LVM) or temporarily stopping backups, and do a full copy of the pool to the remote side (if you have enough bandwidth), etc... Not everyone is willing to use these, or is prepared to convert to such a system. * We would like to use Rsync for this, since it will skip identical parts, yet guarantee that the whole file is byte-per-byte identical to the original. Unfortunately, as far as I know, rsync doesn't support data on block devices to be synced, only the block device node itself. In addition to that, rsync needs to read and process the whole file on the receiver side, calculate checksums, send them all to the sender side, wait for the sender to reconstruct the data using the checksums, send this reconstruction, and apply this reconstruction at the receiver side. This requires at least the sum of the times to read through the whole data on both sides if it is a single file (correct me if i'm wrong, i don't know rsync internals). Data hardly moves on-disk in the case of a BackupPC pool, so we would like to disable or at least limit the range in which rsync searches for matching data.
To overcome this issue, i wrote a perl/fuse filesystem that allows you to "mount" a block device (or real file) as a directory containing files part0001.img, part0002.img, ... each representing 1 GiB of data of the original device: https://svn.ulyssis.org/repos/sipa/backuppc-fuse/devfiles.pl This directory can be rsynced in a normal way with an "ordinary" directory on an offsite backup. In case a restore is necessary, doing 'ssh remote "cat /backup/part*.img" >/dev/sdXY' (or equivalent) suffices. Although devfiles.pl has (limited) write support, rsync'ing to the resulting directory is not yet possible - maybe i can try to have this working if people have a need for it. This would allow restoration by simply rsync'ing in the opposite direction. Doing the synchronisation in groups of 1GiB prevents rsync from searching too far, and splitting it in multiple files allows some parallellism (sender transmitting data to receiver, while receiver already checksums the next file; this is heavily limited by disk I/O however). In our case, the BackupPC pool is stored on an XFS filesystem on an LVM volume, allowing a xfsfreeze/sync/snapshot/xfsunfreeze, and using devfiles.pl on the snapshot. Instead of xfsfreeze+unfreeze, a backuppc stop/umount + mount/backuppc start is also possible. If no system for making snapshots is available, you would need to suspend backuppc during the whole synchronisation. In fact, the BackupPC volume is already encrypted on our backup server itself, allowing very cheap encrypted offsite backups (simply not sending the keyfile to the remote side is enough...) The result: offsite backups of our 400GiB pool, containing 350GiB data, of which about 2GiB changes daily, is synchronised 5 times a week with offsite backup in 12-15 hours, requiring nearly no bandwidth. This seems mostly limited by the slow disk I/O on the receiver side (25MiB/s). Hope you find this interesting/useful, -- Pieter ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/