On Sun, Mar 10, 2013 at 04:12:31AM +0100, Holger Parplies wrote: > Hi, > > Peter Carlsson wrote on 2013-03-09 11:10:59 +0100 [Re: [BackupPC-users] > Archiving incremental backups?]: > > [...] > > I regulary do an archive to a USB HDD that I store offsite. This is a > > manual step since I have to bring the HDD, do the archiving, and then > > store the HDD offsite again. > > > > To know for sure that files that are modified in between these regular > > archives, I want to make tar archives of the incremental backups and > > move them on a daily basis offsite over the Internet. > > > > Why I only want to do this for the incremental backuped files is to > > reduce the amount of data over the Internet. > > in theory, you could use rsync to transfer incremental changes over the > internet. This could even be substantially less traffic than an incremental > tar, presuming you have changing files that can efficiently be transferred > with rsync (like growing log files or large files where only small portions > change from day to day). In the worst case (only new files, no changed files) > there should not be much difference, except that rsync requires more > computational power, and that rsync (with the right options) will track > deletions. rsync also has built-in capability to limit transfer bandwidth. > > The thing is, you would need to keep an image of your target file system on > both ends, i.e. unpack the tar file on your USB HDD and have it accessible > over > the internet, and have a local copy on or near your BackupPC server. You could > use the original file system (the one you back up in the first place) instead > of a copy, presuming your concern is not to mirror backup state (your file > system will probably have changed since the last backup), and the file system > can handle the additional load of the rsync run. > > One thing I would like to note, though, is that you really want at least two > independent offsite copies, so you will not be left without one if things > break while you are replacing the old copy with a new one. This is especially > true with tar archives. An rsync run will generally not destroy much (maybe > one file) if it fails prematurely, though it will leave you with a state of > your offsite copy that probably never existed on the original (which is easily > fixed if you can just restart rsync). But you should note that requiring the > offsite copy to be online (at least during the incremental update) makes it > somewhat vulnerable, so having an additional *offline* offsite backup would > be a good idea. > > > But what I want to achieve is, at least in my > > opinion, better than to only have the full manual archives that at > > best will be done one or two times a month. > > With rsync, in my opinion, you can almost get away without the full manual > archives. The same note applies here as in the frequently asked question, > "with rsync, do I need full backups at all?". Ideally, you would turn on the > rsync option "--ignore-times" regularly (e.g. once a month) to catch any > (rare) changes rsync might have missed (but that's just a detail). > > I don't know if what I described seems possible in your scenario. We should > figure that out first before going into too much detail. > > > The most important thing is that it is simple and automatic, otherwise > > it will never be done. > > It should be possible to make this automatic (except for exchanging or syncing > the offsite drives once in a while, but that's much like your monthly manual > archives now). It doesn't seem overly complex, but that depends somewhat on > your setup. How were you planning to get the incrementals over the internet? > Have you done size estimates to see if it is feasible? > > > Jeffrey hinted at the possibility to tar up the pc/host/num directory for an > incremental backup. While that is certainly possible, it leaves you with the > problem that you would need to unmangle names and interpret attrib files. > That's not too difficult, but it would require some coding. The thing is, how > (and under what circumstances) would you ever *use* the offsite backup? In the > event of a catastrophe? Bring in the disk, restore the full backup, restore > all incrementals? I'd be in favour of having a working file system image (or > better, two identical ones - one as a backup remaining offsite) which you can > just plug in and use, if things need to go really fast, or copy over without > needing to think much (i.e. without much that can go wrong). > > Another thing to keep in mind are backup levels. Normally, your incrementals > will tend to be relative to the last full, meaning they will grow from day to > day (because day 2 repeats the changes from day 1 and so on). You *can* fix > that (at the cost of more complexity at backup time), but you probably don't > want to bother with this approach anyway :). > > > There's, of course, the third possibility of just setting up an offsite > BackupPC server that makes independent backups of the target host. You'd > want to have a VPN for that, but my guess is that incremental tars would be > sent over one, too :-). > > Regards, > Holger
Hi, Thanks for your detailed reply! I realize after yours and others explanations, that I have not thought about all the shortcomings, but I was thinking this could be a good compromise. This would allow me (at least with some effort) to restore modified files even if a crash happened between two full archives. I will go back to my drawing board and think more about what I want to achieve, now that I have additional information. Best regards, Peter Carlsson ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_mar _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/