Re: incremental full file backups to smaller mediums possible?
Am Samstag, 18. April 2015, 01:03:23 schrieb Christoph Anton Mitterer: On Thu, 2015-04-09 at 16:33 +, Hugo Mills wrote: btrfs sub find-new might be more helpful to you here. That will give you the list of changed files; then just feed that list to your existing bin-packing algorithm for working out what goes on which disks, and you're done. hmm that sounds interesting, but a) it lacks much documentation (how does that with the generations work? when is there a new generation? what, if the generation overflows?) I have no idea about when there is a new generation. I think it depends on the commit intervall. You can see the current generation with: merkaba:~ btrfs subvol show / | grep Generation Generation (Gen): 468143 b) it seems to only show me new files, but not files that have been moved, or removed, or when properties have changed. It should show renamed or moved files. But yeah, if it does, at least not in a way that shows that a rename or move operation took place. c) it seems to work only for one subvolume? Its per subvolume, I think. Thats why it is under btrfs subvolume Ciao, -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 signature.asc Description: This is a digitally signed message part.
Re: incremental full file backups to smaller mediums possible?
On Thu, 2015-04-09 at 16:33 +, Hugo Mills wrote: btrfs sub find-new might be more helpful to you here. That will give you the list of changed files; then just feed that list to your existing bin-packing algorithm for working out what goes on which disks, and you're done. hmm that sounds interesting, but a) it lacks much documentation (how does that with the generations work? when is there a new generation? what, if the generation overflows?) b) it seems to only show me new files, but not files that have been moved, or removed, or when properties have changed. c) it seems to work only for one subvolume? Cheers, Chris. smime.p7s Description: S/MIME cryptographic signature
incremental full file backups to smaller mediums possible?
Hey. I wondered whether this is possible in btrfs (or could be implemented),... it's in a way similar to send/receive, but AFAIU not fully solvable with that. What I want to do is making incremental backups of a (btrfs) filesystem to smaller mediums (that is for example: from a big RAID filesystem to many BluRays or similar things). Right now I make this as follows: Every time backups should be made, I create a ext4 image which would just fit inside the UDF fs of a BluRay (the ext4 image in turn is dm-crypt encrypted) and copy as many *complete*[0] files from the source filesystem to that image as possible. That image file is then burned to BluRay. I also write an SHA512 sum of the whole image file to the BluRay so that one can check whether it can be still read correctly. Then the process continues with the remaining files. The main idea behind writing only complete files to the separate parts of the backup is, that I don't want to have a backup set, where I may get into troubles if just one of the disks fails. Obviously, when I would do regularly backups that way, then I'd write each time a big amount of data which hasn't changed at all, which is not just a costly issue, but also quite time consuming. So ideally, I'd want to do just an incremental backup of all the files that have been added/modified + the information of what has been deleted/moved/changed it's properties. All that already smells quite a lot after making a snapshot at the last backup, and then doing send/receive for the next one. Now the problem, though, is, that I don't want to loose the property of working in terms of complete files (and I guess send/recieve does not), i.e. when a big file has just one block changed, I still would like to have the complete file on the next incremental dump (and not just information about the diff). The only exception is when file metadata changes (i.e. pathname, date, permissions, xattrs, etc.). If *just* one of these changes I wouldn't want to backup the full raw data of the file again. Another issue is, that send gives me one big file, while I would need however n files of max size m, where m is e.g. what fits in a BluRay's UDF... or perhaps on a 3,5 HD Floppy ;) Because of the complete files constraint I cannot just split the file from btrfs send,... because the split would need to happen so that no files are split up. Any ideas whether and how this could be done / assisted by btrfs? Thanks, Chris. [0] Obviously there is a problem, when a single file would be larger than a BluRay, but so far this hasn't happened to me. If so, I'd of course need to split the file. smime.p7s Description: S/MIME cryptographic signature
Re: incremental full file backups to smaller mediums possible?
On Thu, Apr 09, 2015 at 06:14:33PM +0200, Christoph Anton Mitterer wrote: Hey. I wondered whether this is possible in btrfs (or could be implemented),... it's in a way similar to send/receive, but AFAIU not fully solvable with that. What I want to do is making incremental backups of a (btrfs) filesystem to smaller mediums (that is for example: from a big RAID filesystem to many BluRays or similar things). Right now I make this as follows: Every time backups should be made, I create a ext4 image which would just fit inside the UDF fs of a BluRay (the ext4 image in turn is dm-crypt encrypted) and copy as many *complete*[0] files from the source filesystem to that image as possible. That image file is then burned to BluRay. I also write an SHA512 sum of the whole image file to the BluRay so that one can check whether it can be still read correctly. Then the process continues with the remaining files. The main idea behind writing only complete files to the separate parts of the backup is, that I don't want to have a backup set, where I may get into troubles if just one of the disks fails. Obviously, when I would do regularly backups that way, then I'd write each time a big amount of data which hasn't changed at all, which is not just a costly issue, but also quite time consuming. So ideally, I'd want to do just an incremental backup of all the files that have been added/modified + the information of what has been deleted/moved/changed it's properties. All that already smells quite a lot after making a snapshot at the last backup, and then doing send/receive for the next one. Now the problem, though, is, that I don't want to loose the property of working in terms of complete files (and I guess send/recieve does not), i.e. when a big file has just one block changed, I still would like to have the complete file on the next incremental dump (and not just information about the diff). The only exception is when file metadata changes (i.e. pathname, date, permissions, xattrs, etc.). If *just* one of these changes I wouldn't want to backup the full raw data of the file again. Any ideas whether and how this could be done / assisted by btrfs? btrfs sub find-new might be more helpful to you here. That will give you the list of changed files; then just feed that list to your existing bin-packing algorithm for working out what goes on which disks, and you're done. Hugo. -- Hugo Mills | Dullest spy film ever: The Eastbourne Ultimatum hugo@... carfax.org.uk | http://carfax.org.uk/ | PGP: E2AB1DE4 | The Thick of It signature.asc Description: Digital signature
