Re: Copying a disk containing a btrfs filesystem
On Friday 11 April 2014 12:39:31 Brendan Hide wrote: If you're 100% happy with your old disk's *content*/layout/etc (just not happy with the disk's reliability), try an overnight/over-weekend ddrescue instead Thanks again to everyone who replied and especially for suggesting ddrescue. In the meantime, I've got a suitable new disk and copied the data to it using ddrescue (it took about 12 hours for 1TB). The disk had one fault extending over a few successive sectors. For some reason, I had to repair the first partition, which I use for swap. The large second with btrfs on it seems fine. I ran btrfs scrub on it and it found one error: Apr 16 09:48:27 fuchsia kernel: [ 6792.829186] btrfs: checksum error at logical 74443923456 on dev /dev/sdb2, sector 145398288, root 1228, inode 59102093, offset 929792, length 4096, links 1 (path: usr/lib/debug/.build-id/8f/b82df57b7b6fff7033f6abd7de914b82f98160.debug) That file is part of a historical snapshot which I have deleted by now and thus presumably dealt with the problem. Michael -- Michael Schuerig mailto:mich...@schuerig.de http://www.schuerig.de/michael/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Copying a disk containing a btrfs filesystem
SMART indicates that my notebook disk may soon be failing (an unreadable/uncorrectable sector), therefore I intend to exchange it. The disk contains a single btrfs filesystem with several nested(!) subvolumes, each with several read-only snapshots in a .snapshots subdirectory. As far as I can tell, btrfs currently does not offer a sensible way to duplicate the entire contents of the old disk onto a new one. I can use cp, rsync, or send/receive to copy the main subvolumes. But unless I'm missing something obvious, the snapshots are effectively lost. btrfs send optionally takes multiple clone sources, but I've never seen an example of its usage. If that's what experimental means, I'm willing to accept it. However, I'd like to emphasize that there's still something missing. Of course, most of all I'd like to be proved wrong. Michael -- Michael Schuerig mailto:mich...@schuerig.de http://www.schuerig.de/michael/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Copying a disk containing a btrfs filesystem
On Thursday 10 April 2014 13:58:45 Duncan wrote: Michael Schuerig posted on Thu, 10 Apr 2014 15:21:01 +0200 as excerpted: SMART indicates that my notebook disk may soon be failing (an unreadable/uncorrectable sector), therefore I intend to exchange it. The disk contains a single btrfs filesystem with several nested(!) subvolumes, each with several read-only snapshots in a .snapshots subdirectory. As far as I can tell, btrfs currently does not offer a sensible way to duplicate the entire contents of the old disk onto a new one. I can use cp, rsync, or send/receive to copy the main subvolumes. But unless I'm missing something obvious, the snapshots are effectively lost. btrfs send optionally takes multiple clone sources, but I've never seen an example of its usage. If that's what experimental means, I'm willing to accept it. However, I'd like to emphasize that there's still something missing. Of course, most of all I'd like to be proved wrong. It's not a btrfs tool, but several of the tools you mentioned aren't, and you didn't mention dd (or ddrescue, if your source device starts giving you issues while you're cloning). Using it you'd clone the entire raw device, including any not yet allocated areas, in a straight-across bit- perfect copy. Of course you'd need a target of either the same size or larger in ordered to do so... That should give you a bit-perfect copy including the filesystem UUIDs, etc, which will confuse btrfs if you try to mount anything btrfs with both devices attached, so don't. Do your clone, then umount and disconnect the old device before trying to mount the new one. Thanks for pointing out dd and especially ddrescue, which I didn't know. I regularly use dd to copy images onto USB thumb drives, but I'm a bit wary regarding its use on hard disks. I've always been concerned that if I get another disk, not necessarily of the same make, that has nominally the same capacity, it still might be a few blocks smaller. Well, I just found one that has the right size, apparently. Thanks again, Michael -- Michael Schuerig mailto:mich...@schuerig.de http://www.schuerig.de/michael/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Copying a disk containing a btrfs filesystem
On Thursday 10 April 2014 15:15:02 Duncan wrote: Meanwhile (2), given the existence of those tested backups, there's yet another way to accomplish things. Simply restore from the backups the same way you would if the working copy went down and you had to restore it, only restore to the new device instead of the old one. =:^) As the OP, let me insist that I have multiple backups. However and unfortunately, those backups do not contain the snapshots that I'd like to preserve when exchanging the disk. What makes the case complicate is not the question how to preserve and copy the current data; it's how to retain the historic data embodied in snapshots. Michael -- Michael Schuerig mailto:mich...@schuerig.de http://www.schuerig.de/michael/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Copying a disk containing a btrfs filesystem
On Thursday 10 April 2014 18:01:31 George Eleftheriou wrote: What makes the case complicate is not the question how to preserve and copy the current data; it's how to retain the historic data embodied in snapshots. You can always rsync (incrementally with --link-dest) to another place the sequence of snapshots, provided of course there is enough space in this other place. However I understand this may not be the coolest thing to do especially taking into account the SMART warnings for imminent deterioration. In the case at hand, I'll go with ddrescue. As far as backups are concerned, rsync --link-dest may be an option, however even at best it would only give file-granularity COW as opposed to block-granularity. I'm not sure if this would work for me in terms of disk space consumption. I *think* send/receive with clone sources might be the key to a solution. I'm still hoping that someone with a far better understanding of btrfs than me gives it a try first... Michael -- Michael Schuerig mailto:mich...@schuerig.de http://www.schuerig.de/michael/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Copying a disk containing a btrfs filesystem
On Thursday 10 April 2014 19:17:13 Jan Kouba wrote: Dne Čt 10. dubna 2014 15:21:01, Michael Schuerig napsal(a): SMART indicates that my notebook disk may soon be failing (an unreadable/uncorrectable sector), therefore I intend to exchange it. The disk contains a single btrfs filesystem with several nested(!) subvolumes, each with several read-only snapshots in a .snapshots subdirectory. As far as I can tell, btrfs currently does not offer a sensible way to duplicate the entire contents of the old disk onto a new one. Yes it does You can make the old disk a seeding device and use it to seed the new one like this: Intriguing. I didn't find much on the topic, but these pages seem pertinent http://en.wikipedia.org/wiki/Btrfs#Seed_devices https://btrfs.wiki.kernel.org/index.php/Seed-device http://unix.stackexchange.com/a/97819/64935 btrfstune -S 1 old_device Does the 1 mean true or is this indeed an integer-value arg? I've only seen the 1 in examples, are there other possible values? mount old_device /mnt # this will be mounted read-only btrfs dev add new_device /mnt mount -o remount,rw /mnt btrfs dev delete old_device Is the last one the step where data is copied from the seed device to the second device? No balance required before deleting the device? Thanks! Michael -- Michael Schuerig mailto:mich...@schuerig.de http://www.schuerig.de/michael/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
SELinux on btrfs
I'm currently considering to use SELinux on an existing system with btrfs filesystems. This would be my first with SELinux and I wouldn't expect everything to go smoothly. I'm already aware that SELinux's automatic labelling of files is not aware of subvolumes[*]. I already have quite a few read-only snapshots that I don't want to forfeit, however, I'm not at all sure how SELinux would interact with them. Are there any other considerations I ought to be aware of? Michael [*] https://wiki.debian.org/SELinux/Setup#btrfs -- Michael Schuerig mailto:mich...@schuerig.de http://www.schuerig.de/michael/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Snapper on Ubuntu
On Saturday 15 March 2014 15:05:22 Hendrik Friedel wrote: Hello, I am not sure, whether this is the right place to ask this question -if not, please advise. Ubuntu installs on btrfs, creating subvolumes for the homes (/home), the root home (/root) and the root (/) named @home, @root and @ respectively. When I install snapper I configure it like this snapper -c rt create-config / snapper -c home create-config /home snapper -c root create-config /root snapper -c Video create-config /mnt/BTRFS/Video/ After executing snapper create several times, this results in #btrfs subvolume list / ID 258 gen 2615 top level 5 path @ ID 259 gen 2611 top level 5 path @root ID 260 gen 2555 top level 5 path @home ID 281 gen 2555 top level 5 path @home/.snapshots ID 282 gen 2606 top level 5 path @root/.snapshots ID 283 gen 2562 top level 5 path @root/.snapshots/1/snapshot ID 284 gen 2563 top level 5 path @root/.snapshots/2/snapshot ID 285 gen 2573 top level 5 path @root/.snapshots/3/snapshot [...] So, this all works for @root only, not for the other subvolumes. Do you have any suggestions, how to find the cause? I think you may have forgotten to specify the config snapper is supposed to use. Try # snapper -c home create # snapper -c Video create Michael -- Michael Schuerig mailto:mich...@schuerig.de http://www.schuerig.de/michael/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Incremental backup for a raid1
On Friday 14 March 2014 06:42:27 Duncan wrote: N-way-mirroring is actually my most hotly anticipated feature for a different reason[2], but for you it would work like this: 1) Setup the 3-way (or 4-way if preferred) mirroring and balance to ensured copies of all data on all devices. 2) Optionally scrub to ensure the integrity of all copies. 3) Disconnect the backup device(s). (Don't btrfs device delete, this would remove the copy. Just disconnect.) 4) Store the backups. 5) Periodically get them out and reconnect. 6) Rebalance to update. (Since the devices remain members of the mirror, simply outdated, the balance should only update, not rewrite the entire thing.) 7) Optionally scrub to verify. 8) Repeat steps 3-7 as necessary. Judging from your description, N-way mirroring is (going to be) exactly what I was hoping for. Michael -- Michael Schuerig mailto:mich...@schuerig.de http://www.schuerig.de/michael/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Incremental backup for a raid1
My backup use case is different from the what has been recently discussed in another thread. I'm trying to guard against hardware failure and other causes of destruction. I have a btrfs raid1 filesystem spread over two disks. I want to backup this filesystem regularly and efficiently to an external disk (same model as the ones in the raid) in such a way that * when one disk in the raid fails, I can substitute the backup and rebalancing from the surviving disk to the substitute only applies the missing changes. * when the entire raid fails, I can re-build a new one from the backup. The filesystem is mounted at its root and has several nested subvolumes and snapshots (in a .snapshots subdir on each subvol). Is it possible to do what I'm looking for? Michael -- Michael Schuerig mailto:mich...@schuerig.de http://www.schuerig.de/michael/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Incremental backup for a raid1
On Thursday 13 March 2014 14:48:55 Andrew Skretvedt wrote: On 2014-Mar-13 14:28, Hugo Mills wrote: On Thu, Mar 13, 2014 at 08:12:44PM +0100, Michael Schuerig wrote: My backup use case is different from the what has been recently discussed in another thread. I'm trying to guard against hardware failure and other causes of destruction. I have a btrfs raid1 filesystem spread over two disks. I want to backup this filesystem regularly and efficiently to an external disk (same model as the ones in the raid) in such a way that * when one disk in the raid fails, I can substitute the backup and rebalancing from the surviving disk to the substitute only applies the missing changes. * when the entire raid fails, I can re-build a new one from the backup. The filesystem is mounted at its root and has several nested subvolumes and snapshots (in a .snapshots subdir on each subvol). [...] I'm new; btrfs noob; completely unqualified to write intelligently on this topic, nevertheless: I understand your setup to be btrfs RAID1 with /dev/A /dev/B, and a backup device someplace /dev/C Could you, at the time you wanted to backup the filesystem: 1) in the filesystem, break RAID1: /dev/A /dev/B -- remove /dev/B 2) reestablish RAID1 to the backup device: /dev/A /dev/C -- added 3) balance to effect the backup (i.e. rebuilding the RAID1 onto /dev/C) 4) break/reconnect the original devices: remove /dev/C; re-add /dev/B to the fs I've thought of this but don't dare try it without approval from the experts. At any rate, for being practical, this approach hinges on an ability to rebuild the raid1 incrementally. That is, the rebuild would have to start from what already is present on disk B (or C, when it is re-added). Starting from an effectively blank disk each time would be prohibitive. Even if this would work, I'd much prefer keeping the original raid1 intact and to only temporarily add another mirror: lazy mirroring, to give the thing a name. Michael -- Michael Schuerig mailto:mich...@schuerig.de http://www.schuerig.de/michael/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Incremental backup for a raid1
On Thursday 13 March 2014 16:04:33 Chris Murphy wrote: On Mar 13, 2014, at 3:14 PM, Michael Schuerig michael.li...@schuerig.de wrote: On Thursday 13 March 2014 14:48:55 Andrew Skretvedt wrote: On 2014-Mar-13 14:28, Hugo Mills wrote: On Thu, Mar 13, 2014 at 08:12:44PM +0100, Michael Schuerig wrote: My backup use case is different from the what has been recently discussed in another thread. I'm trying to guard against hardware failure and other causes of destruction. I have a btrfs raid1 filesystem spread over two disks. I want to backup this filesystem regularly and efficiently to an external disk (same model as the ones in the raid) in such a way that * when one disk in the raid fails, I can substitute the backup and rebalancing from the surviving disk to the substitute only applies the missing changes. * when the entire raid fails, I can re-build a new one from the backup. The filesystem is mounted at its root and has several nested subvolumes and snapshots (in a .snapshots subdir on each subvol). [...] I'm new; btrfs noob; completely unqualified to write intelligently on this topic, nevertheless: I understand your setup to be btrfs RAID1 with /dev/A /dev/B, and a backup device someplace /dev/C Could you, at the time you wanted to backup the filesystem: 1) in the filesystem, break RAID1: /dev/A /dev/B -- remove /dev/B 2) reestablish RAID1 to the backup device: /dev/A /dev/C -- added 3) balance to effect the backup (i.e. rebuilding the RAID1 onto /dev/C) 4) break/reconnect the original devices: remove /dev/C; re-add /dev/B to the fs I've thought of this but don't dare try it without approval from the experts. At any rate, for being practical, this approach hinges on an ability to rebuild the raid1 incrementally. That is, the rebuild would have to start from what already is present on disk B (or C, when it is re-added). Starting from an effectively blank disk each time would be prohibitive. Even if this would work, I'd much prefer keeping the original raid1 intact and to only temporarily add another mirror: lazy mirroring, to give the thing a name. [...] In the btfs device add case, you now have a three disk raid1 which is a whole different beast. Since this isn't n-way raid1, each disk is not stand alone. You're only assured data survives a one disk failure meaning you must have two drives. Yes, I understand that. Unless someone convinces me that it's a bad idea, I keep wishing for a feature that allows to intermittently add a third disk to a two disk raid1 and update that disk so that it could replace one of the others. So the btrfs replace scenario might work but it seems like a bad idea. And overall it's a use case for which send/receive was designed anyway so why not just use that? Because it's not just. Doing it right doesn't seem trivial. For one thing, there are multiple subvolumes; not at the top-level but nested inside a root subvolume. Each of them already has snapshots of its own. If there already is a send/receive script that can handle such a setup I'll happily have a look at it. Michael -- Michael Schuerig mailto:mich...@schuerig.de http://www.schuerig.de/michael/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
