Re: Copying a disk containing a btrfs filesystem
On Friday 11 April 2014 12:39:31 Brendan Hide wrote: If you're 100% happy with your old disk's *content*/layout/etc (just not happy with the disk's reliability), try an overnight/over-weekend ddrescue instead Thanks again to everyone who replied and especially for suggesting ddrescue. In the meantime, I've got a suitable new disk and copied the data to it using ddrescue (it took about 12 hours for 1TB). The disk had one fault extending over a few successive sectors. For some reason, I had to repair the first partition, which I use for swap. The large second with btrfs on it seems fine. I ran btrfs scrub on it and it found one error: Apr 16 09:48:27 fuchsia kernel: [ 6792.829186] btrfs: checksum error at logical 74443923456 on dev /dev/sdb2, sector 145398288, root 1228, inode 59102093, offset 929792, length 4096, links 1 (path: usr/lib/debug/.build-id/8f/b82df57b7b6fff7033f6abd7de914b82f98160.debug) That file is part of a historical snapshot which I have deleted by now and thus presumably dealt with the problem. Michael -- Michael Schuerig mailto:mich...@schuerig.de http://www.schuerig.de/michael/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Copying a disk containing a btrfs filesystem
Hi, Michael Btrfs send/receive can transfer incremental snapshots as well - you're looking for the -p or parent parameter. On the other hand, it might not be the right tool for the job. If you're 100% happy with your old disk's *content*/layout/etc (just not happy with the disk's reliability), try an overnight/over-weekend ddrescue instead: http://www.forensicswiki.org/wiki/Ddrescue What I've done in the past is scripted ddrescue to recover as much data as possible. Its like using dd between two disks except that it can keep a log of bad sectors that can be retried later. The log also helps by ensuring that if you cancel the operation, you can start it again and it will continue where it left off. Additionally, you can have it skip big sections of the disk when it comes across bad sectors - and it can trim these sections on subsequent runs. Btrfs send/receive has the advantage that you can run it while your system is still active. DDrescue has the advantage that it is very good at recovering 99% of your data where a disk has lots of bad sectors. For btrfs, send/receive the main subvolumes then, afterward, send/receive the snapshots using the parent parameter, -p. There *is* the possibility that this needs to be reversed - as in, the backup should be treated as the parent instead of the other way around: btrfs send /home | btrfs receive /mnt/new-disk/home btrfs send -p /home /backups/home-2014-04-08 | btrfs receive /mnt/new-disk/backups/. Below is from the last scriptlet I made when I last did the ddrescue route (in that case I was recovering a failing NTFS drive). It was particularly bad and took a whole weekend to recover. The new disk worked flawlessly however. :) How I've used ddrescue in the past is to connect the failing and new disk to a server. Alternatively, using USB, you could boot from a rescue CD/flash drive and do the rescue there. a) Identify the disks in /dev/disk/by-whatever-you-choose and put those values into the bash script below. ensure that it refers to the disk as a whole (not a partition for example). This ensures that re-ordering of the drives after a reboot won't affect the process. b) Set up a log file location on a separate filesystem - a flash drive is ideal unless you've gone the server route, where I normally just put the log into a path on the the server as so: /root/brendan-laptop.recovery.log #!/bin/bash src_disk=/dev/disk/by-id/ata-ST3250410AS_6RYF5NP7 dst_disk=/dev/disk/by-id/ata-ST3500418AS_9VM2ZZQS #log=/path/to/log log=~/brendan-laptop.recovery.log #Sector size (default is 512 - newer disks should probably be 4096) sector_size=4096 #Force writing to a block device - disable if you're backing up to an image file #force= force=-f # We want to skip bigger chunks to get as much data as possible before the source disk really dies # For the same reason, we also want to start with (attempting) to maintain a high read rate #Minimum read rate in MB/s before skipping min_readrate=10485760 #Default skip size is 64k. for skip_size in 65536 16384; do #Going forward ddrescue -r 1 -a $min_readrate -b $sector_size -d $force -K $skip_size $src_disk $dst_disk $log #Going in reverse ddrescue -r 1 -a $min_readrate -b $sector_size -d $force -K $skip_size -R $src_disk $dst_disk $log done #Default skip size is 64k. #This time re-trying all failed/skipped sections for skip_size in 4096; do #Going forward ddrescue -r 1 -a $min_readrate -b $sector_size -d $force -K $skip_size -A $src_disk $dst_disk $log #Going in reverse ddrescue -r 1 -a $min_readrate -b $sector_size -d $force -K $skip_size -R -A $src_disk $dst_disk $log done #Default skip size is 64k. for skip_size in 1024 256 64 ; do #Going forward ddrescue -r 1 -b $sector_size -d $force -K $skip_size $src_disk $dst_disk $log #Going in reverse ddrescue -r 1 -b $sector_size -d $force -K $skip_size -R $src_disk $dst_disk $log done echo Done. Run an chkdsk/fsck/whatever-might-be appropriate for the new disk's filesystem(s) On 10/04/14 15:21, Michael Schuerig wrote: SMART indicates that my notebook disk may soon be failing (an unreadable/uncorrectable sector), therefore I intend to exchange it. The disk contains a single btrfs filesystem with several nested(!) subvolumes, each with several read-only snapshots in a .snapshots subdirectory. As far as I can tell, btrfs currently does not offer a sensible way to duplicate the entire contents of the old disk onto a new one. I can use cp, rsync, or send/receive to copy the main subvolumes. But unless I'm missing something obvious, the snapshots are effectively lost. btrfs send optionally takes multiple clone sources, but I've never seen an example of its usage. If that's what experimental means, I'm willing to accept it. However, I'd like to emphasize that there's still something missing. Of course, most of all I'd like to be proved wrong. Michael -- __ Brendan Hide
Re: Copying a disk containing a btrfs filesystem
Michael Schuerig posted on Thu, 10 Apr 2014 15:21:01 +0200 as excerpted: SMART indicates that my notebook disk may soon be failing (an unreadable/uncorrectable sector), therefore I intend to exchange it. The disk contains a single btrfs filesystem with several nested(!) subvolumes, each with several read-only snapshots in a .snapshots subdirectory. As far as I can tell, btrfs currently does not offer a sensible way to duplicate the entire contents of the old disk onto a new one. I can use cp, rsync, or send/receive to copy the main subvolumes. But unless I'm missing something obvious, the snapshots are effectively lost. btrfs send optionally takes multiple clone sources, but I've never seen an example of its usage. If that's what experimental means, I'm willing to accept it. However, I'd like to emphasize that there's still something missing. Of course, most of all I'd like to be proved wrong. It's not a btrfs tool, but several of the tools you mentioned aren't, and you didn't mention dd (or ddrescue, if your source device starts giving you issues while you're cloning). Using it you'd clone the entire raw device, including any not yet allocated areas, in a straight-across bit- perfect copy. Of course you'd need a target of either the same size or larger in ordered to do so... That should give you a bit-perfect copy including the filesystem UUIDs, etc, which will confuse btrfs if you try to mount anything btrfs with both devices attached, so don't. Do your clone, then umount and disconnect the old device before trying to mount the new one. In fact, there are entire purpose-built special live-image distributions such as Clonezilla for managing storage devices like this. http://clonezilla.org/ Or use a more general purpose rescue live-image distro such as SysRescue: http://www.sysresccd.org/ Both of these tools support MS Windows systems as well. Image-cloning is of course OS-agnostic -- what's on the cloned image doesn't matter, just whether the source device is readable and the destination writable. And sysrescue is often used in the MS world to reset lost passwords and etc. I used to use it for that back when I still worked on friends' MS systems for gratis (well, traded for a good dinner or whatever), before I decided that wasn't a worthwhile use of my time, tho I'd be happy to teach them about Linux if they wanted, or might still do it if they paid me enough, but then there's other computer repair services that will do that if they're willing to pay. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Copying a disk containing a btrfs filesystem
I would see one (dangerous? risky? needing more options perhaps?) solution: dd if=/dev/sda of=/dev/sdb /dev/sda: old disk /dev/sdb: new disk Maybe there is another much more complicated solution: Plug the old disk in a USB dock/case, do the same for the new disk in another dock/case, plug both docks/cases in another linux system running recent kernel and btrfs-progs and convert to a -dconvert=raid1 -mconvert=raid1 profile with a balance. Then degrade it by removing the old disk and rebalance-convert back to single or DUP profile (i am not quite sure this is even possible). Just an idea. I wouldn't trust me. On Thu, Apr 10, 2014 at 3:21 PM, Michael Schuerig michael.li...@schuerig.de wrote: SMART indicates that my notebook disk may soon be failing (an unreadable/uncorrectable sector), therefore I intend to exchange it. The disk contains a single btrfs filesystem with several nested(!) subvolumes, each with several read-only snapshots in a .snapshots subdirectory. As far as I can tell, btrfs currently does not offer a sensible way to duplicate the entire contents of the old disk onto a new one. I can use cp, rsync, or send/receive to copy the main subvolumes. But unless I'm missing something obvious, the snapshots are effectively lost. btrfs send optionally takes multiple clone sources, but I've never seen an example of its usage. If that's what experimental means, I'm willing to accept it. However, I'd like to emphasize that there's still something missing. Of course, most of all I'd like to be proved wrong. Michael -- Michael Schuerig mailto:mich...@schuerig.de http://www.schuerig.de/michael/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Copying a disk containing a btrfs filesystem
On Thursday 10 April 2014 13:58:45 Duncan wrote: Michael Schuerig posted on Thu, 10 Apr 2014 15:21:01 +0200 as excerpted: SMART indicates that my notebook disk may soon be failing (an unreadable/uncorrectable sector), therefore I intend to exchange it. The disk contains a single btrfs filesystem with several nested(!) subvolumes, each with several read-only snapshots in a .snapshots subdirectory. As far as I can tell, btrfs currently does not offer a sensible way to duplicate the entire contents of the old disk onto a new one. I can use cp, rsync, or send/receive to copy the main subvolumes. But unless I'm missing something obvious, the snapshots are effectively lost. btrfs send optionally takes multiple clone sources, but I've never seen an example of its usage. If that's what experimental means, I'm willing to accept it. However, I'd like to emphasize that there's still something missing. Of course, most of all I'd like to be proved wrong. It's not a btrfs tool, but several of the tools you mentioned aren't, and you didn't mention dd (or ddrescue, if your source device starts giving you issues while you're cloning). Using it you'd clone the entire raw device, including any not yet allocated areas, in a straight-across bit- perfect copy. Of course you'd need a target of either the same size or larger in ordered to do so... That should give you a bit-perfect copy including the filesystem UUIDs, etc, which will confuse btrfs if you try to mount anything btrfs with both devices attached, so don't. Do your clone, then umount and disconnect the old device before trying to mount the new one. Thanks for pointing out dd and especially ddrescue, which I didn't know. I regularly use dd to copy images onto USB thumb drives, but I'm a bit wary regarding its use on hard disks. I've always been concerned that if I get another disk, not necessarily of the same make, that has nominally the same capacity, it still might be a few blocks smaller. Well, I just found one that has the right size, apparently. Thanks again, Michael -- Michael Schuerig mailto:mich...@schuerig.de http://www.schuerig.de/michael/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Copying a disk containing a btrfs filesystem
On Thursday 10 April 2014 15:15:02 Duncan wrote: Meanwhile (2), given the existence of those tested backups, there's yet another way to accomplish things. Simply restore from the backups the same way you would if the working copy went down and you had to restore it, only restore to the new device instead of the old one. =:^) As the OP, let me insist that I have multiple backups. However and unfortunately, those backups do not contain the snapshots that I'd like to preserve when exchanging the disk. What makes the case complicate is not the question how to preserve and copy the current data; it's how to retain the historic data embodied in snapshots. Michael -- Michael Schuerig mailto:mich...@schuerig.de http://www.schuerig.de/michael/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Copying a disk containing a btrfs filesystem
What makes the case complicate is not the question how to preserve and copy the current data; it's how to retain the historic data embodied in snapshots. You can always rsync (incrementally with --link-dest) to another place the sequence of snapshots, provided of course there is enough space in this other place. However I understand this may not be the coolest thing to do especially taking into account the SMART warnings for imminent deterioration. On Thu, Apr 10, 2014 at 5:15 PM, Duncan 1i5t5.dun...@cox.net wrote: claim, I really should have thought of and mentioned the restoring from backups method first, instead of having it occur to me as an after- thought. I must be getting sloppy and careless in my thinking! =:^( ) Maybe this is related to the fact that btrfs is maturing, gradually causing you a sense of complete safety ;-) -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Copying a disk containing a btrfs filesystem
On Thursday 10 April 2014 18:01:31 George Eleftheriou wrote: What makes the case complicate is not the question how to preserve and copy the current data; it's how to retain the historic data embodied in snapshots. You can always rsync (incrementally with --link-dest) to another place the sequence of snapshots, provided of course there is enough space in this other place. However I understand this may not be the coolest thing to do especially taking into account the SMART warnings for imminent deterioration. In the case at hand, I'll go with ddrescue. As far as backups are concerned, rsync --link-dest may be an option, however even at best it would only give file-granularity COW as opposed to block-granularity. I'm not sure if this would work for me in terms of disk space consumption. I *think* send/receive with clone sources might be the key to a solution. I'm still hoping that someone with a far better understanding of btrfs than me gives it a try first... Michael -- Michael Schuerig mailto:mich...@schuerig.de http://www.schuerig.de/michael/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Copying a disk containing a btrfs filesystem
Dne Čt 10. dubna 2014 15:21:01, Michael Schuerig napsal(a): SMART indicates that my notebook disk may soon be failing (an unreadable/uncorrectable sector), therefore I intend to exchange it. The disk contains a single btrfs filesystem with several nested(!) subvolumes, each with several read-only snapshots in a .snapshots subdirectory. As far as I can tell, btrfs currently does not offer a sensible way to duplicate the entire contents of the old disk onto a new one. Yes it does You can make the old disk a seeding device and use it to seed the new one like this: btrfstune -S 1 old_device mount old_device /mnt # this will be mounted read-only btrfs dev add new_device /mnt mount -o remount,rw /mnt btrfs dev delete old_device According to my experiments, the filesystem on the new device will have different UUID, so if you are mounting using UUIDs, you must change it everywhere, but other than that the new filesystem should have the same content as the old one (including subvolumes). I can use cp, rsync, or send/receive to copy the main subvolumes. But unless I'm missing something obvious, the snapshots are effectively lost. btrfs send optionally takes multiple clone sources, but I've never seen an example of its usage. If that's what experimental means, I'm willing to accept it. However, I'd like to emphasize that there's still something missing. Of course, most of all I'd like to be proved wrong. Michael -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Copying a disk containing a btrfs filesystem
Am Donnerstag, 10. April 2014, 17:51:26 schrieb Michael Schuerig: On Thursday 10 April 2014 15:15:02 Duncan wrote: Meanwhile (2), given the existence of those tested backups, there's yet another way to accomplish things. Simply restore from the backups the same way you would if the working copy went down and you had to restore it, only restore to the new device instead of the old one. =:^) As the OP, let me insist that I have multiple backups. However and unfortunately, those backups do not contain the snapshots that I'd like to preserve when exchanging the disk. What makes the case complicate is not the question how to preserve and copy the current data; it's how to retain the historic data embodied in snapshots. Well I think this is scriptable – although this would be some work. Like this – pseudo shell code without verifying exact argument order: - for each subvolume create according subvolume in destination - for each snapshot – from oldest to newest: - rsync / btrfs send snapshot - snapshot with same name - rsync next newer snapshot I have snapshots on backup drive… so I´d probably ditch snapshots on production side… but if you want them… thats another idea. Although I like the balance idea as well. I used it to RAID-1 my /home to dual SSD setup. But I didn´t to the degrading step. Ciao, -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Copying a disk containing a btrfs filesystem
On Thursday 10 April 2014 19:17:13 Jan Kouba wrote: Dne Čt 10. dubna 2014 15:21:01, Michael Schuerig napsal(a): SMART indicates that my notebook disk may soon be failing (an unreadable/uncorrectable sector), therefore I intend to exchange it. The disk contains a single btrfs filesystem with several nested(!) subvolumes, each with several read-only snapshots in a .snapshots subdirectory. As far as I can tell, btrfs currently does not offer a sensible way to duplicate the entire contents of the old disk onto a new one. Yes it does You can make the old disk a seeding device and use it to seed the new one like this: Intriguing. I didn't find much on the topic, but these pages seem pertinent http://en.wikipedia.org/wiki/Btrfs#Seed_devices https://btrfs.wiki.kernel.org/index.php/Seed-device http://unix.stackexchange.com/a/97819/64935 btrfstune -S 1 old_device Does the 1 mean true or is this indeed an integer-value arg? I've only seen the 1 in examples, are there other possible values? mount old_device /mnt # this will be mounted read-only btrfs dev add new_device /mnt mount -o remount,rw /mnt btrfs dev delete old_device Is the last one the step where data is copied from the seed device to the second device? No balance required before deleting the device? Thanks! Michael -- Michael Schuerig mailto:mich...@schuerig.de http://www.schuerig.de/michael/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Copying a disk containing a btrfs filesystem
Dne Čt 10. dubna 2014 20:36:32, Michael Schuerig napsal(a): On Thursday 10 April 2014 19:17:13 Jan Kouba wrote: Dne Čt 10. dubna 2014 15:21:01, Michael Schuerig napsal(a): SMART indicates that my notebook disk may soon be failing (an unreadable/uncorrectable sector), therefore I intend to exchange it. The disk contains a single btrfs filesystem with several nested(!) subvolumes, each with several read-only snapshots in a .snapshots subdirectory. As far as I can tell, btrfs currently does not offer a sensible way to duplicate the entire contents of the old disk onto a new one. Yes it does You can make the old disk a seeding device and use it to seed the new one like this: Intriguing. I didn't find much on the topic, but these pages seem pertinent http://en.wikipedia.org/wiki/Btrfs#Seed_devices https://btrfs.wiki.kernel.org/index.php/Seed-device http://unix.stackexchange.com/a/97819/64935 btrfstune -S 1 old_device Does the 1 mean true or is this indeed an integer-value arg? I've only seen the 1 in examples, are there other possible values? Yes, 1 means true, 0 false. If you run it twice with the same argument, it says something like Seeding flag already set/unset.. mount old_device /mnt # this will be mounted read-only btrfs dev add new_device /mnt mount -o remount,rw /mnt btrfs dev delete old_device Is the last one the step where data is copied from the seed device to the second device? No balance required before deleting the device? Yes, everything is copied, no balance needed. Well actually if you don't need to preserve the filesystem on the old device, you can just add the new device into the filesystem and remove the old one like this: mount old_device /mnt btrfs dev add new_device /mnt btrfs dev delete old_device /mnt The last command will copy everything from old_device to the new_device and somehow wipe the old_device so it can't be mounted anymore. This method will also preserve the UUID of the filesystem so there is no need to change any config files. Jan Kouba -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Copying a disk containing a btrfs filesystem
Michael Schuerig posted on Thu, 10 Apr 2014 18:24:00 +0200 as excerpted: I *think* send/receive with clone sources might be the key to a solution. I'm still hoping that someone with a far better understanding of btrfs than me gives it a try first... Well, there's both that (which I don't know enough about to discuss in any intelligent way at all), and the fact that at least until very (very) recently (like recent enough I'm not sure it'll all be in 2.15 yet), btrfs send/receive still had some serious corner-case bugs, such that to my knowledge if it worked without error it was reliable, there were numbers of cases (like where it was originally subdir /a/b/, and then switched to /b/a, reversing the nesting) where it would spit out errors, leaving the receive in a half-received state. So it worked for some people but not others, and for some it would work for awhile, then would start erroring out, whenever they hit whatever corner-case. Regulars on this list will have seen all the recent work going into send/ receive bug tracing and fixing, however, which will of course ultimately make it far more reliable than it had been. Surely it's already far more reliable, as a fair number of those corner cases have now been found and dealt with. But what I do NOT know yet, is where in the process we are, in terms of /reliable/ send/receive. Relatively, we're certainly closer than we were, but did that move use from say 80% to 90% reliable, or from 80% to 97 or 98% reliable (which is about where I'd put general btrfs at this point), or from 90% to 98-ish% reliable, or... ? Without that knowledge, even if you did know the specific CL options you needed, I'd still consider it very much a great if it works, but don't count on it until you actually see it complete without errors, and even then, be aware that the next time you try it could error out, because it's still very much in bug-fixing-phase solution. Meanwhile, the ddrescue solution is mature technology, proven to work with many different types of filesystems over many years, now, so that's the basket I'd be putting my resource eggs into at this point. =:^) -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
