Re: How to handle a RAID5 arrawy with a failing drive? - raid5 mostly works, just no rebuilds
On Wed, Mar 19, 2014 at 10:53:33AM -0600, Chris Murphy wrote: On Mar 19, 2014, at 9:40 AM, Marc MERLIN m...@merlins.org wrote: After adding a drive, I couldn't quite tell if it was striping over 11 drive2 or 10, but it felt that at least at times, it was striping over 11 drives with write failures on the missing drive. I can't prove it, but I'm thinking the new data I was writing was being striped in degraded mode. Well it does sound fragile after all to add a drive to a degraded array, especially when it's not expressly treating the faulty drive as faulty. I think iotop will show what block devices are being written to. And in a VM it's easy (albeit rudimentary) with sparse files, as you can see them grow. Yes, although it's limited, you apparently only lose new data that was added after you went into degraded mode and only if you add another drive where you write more data. In real life this shouldn't be too common, even if it is indeed a bug. It's entirely plausible a drive power/data cable becomes lose, runs for hours degraded before the wayward device is reseated. It'll be common enough. It's definitely not OK for all of that data in the interim to vanish just because the volume has resumed from degraded to normal. Two states of data, normal vs degraded, is scary. It sounds like totally silent data loss. So yeah if it's reproducible it's worthy of a separate bug. I just got around to filing that bug: https://bugzilla.kernel.org/show_bug.cgi?id=72811 In other news, I was able to 1) remove a drive 2) mount degraded 3) add a new drive 4) rebalance (that took 2 days with little data, 4 deadlocks and reboots though) 5) remove the missing drive from the filesystem 6) remount the array without -o degraded Now, I'm testing 1) add a new drive 2 remove a working drive 3) automatic rebalance from #2 should rebuild on the new drive automatically Marc -- A mouse is a device used to point at the xterm you want to type in - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: How to handle a RAID5 arrawy with a failing drive? - raid5 mostly works, just no rebuilds
Marc MERLIN posted on Wed, 19 Mar 2014 08:40:31 -0700 as excerpted: That's the thing though. If the bad device hadn't been forcibly removed, and apparently the only way to do this was to unmount, make the device node disappear, and remount in degraded mode, it looked to me like btrfs was still consideing that the drive was part of the array and trying to write to it. After adding a drive, I couldn't quite tell if it was striping over 11 drive2 or 10, but it felt that at least at times, it was striping over 11 drives with write failures on the missing drive. I can't prove it, but I'm thinking the new data I was writing was being striped in degraded mode. FWIW, there's at least two problems here, one a bug (or perhaps it'd more accurately be described as an as yet incomplete feature) unrelated to btrfs raid5/6 mode, the other the incomplete raid5/6 support. Both are known issues, however. The incomplete raid5/6 is discussed well enough elsewhere including in this thread as a whole, which leaves the other issue. The other issue, not specifically raid5/6 mode related, is that currently, in-kernel btrfs is basically oblivious to disappearing drives, thus explaining some of the more complex bits of the behavior you described. Yes, the kernel has the device data and other layers know when a device goes missing, but it's basically a case of the right hand not knowing what the left hand is doing -- once setup on a set of devices, in-kernel btrfs basically doesn't do anything with the device information available to it, at least in terms of removing a device from its listing when it goes missing. (It does seem to transparently handle a missing btrfs component device reappearing, arguably /too/ transparently!) Basically all btrfs does is log errors when a component device disappears. It doesn't do anything with the disappeared device, and really doesn't know it has disappeared at all, until an unmount and (possibly degraded) remount, at which point it re-enumerates the devices and again knows what's actually there... until a device disappears again. There's actually patches being worked on to fix that situation as we speak, and it's possible they're actually in btrfs-next already. (I've seen the patches and discussion go by on the list but haven't tracked them to the extent that I know current status, other than that they're not in mainline yet.) Meanwhile, counter-intuitively, btrfs-userspace is sometimes more aware of current device status than btrfs-kernel is ATM, since parts of userspace actually either get current status from the kernel, or trigger a rescan in ordered to get it. But even after a rescan updates what userspace knows and thus what the kernel as a whole knows, btrfs-kernel still doesn't actually use that new information available to it in the same kernel that btrfs-userspace used to get it from! Knowing that rather counterintuitive little inconsistency, that isn't actually so little, goes quite a way toward explaining what otherwise looks like illogical btrfs behavior -- how could kernel-btrfs not know the status of its own devices? -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: How to handle a RAID5 arrawy with a failing drive? - raid5 mostly works, just no rebuilds
I think after the balance it was a fine, non-degraded RAID again... As
far as I remember.
Tobby
2014-03-20 1:46 GMT+01:00 Marc MERLIN m...@merlins.org:
On Thu, Mar 20, 2014 at 01:44:20AM +0100, Tobias Holst wrote:
I tried the RAID6 implementation of btrfs and I looks like I had the
same problem. Rebuild with balance worked but when a drive was
removed when mounted and then readded, the chaos began. I tried it a
few times. So when a drive fails (and this is just because of
connection lost or similar non severe problems), then it is necessary
to wipe the disc first before readding it, so btrfs will add it as a
new disk and not try to readd the old one.
Good to know you got this too.
Just to confirm: did you get it to rebuild, or once a drive is lost/gets
behind, you're in degraded mode forever for those blocks?
Or were you able to balance?
Marc
--
A mouse is a device used to point at the xterm you want to type in - A.S.R.
Microsoft is to operating systems
what McDonalds is to gourmet
cooking
Home page: http://marc.merlins.org/
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: How to handle a RAID5 arrawy with a failing drive? - raid5 mostly works, just no rebuilds
On Tue, Mar 18, 2014 at 09:02:07AM +, Duncan wrote: First just a note that you hijacked Mr Manana's patch thread. Replying (...) I did, I use mutt, I know about in Reply-To, I was tired, I screwed up, sorry, and there was no undo :) Since you don't have to worry about the data I'd suggest blowing it away and starting over. Btrfs raid5/6 code is known to be incomplete at this point, to work in normal mode and write everything out, but with incomplete recovery code. So I'd treat it like the raid-0 mode it effectively is, and consider it lost if a device drops. Which I haven't. My use-case wouldn't be looking at raid5/6 (or raid0) anyway, but even if it were, I'd not touch the current code unless it /was/ just for something I'd consider risking on a raid0. Other than Thank you for the warning, and yes I know the risk and the data I'm putting on it is ok with that risk :) So, I was bit quiet because I diagnosed problems with the underlying hardware. My disk array was creating disk faults due to insufficient power coming in. Now that I fixed that and made sure the drives work with a full run of hdrecover of all the drives in parallel (exercises the drives while making sure all their blocks work), I did tests again: Summary: 1) You can grow and shrink a raid5 volume while it's mounted = very cool 2) shrinking causes a rebalance 3) growing requires you to run rebalance 4) btrfs cannot replace a drive in raid5, whether it's there or not that's the biggest thing missing: just no rebuilds in any way 5) you can mount a raid5 with a missing device with -o degraded 6) adding a drive to a degraded arrays will grow the array, not rebuild the missing bits 7) you can remove a drive from an array, add files, and then if you plug the drive in, it apparently gets auto sucked in back in the array. There is no rebuild that happens, you now have an inconsistent array where one drive is not at the same level than the other ones (I lost all files I added after the drive was removed when I added the drive back). In other words, everything seems to work except there is no rebuild that I could see anywhere. Here are all the details: Creation polgara:/dev/disk/by-id# mkfs.btrfs -f -d raid5 -m raid5 -L backupcopy /dev/mapper/crypt_sd[bdfghijkl]1 WARNING! - Btrfs v3.12 IS EXPERIMENTAL WARNING! - see http://btrfs.wiki.kernel.org before using Turning ON incompat feature 'extref': increased hardlink limit per file to 65536 Turning ON incompat feature 'raid56': raid56 extended format adding device /dev/mapper/crypt_sdd1 id 2 adding device /dev/mapper/crypt_sdf1 id 3 adding device /dev/mapper/crypt_sdg1 id 4 adding device /dev/mapper/crypt_sdh1 id 5 adding device /dev/mapper/crypt_sdi1 id 6 adding device /dev/mapper/crypt_sdj1 id 7 adding device /dev/mapper/crypt_sdk1 id 8 adding device /dev/mapper/crypt_sdl1 id 9 fs created label backupcopy on /dev/mapper/crypt_sdb1 nodesize 16384 leafsize 16384 sectorsize 4096 size 4.09TiB polgara:/dev/disk/by-id# mount -L backupcopy /mnt/btrfs_backupcopy/ polgara:/mnt/btrfs_backupcopy# df -h . Filesystem Size Used Avail Use% Mounted on /dev/mapper/crypt_sdb1 4.1T 3.0M 4.1T 1% /mnt/btrfs_backupcopy Let's add one drive polgara:/mnt/btrfs_backupcopy# btrfs device add -f /dev/mapper/crypt_sdm1 /mnt/btrfs_backupcopy/ polgara:/mnt/btrfs_backupcopy# df -h . Filesystem Size Used Avail Use% Mounted on /dev/mapper/crypt_sdb1 4.6T 3.0M 4.6T 1% /mnt/btrfs_backupcopy Oh look it's bigger now. We need to manual rebalance to use the new drive: polgara:/mnt/btrfs_backupcopy# btrfs balance start . Done, had to relocate 6 out of 6 chunks polgara:/mnt/btrfs_backupcopy# btrfs device delete /dev/mapper/crypt_sdm1 . BTRFS info (device dm-9): relocating block group 23314563072 flags 130 BTRFS info (device dm-9): relocating block group 22106603520 flags 132 BTRFS info (device dm-9): found 6 extents BTRFS info (device dm-9): relocating block group 12442927104 flags 129 BTRFS info (device dm-9): found 1 extents polgara:/mnt/btrfs_backupcopy# df -h . Filesystem Size Used Avail Use% Mounted on /dev/mapper/crypt_sdb1 4.1T 4.7M 4.1T 1% /mnt/btrfs_backupcopy Ah, it's smaller again. Note that it's not degraded, you can just keep removing drives and it'll do a force reblance to fit the data in the remaining drives. Ok, I've unounted the filesystem, and will manually remove a device: polgara:~# dmsetup remove crypt_sdl1 polgara:~# mount -L backupcopy /mnt/btrfs_backupcopy/ mount: wrong fs type, bad option, bad superblock on /dev/mapper/crypt_sdk1, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so BTRFS: open /dev/dm-9 failed BTRFS info (device dm-7): disk space caching is enabled BTRFS: failed to read chunk tree on dm-7 BTRFS: open_ctree failed So a normal mount fails. You
Re: How to handle a RAID5 arrawy with a failing drive? - raid5 mostly works, just no rebuilds
On Mar 19, 2014, at 12:09 AM, Marc MERLIN m...@merlins.org wrote: 7) you can remove a drive from an array, add files, and then if you plug the drive in, it apparently gets auto sucked in back in the array. There is no rebuild that happens, you now have an inconsistent array where one drive is not at the same level than the other ones (I lost all files I added after the drive was removed when I added the drive back). Seems worthy of a dedicated bug report and keeping an eye on in the future, not good. polgara:/mnt/btrfs_backupcopy# df -h . Filesystem Size Used Avail Use% Mounted on /dev/mapper/crypt_sdb1 4.1T 3.0M 4.1T 1% /mnt/btrfs_backupcopy Let's add one drive polgara:/mnt/btrfs_backupcopy# btrfs device add -f /dev/mapper/crypt_sdm1 /mnt/btrfs_backupcopy/ polgara:/mnt/btrfs_backupcopy# df -h . Filesystem Size Used Avail Use% Mounted on /dev/mapper/crypt_sdb1 4.6T 3.0M 4.6T 1% /mnt/btrfs_backupcopy Oh look it's bigger now. We need to manual rebalance to use the new drive: You don't have to. As soon as you add the additional drive, newly allocated chunks will stripe across all available drives. e.g. 1 GB allocations striped across 3x drives, if I add a 4th drive, initially any additional writes are only to the first three drives but once a new data chunk is allocated it gets striped across 4 drives. In other words, btrfs happily added my device that was way behind and gave me an incomplete fileystem instead of noticing that sdj1 was behind and giving me a degraded filesystem. Moral of the story: do not ever re-add a device that got kicked out if you wrote new data after that, or you will end up with an older version of your filesystem (on the plus side, it's consistent and apparently without data corruption. That said, btrfs scrub complained loudly of many errors it didn't know how to fix. Sure the whole thing isn't corrupt. But if anything written while degraded vanishes once the missing device is reattached, and you remount normally (non-degraded), that's data loss. Yikes! There you go, hope this helps. Yes. Thanks! Chris Murphy-- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: How to handle a RAID5 arrawy with a failing drive? - raid5 mostly works, just no rebuilds
On Wed, Mar 19, 2014 at 12:32:55AM -0600, Chris Murphy wrote: On Mar 19, 2014, at 12:09 AM, Marc MERLIN m...@merlins.org wrote: 7) you can remove a drive from an array, add files, and then if you plug the drive in, it apparently gets auto sucked in back in the array. There is no rebuild that happens, you now have an inconsistent array where one drive is not at the same level than the other ones (I lost all files I added after the drive was removed when I added the drive back). Seems worthy of a dedicated bug report and keeping an eye on in the future, not good. Since it's not supposed to be working, I didn't file a bug, but I figured it'd be good for people to know about it in the meantime. polgara:/mnt/btrfs_backupcopy# btrfs device add -f /dev/mapper/crypt_sdm1 /mnt/btrfs_backupcopy/ polgara:/mnt/btrfs_backupcopy# df -h . Filesystem Size Used Avail Use% Mounted on /dev/mapper/crypt_sdb1 4.6T 3.0M 4.6T 1% /mnt/btrfs_backupcopy Oh look it's bigger now. We need to manual rebalance to use the new drive: You don't have to. As soon as you add the additional drive, newly allocated chunks will stripe across all available drives. e.g. 1 GB allocations striped across 3x drives, if I add a 4th drive, initially any additional writes are only to the first three drives but once a new data chunk is allocated it gets striped across 4 drives. That's the thing though. If the bad device hadn't been forcibly removed, and apparently the only way to do this was to unmount, make the device node disappear, and remount in degraded mode, it looked to me like btrfs was still consideing that the drive was part of the array and trying to write to it. After adding a drive, I couldn't quite tell if it was striping over 11 drive2 or 10, but it felt that at least at times, it was striping over 11 drives with write failures on the missing drive. I can't prove it, but I'm thinking the new data I was writing was being striped in degraded mode. Sure the whole thing isn't corrupt. But if anything written while degraded vanishes once the missing device is reattached, and you remount normally (non-degraded), that's data loss. Yikes! Yes, although it's limited, you apparently only lose new data that was added after you went into degraded mode and only if you add another drive where you write more data. In real life this shouldn't be too common, even if it is indeed a bug. Cheers, Marc -- A mouse is a device used to point at the xterm you want to type in - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: How to handle a RAID5 arrawy with a failing drive? - raid5 mostly works, just no rebuilds
On Mar 19, 2014, at 9:40 AM, Marc MERLIN m...@merlins.org wrote: After adding a drive, I couldn't quite tell if it was striping over 11 drive2 or 10, but it felt that at least at times, it was striping over 11 drives with write failures on the missing drive. I can't prove it, but I'm thinking the new data I was writing was being striped in degraded mode. Well it does sound fragile after all to add a drive to a degraded array, especially when it's not expressly treating the faulty drive as faulty. I think iotop will show what block devices are being written to. And in a VM it's easy (albeit rudimentary) with sparse files, as you can see them grow. Yes, although it's limited, you apparently only lose new data that was added after you went into degraded mode and only if you add another drive where you write more data. In real life this shouldn't be too common, even if it is indeed a bug. It's entirely plausible a drive power/data cable becomes lose, runs for hours degraded before the wayward device is reseated. It'll be common enough. It's definitely not OK for all of that data in the interim to vanish just because the volume has resumed from degraded to normal. Two states of data, normal vs degraded, is scary. It sounds like totally silent data loss. So yeah if it's reproducible it's worthy of a separate bug. Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: How to handle a RAID5 arrawy with a failing drive? - raid5 mostly works, just no rebuilds
On Wed, Mar 19, 2014 at 10:53:33AM -0600, Chris Murphy wrote: Yes, although it's limited, you apparently only lose new data that was added after you went into degraded mode and only if you add another drive where you write more data. In real life this shouldn't be too common, even if it is indeed a bug. It's entirely plausible a drive power/data cable becomes lose, runs for hours degraded before the wayward device is reseated. It'll be common enough. It's definitely not OK for all of that data in the interim to vanish just because the volume has resumed from degraded to normal. Two states of data, normal vs degraded, is scary. It sounds like totally silent data loss. So yeah if it's reproducible it's worthy of a separate bug. Actually what I did is more complex, I first added a drive to a degraded array, and then re-added the drive that had been removed. I don't know if re-adding the same drive that was removed would cause the bug I saw. For now, my array is back to actually trying to store the backup I had meant for it, and the drives seems stable now that I fixed the power issue. Does someone else want to try? :) Marc -- A mouse is a device used to point at the xterm you want to type in - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: How to handle a RAID5 arrawy with a failing drive?
Marc MERLIN posted on Sun, 16 Mar 2014 15:20:26 -0700 as excerpted: Do I have other options? (data is not important at all, I just want to learn how to deal with such a case with the current code) First just a note that you hijacked Mr Manana's patch thread. Replying to a post and changing the topic (the usual cause of such hijacks) does NOT change the thread, as the References and In-Reply-To headers still includes the Message-IDs from the original thread, and that's what good clients thread by since the subject line isn't a reliable means of threading. To start a NEW thread, don't reply to an existing thread, compose a NEW message, starting a NEW thread. =:^) Back on topic... Since you don't have to worry about the data I'd suggest blowing it away and starting over. Btrfs raid5/6 code is known to be incomplete at this point, to work in normal mode and write everything out, but with incomplete recovery code. So I'd treat it like the raid-0 mode it effectively is, and consider it lost if a device drops. There *IS* a post from an earlier thread where someone mentioned a recovery under some specific circumstance that worked for him, but I'd consider that the exception not the norm since the code is known to be incomplete and I think he just got lucky and didn't hit the particular missing code in his specific case. Certainly you could try to go back and see what he did and under what conditions, and that might actually be worth doing if you had valuable data you'd be losing otherwise, but since you don't, while of course it's up to you, I'd not bother were it me. Which I haven't. My use-case wouldn't be looking at raid5/6 (or raid0) anyway, but even if it were, I'd not touch the current code unless it /was/ just for something I'd consider risking on a raid0. Other than pure testing, the /only/ case I'd consider btrfs raid5/6 for right now, would be something that I'd consider raid0 riskable currently, but with the bonus of it upgrading for free to raid5/6 when the code is complete without any further effort on my part, since it's actually being written as raid5/6 ATM, the recovery simply can't be relied upon as raid5/6, so in recovery terms you're effectively running raid0 until it can be. Other than that and for /pure/ testing, I just don't see the point of even thinking about raid5/6 at this point. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: How to handle a RAID5 arrawy with a failing drive?
On Sun, Mar 16, 2014 at 11:12:43PM -0600, Chris Murphy wrote: On Mar 16, 2014, at 9:44 PM, Marc MERLIN m...@merlins.org wrote: On Sun, Mar 16, 2014 at 08:56:35PM -0600, Chris Murphy wrote: If I add a device, isn't it going to grow my raid to make it bigger instead of trying to replace the bad device? Yes if it's successful. No if it fails which is the problem I'm having. That's where I don't follow you. You just agreed that it will grow my raid. So right now it's 4.5TB with 10 drives, if I add one drive, it will grow to 5TB with 11 drives. How does that help? If you swap the faulty drive for a good drive, I'm thinking then you'll be able to device delete the bad device, which ought to be missing at that point; or if that fails you should be able to do a balance, and then be able to device delete the faulty drive. The problem I'm having is that when I detach one device out of a 3 device raid5, btrfs fi show doesn't list it as missing. It's listed without the /dev/sdd designation it had when attached, but now it's just blank. Ok, I tried unmounting and remounting degraded this morning: polgara:~# mount -v -t btrfs -o compress=zlib,space_cache,noatime,degraded LABEL=backupcopy /mnt/btrfs_backupcopy Mar 17 08:57:35 polgara kernel: [123824.344085] BTRFS: device label backupcopy devid 9 transid 3837 /dev/mapper/crypt_sdk1 Mar 17 08:57:35 polgara kernel: [123824.454641] BTRFS info (device dm-9): allowing degraded mounts Mar 17 08:57:35 polgara kernel: [123824.454978] BTRFS info (device dm-9): disk space caching is enabled Mar 17 08:57:35 polgara kernel: [123824.497437] BTRFS: bdev /dev/mapper/crypt_sde1 errs: wr 3888, rd 321927975, flush 0, corrupt 0, gen 0 /dev/mapper/crypt_sdk1 on /mnt/btrfs_backupcopy type btrfs (rw,noatime,compress=zlib,space_cache,degraded) What's confusing is that mounting in degraded mode shows all devices: polgara:~# btrfs fi show Label: backupcopy uuid: 7d8e1197-69e4-40d8-8d86-278d275af896 Total devices 10 FS bytes used 376.27GiB devid1 size 465.76GiB used 42.42GiB path /dev/dm-0 devid2 size 465.76GiB used 42.40GiB path /dev/dm-1 devid3 size 465.75GiB used 42.40GiB path /dev/mapper/crypt_sde1 this is missing devid4 size 465.76GiB used 42.40GiB path /dev/dm-3 devid5 size 465.76GiB used 42.40GiB path /dev/dm-4 devid6 size 465.76GiB used 42.40GiB path /dev/dm-5 devid7 size 465.76GiB used 42.40GiB path /dev/dm-6 devid8 size 465.76GiB used 42.40GiB path /dev/mapper/crypt_sdj1 devid9 size 465.76GiB used 42.40GiB path /dev/mapper/crypt_sdk1 devid10 size 465.76GiB used 42.40GiB path /dev/dm-8 Ok, so mount in degraded mode works. Adding a new device failed though: polgara:~# btrfs device add /dev/mapper/crypt_sdm1 /mnt/btrfs_backupcopy BTRFS: bad tree block start 852309604880683448 156237824 [ cut here ] WARNING: CPU: 0 PID: 1963 at fs/btrfs/super.c:257 __btrfs_abort_transaction+0x50/0x100() BTRFS: Transaction aborted (error -5) Modules linked in: xts gf128mul ipt_MASQUERADE ipt_REJECT xt_tcpudp xt_conntrack xt_LOG iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables cpufreq_userspace cpufreq_powersave cpufreq_conservative cpufreq_stats ppdev rfcomm bnep autofs4 binfmt_misc uinput nfsd auth_rpcgss nfs_acl nfs lockd fscache sunrpc fuse dm_crypt dm_mod configs parport_pc lp parport input_polldev loop firewire_sbp2 firewire_core crc_itu_t ecryptfs btusb bluetooth 6lowpan_iphc rfkill usbkbd usbmouse joydev hid_generic usbhid hid iTCO_wdt iTCO_vendor_support gpio_ich coretemp kvm_intel kvm microcode snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec pcspkr snd_hwdep i2c_i801 snd_pcm_oss snd_mixer_oss lpc_ich snd_pcm snd_seq_midi snd_seq_midi_event sg sr_mod cdrom snd_rawmidi snd_seq snd_seq_device snd_timer atl1 mii mvsas snd nouveau libsas scsi_transport_ soundcore ttm ehci_pci asus_atk0110 floppy uhci_hcd ehci_hcd usbcore acpi_cpufreq usb_common processor evdev CPU: 0 PID: 1963 Comm: btrfs Tainted: GW 3.14.0-rc5-amd64-i915-preempt-20140216c #1 Hardware name: System manufacturer P5KC/P5KC, BIOS 050205/24/2007 88004b5c9988 816090b3 88004b5c99d0 88004b5c99c0 81050025 8120913a fffb 8800144d5800 88007bd3ba00 81839280 88004b5c9a20 Call Trace: [816090b3] dump_stack+0x4e/0x7a [81050025] warn_slowpath_common+0x7f/0x98 [8120913a] ? __btrfs_abort_transaction+0x50/0x100 [8105008a] warn_slowpath_fmt+0x4c/0x4e [8120913a] __btrfs_abort_transaction+0x50/0x100 [81216fed] __btrfs_free_extent+0x6ce/0x712 [8121bc89] __btrfs_run_delayed_refs+0x939/0xbdf [8121dac8]
Re: How to handle a RAID5 arrawy with a failing drive?
On Mar 17, 2014, at 10:13 AM, Marc MERLIN m...@merlins.org wrote: What's confusing is that mounting in degraded mode shows all devices: polgara:~# btrfs fi show Label: backupcopy uuid: 7d8e1197-69e4-40d8-8d86-278d275af896 Total devices 10 FS bytes used 376.27GiB devid1 size 465.76GiB used 42.42GiB path /dev/dm-0 devid2 size 465.76GiB used 42.40GiB path /dev/dm-1 devid3 size 465.75GiB used 42.40GiB path /dev/mapper/crypt_sde1 this is missing devid4 size 465.76GiB used 42.40GiB path /dev/dm-3 devid5 size 465.76GiB used 42.40GiB path /dev/dm-4 devid6 size 465.76GiB used 42.40GiB path /dev/dm-5 devid7 size 465.76GiB used 42.40GiB path /dev/dm-6 devid8 size 465.76GiB used 42.40GiB path /dev/mapper/crypt_sdj1 devid9 size 465.76GiB used 42.40GiB path /dev/mapper/crypt_sdk1 devid10 size 465.76GiB used 42.40GiB path /dev/dm-8 /dev/mapper/crypt_sde1 is completely unavailable, as in not listed by lsblk? If it's not connected and not listed by lsblk yet it's listed by btrfs fi show that's a bug. eventually it turned into: BTRFS: bdev /dev/mapper/crypt_sde1 errs: wr 3891, rd 321927996, flush 0, corrupt 0, gen 0 BTRFS: bdev /dev/mapper/crypt_sde1 errs: wr 3891, rd 321927997, flush 0, corrupt 0, gen 0 [snip] BTRFS: error (device dm-9) in __btrfs_free_extent:5755: errno=-5 IO failure BTRFS info (device dm-9): forced readonly BTRFS: error (device dm-9) in btrfs_run_delayed_refs:2713: errno=-5 IO failure I think it's a lost cause at this point. Your setup is substantially more complicated than my simple setup, and I can't even get the simple setup to recover from an idealized single device raid5 failure. The only apparent way out is to mount degraded, backup, and then start over. In your case it looks like at least two devices are reporting, or Btrfs thinks they're reporting, I/O errors. Whether this is the physical drive itself, or if it's some other layer (it looks like these are dmcrypt logical block devices). Chris Murphy-- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: How to handle a RAID5 arrawy with a failing drive?
On Mar 16, 2014, at 4:20 PM, Marc MERLIN m...@merlins.org wrote: If I yank, sde1 and reboot, the array will not come back up from what I understand, or is that incorrect? Do rebuilds work at all with a missing drive to a spare drive? The part that isn't working well enough is faulty status. The drive keeps hanging around producing a lot of errors, instead of getting booted. btrfs replace start ought to still work, but if the faulty drive is fussy it might slow down the rebuild, or even prevent it. The more conservative approach is to pull the drive. If you've previously tested this hardware setup to tolerate hot swap, you can give that a shot. Otherwise, to avoid instability and additional problems, unmount the file system first. Do the hot swap. Then mount it -o degraded. Then use btrfs replace start. Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: How to handle a RAID5 arrawy with a failing drive?
On Mar 16, 2014, at 4:55 PM, Chris Murphy li...@colorremedies.com wrote: Then use btrfs replace start. Looks like in 3.14rc6 replace isn't yet supported. I get dev_replace cannot yet handle RAID5/RAID6. When I do: btrfs device add new mp The command hangs, no kernel messages. Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: How to handle a RAID5 arrawy with a failing drive?
On Sun, Mar 16, 2014 at 05:12:10PM -0600, Chris Murphy wrote: On Mar 16, 2014, at 4:55 PM, Chris Murphy li...@colorremedies.com wrote: Then use btrfs replace start. Looks like in 3.14rc6 replace isn't yet supported. I get dev_replace cannot yet handle RAID5/RAID6. When I do: btrfs device add new mp The command hangs, no kernel messages. Ok, that's kind of what I thought. So, for now, with raid5: - btrfs seems to handle a drive not working - you say I can mount with the drive missing in degraded mode (I haven't tried that, I will) - but no matter how I remove the faulty drive, there is no rebuild on a new drive procedure that works yet Correct? Thanks, Marc -- A mouse is a device used to point at the xterm you want to type in - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: How to handle a RAID5 arrawy with a failing drive?
On Sun, Mar 16, 2014 at 4:17 PM, Marc MERLIN m...@merlins.org wrote: On Sun, Mar 16, 2014 at 05:12:10PM -0600, Chris Murphy wrote: On Mar 16, 2014, at 4:55 PM, Chris Murphy li...@colorremedies.com wrote: Then use btrfs replace start. Looks like in 3.14rc6 replace isn't yet supported. I get dev_replace cannot yet handle RAID5/RAID6. When I do: btrfs device add new mp The command hangs, no kernel messages. Ok, that's kind of what I thought. So, for now, with raid5: - btrfs seems to handle a drive not working - you say I can mount with the drive missing in degraded mode (I haven't tried that, I will) - but no matter how I remove the faulty drive, there is no rebuild on a new drive procedure that works yet Correct? There was a discussion a while back that suggested that a balance would read all blocks and write them out again and that would recover the data. I have no idea if that works or not. Only do this as a last resort once you have already considered all data lost forever. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: How to handle a RAID5 arrawy with a failing drive?
On Sun, Mar 16, 2014 at 05:23:25PM -0600, Chris Murphy wrote: On Mar 16, 2014, at 5:17 PM, Marc MERLIN m...@merlins.org wrote: - but no matter how I remove the faulty drive, there is no rebuild on a new drive procedure that works yet Correct? I'm not sure. From what I've read we should be able to add a device to raid5/6, but I don't know if it's expected we can add a device to a degraded raid5/6. If the add device succeeded, then I ought to be able to remove the missing devid, and then do a balance which should cause reconstruction. https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg30714.html Thanks for the link, that's what I thought I read recently. So, on 3.14, I can confirm polgara:/mnt/btrfs_backupcopy# btrfs replace start 3 /dev/sdm1 /mnt/btrfs_backupcopy [68377.679233] BTRFS warning (device dm-9): dev_replace cannot yet handle RAID5/RAID6 polgara:/mnt/btrfs_backupcopy# btrfs device delete /dev/mapper/crypt_sde1 `pwd` ERROR: error removing the device '/dev/mapper/crypt_sde1' - Invalid argument and yet Mar 16 17:48:35 polgara kernel: [69285.032615] BTRFS: bdev /dev/mapper/crypt_sde1 errs: wr 805, rd 4835, flush 0, corrupt 0, gen 0 Mar 16 17:48:35 polgara kernel: [69285.033791] BTRFS: lost page write due to I/O error on /dev/mapper/crypt_sde1 Mar 16 17:48:35 polgara kernel: [69285.034379] BTRFS: bdev /dev/mapper/crypt_sde1 errs: wr 806, rd 4835, flush 0, corrupt 0, gen 0 Mar 16 17:48:35 polgara kernel: [69285.035361] BTRFS: lost page write due to I/O error on /dev/mapper/crypt_sde1 Mar 16 17:48:35 polgara kernel: [69285.035943] BTRFS: bdev /dev/mapper/crypt_sde1 errs: wr 807, rd 4835, flush 0, corrupt 0, gen 0 So from here, it sounds like I can try: 1) unmount the filesystem 2) hope that remounting it without that device will work 3) btrfs device add to recreate the missing drive. Before I do #1 and get myself in a worse state than I am (working filesystem), does that sound correct? (again, the data is irrelevant, I have a btrfs receive on it that has been running for hours and that I'd have to restart, but that's it). Thanks, Marc -- A mouse is a device used to point at the xterm you want to type in - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: How to handle a RAID5 arrawy with a failing drive?
On Mar 16, 2014, at 6:51 PM, Marc MERLIN m...@merlins.org wrote: polgara:/mnt/btrfs_backupcopy# btrfs device delete /dev/mapper/crypt_sde1 `pwd` ERROR: error removing the device '/dev/mapper/crypt_sde1' - Invalid argument You didn't specify a mount point, is the reason for that error. But also, since you're already effectively degraded with 1 disk you can't remove a 2nd without causing array collapse. You have to add a new device first *and* you have to rebuild with balance. Then presumably we can remove the device. But I'm stuck adding so I can't test anything else. So from here, it sounds like I can try: 1) unmount the filesystem 2) hope that remounting it without that device will work 3) btrfs device add to recreate the missing drive. Before I do #1 and get myself in a worse state than I am (working filesystem), does that sound correct? (again, the data is irrelevant, I have a btrfs receive on it that has been running for hours and that I'd have to restart, but that's it). Well at this point I'd leave it alone because at least for me, device add hangs that command and all other subsequent btrfs user space commands. So for all I know (untested) the whole volume will block on this device add and is effectively useless. Chris Murphy-- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: How to handle a RAID5 arrawy with a failing drive?
On Sun, Mar 16, 2014 at 07:06:23PM -0600, Chris Murphy wrote: On Mar 16, 2014, at 6:51 PM, Marc MERLIN m...@merlins.org wrote: polgara:/mnt/btrfs_backupcopy# btrfs device delete /dev/mapper/crypt_sde1 `pwd` ERROR: error removing the device '/dev/mapper/crypt_sde1' - Invalid argument You didn't specify a mount point, is the reason for that error. But also, since you're already effectively degraded with 1 disk you can't remove a 2nd without causing array collapse. You have to add a new device first *and* you have to rebuild with balance. Then presumably we can remove the device. But I'm stuck adding so I can't test anything else. You missed the `pwd` :) I'm trying to remove the drive that is causing issues, that doesn't make things worse, does it? Does btrtfs not know that device is the bad one even thouth it's spamming my logs continuously about it? If I add a device, isn't it going to grow my raid to make it bigger instead of trying to replace the bad device? In swraid5, if I add a device, it will grow the raid, unless the array is running in degraded mode. However, I can't see if btrfs tools know it's in degraded mode or not. If you are sure adding a device won't grow my raid, I'll give it a shot. (again, the data is irrelevant, I have a btrfs receive on it that has been running for hours and that I'd have to restart, but that's it). Well at this point I'd leave it alone because at least for me, device add hangs that command and all other subsequent btrfs user space commands. So for all I know (untested) the whole volume will block on this device add and is effectively useless. Right. I was hoping that my kernel slightly newer than yours and maybe real devices would help, but of course I don't know that. I'll add the new device first after you confirm that there is no chance it'll try to grow the filesystem :) Thanks, Marc -- A mouse is a device used to point at the xterm you want to type in - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: How to handle a RAID5 arrawy with a failing drive?
On Mar 16, 2014, at 7:17 PM, Marc MERLIN m...@merlins.org wrote: On Sun, Mar 16, 2014 at 07:06:23PM -0600, Chris Murphy wrote: On Mar 16, 2014, at 6:51 PM, Marc MERLIN m...@merlins.org wrote: polgara:/mnt/btrfs_backupcopy# btrfs device delete /dev/mapper/crypt_sde1 `pwd` ERROR: error removing the device '/dev/mapper/crypt_sde1' - Invalid argument You didn't specify a mount point, is the reason for that error. But also, since you're already effectively degraded with 1 disk you can't remove a 2nd without causing array collapse. You have to add a new device first *and* you have to rebuild with balance. Then presumably we can remove the device. But I'm stuck adding so I can't test anything else. You missed the `pwd` :) I just don't know what it means, it's not a reference to mount point I'm familiar with. I'm trying to remove the drive that is causing issues, that doesn't make things worse, does it? I don't think you can force a Btrfs volume to go degraded with a device delete command right now, just like there isn't a command to make it go missing or faulty, like md raid. Does btrtfs not know that device is the bad one even thouth it's spamming my logs continuously about it? With raid5, you're always at the minimum number of devices to be normally mounted. Removing one immediately makes it degraded which I don't think it's going to permit. At least, I get an error when I do it even without a device giving me fits. If I add a device, isn't it going to grow my raid to make it bigger instead of trying to replace the bad device? Yes if it's successful. No if it fails which is the problem I'm having. In swraid5, if I add a device, it will grow the raid, unless the array is running in degraded mode. However, I can't see if btrfs tools know it's in degraded mode or not. Only once the device is missing, apparently, and then mounted -o degraded. If you are sure adding a device won't grow my raid, I'll give it a shot. No I'm not sure. And yes I suspect it will make it bigger. But so far a.) replace isn't supported yet; and b.) delete causes the volume to go below the minimum required for normal operation which it won't allow; which leaves c.) add a device but I'm getting a hang. So I'm stuck at this point. (again, the data is irrelevant, I have a btrfs receive on it that has been running for hours and that I'd have to restart, but that's it). Well at this point I'd leave it alone because at least for me, device add hangs that command and all other subsequent btrfs user space commands. So for all I know (untested) the whole volume will block on this device add and is effectively useless. Right. I was hoping that my kernel slightly newer than yours and maybe real devices would help, but of course I don't know that. I'll add the new device first after you confirm that there is no chance it'll try to grow the filesystem :) I confirm nothing since I can't proceed with a device add. Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: How to handle a RAID5 arrawy with a failing drive?
On Sun, Mar 16, 2014 at 08:56:35PM -0600, Chris Murphy wrote: polgara:/mnt/btrfs_backupcopy# btrfs device delete /dev/mapper/crypt_sde1 `pwd` ERROR: error removing the device '/dev/mapper/crypt_sde1' - Invalid argument You didn't specify a mount point, is the reason for that error. But also, since you're already effectively degraded with 1 disk you can't remove a 2nd without causing array collapse. You have to add a new device first *and* you have to rebuild with balance. Then presumably we can remove the device. But I'm stuck adding so I can't test anything else. You missed the `pwd` :) I just don't know what it means, it's not a reference to mount point I'm familiar with. Try echo `pwd` and you'll understand :) I'm trying to remove the drive that is causing issues, that doesn't make things worse, does it? I don't think you can force a Btrfs volume to go degraded with a device delete command right now, just like there isn't a command to make it go missing or faulty, like md raid. Does btrtfs not know that device is the bad one even thouth it's spamming my logs continuously about it? With raid5, you're always at the minimum number of devices to be normally mounted. Removing one immediately makes it degraded which I don't think it's going to permit. At least, I get an error when I do it even without a device giving me fits. Ok, I understand that. If I add a device, isn't it going to grow my raid to make it bigger instead of trying to replace the bad device? Yes if it's successful. No if it fails which is the problem I'm having. That's where I don't follow you. You just agreed that it will grow my raid. So right now it's 4.5TB with 10 drives, if I add one drive, it will grow to 5TB with 11 drives. How does that help? Why would btrfs allow me to remove the faulty device since it does not let you remove a device from a running raid. If I grow it to a bigger raid, it still won't let me remove the device, will it? In swraid5, if I add a device, it will grow the raid, unless the array is running in degraded mode. However, I can't see if btrfs tools know it's in degraded mode or not. Only once the device is missing, apparently, and then mounted -o degraded. Dully noted. If you agree that adding an 11th drive to my array will not help, I'll unmount the filesystem, remount it in degraded mode with 9 drives and try to add the new 11th drive. If you are sure adding a device won't grow my raid, I'll give it a shot. No I'm not sure. And yes I suspect it will make it bigger. But so far a.) replace isn't supported yet; and b.) delete causes the volume to go below the minimum required for normal operation which it won't allow; which leaves c.) add a device but I'm getting a hang. So I'm stuck at this point. Right. So I think we also agree that adding a device to the running filesystem is not what I want to do since it'll grow it and do nothing to let me remove he faulty one. I'll add the new device first after you confirm that there is no chance it'll try to grow the filesystem :) I confirm nothing since I can't proceed with a device add. Fair enough. So unless someone tells me otherwise, I will unmount the filesystem, remount it in degraded mode, and then try to add the 11th drive when the 10th one is missing. Thanks, Marc -- A mouse is a device used to point at the xterm you want to type in - A.S.R. Microsoft is to operating systems what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: How to handle a RAID5 arrawy with a failing drive?
On Mar 16, 2014, at 9:44 PM, Marc MERLIN m...@merlins.org wrote: On Sun, Mar 16, 2014 at 08:56:35PM -0600, Chris Murphy wrote: If I add a device, isn't it going to grow my raid to make it bigger instead of trying to replace the bad device? Yes if it's successful. No if it fails which is the problem I'm having. That's where I don't follow you. You just agreed that it will grow my raid. So right now it's 4.5TB with 10 drives, if I add one drive, it will grow to 5TB with 11 drives. How does that help? If you swap the faulty drive for a good drive, I'm thinking then you'll be able to device delete the bad device, which ought to be missing at that point; or if that fails you should be able to do a balance, and then be able to device delete the faulty drive. The problem I'm having is that when I detach one device out of a 3 device raid5, btrfs fi show doesn't list it as missing. It's listed without the /dev/sdd designation it had when attached, but now it's just blank. Why would btrfs allow me to remove the faulty device since it does not let you remove a device from a running raid. If I grow it to a bigger raid, it still won't let me remove the device, will it? Maybe not, but it seems like it ought to let you balance, which should only be across available devices at which point you should be able to device delete the bad one. That's assuming you've physically detached the faulty device from the start though. In swraid5, if I add a device, it will grow the raid, unless the array is running in degraded mode. However, I can't see if btrfs tools know it's in degraded mode or not. Only once the device is missing, apparently, and then mounted -o degraded. Dully noted. If you agree that adding an 11th drive to my array will not help, I'll unmount the filesystem, remount it in degraded mode with 9 drives and try to add the new 11th drive. That's the only option I see at the moment in any case, other than blowing it all away and starting from scratch. What I don't know is whether you will be able to 'btrfs device delete' what ought to now be a missing device, since you have enough drives added to proceed with that deletion; or if you'll have to balance first. And I don't even know if the balance will work and then let you device delete if it's a dead end at this point. If you are sure adding a device won't grow my raid, I'll give it a shot. No I'm not sure. And yes I suspect it will make it bigger. But so far a.) replace isn't supported yet; and b.) delete causes the volume to go below the minimum required for normal operation which it won't allow; which leaves c.) add a device but I'm getting a hang. So I'm stuck at this point. Right. So I think we also agree that adding a device to the running filesystem is not what I want to do since it'll grow it and do nothing to let me remove he faulty one. The grow is entirely beside the point. You definitely can't btrfs replace, or btrfs device delete, so what else is there but try btrfs device add, or obliterate it and start over? Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
