Re: Btrfs/RAID5 became unmountable after SATA cable fault
It seems like I accidentally managed to break my Btrfs/RAID5 filesystem, yet again, in a similar fashion. This time around, I ran into some random libata driver issue (?) instead of a faulty hardware part but the end result is quiet similar. I issued the command (replacing X with valid letters for every hard-drives in the system): # echo 1 > /sys/block/sdX/device/queue_depth and I ended up with read-only filesystems. I checked dmesg and saw write errors on every disks (not just those in RAID-5). I tried to reboot immediately without success. My root filesystem with a single-disk Btrfs (which is an SSD, so it has "single" profile for both data and metadata) was unmountable, thus the kernel was stuck in a panic-reboot cycle. I managed to fix this one by booting from an USB stick and trying various recovery methods (like mounting it with "-o clear_cache,nospace_cache,recovery" and running "btrfs rescue chunk-recovery") until everything seemed to be fine (it can now be mounted read-write without error messages in the kernel-log, can be fully scrubbed without errors reported, it passes in "btrfs check", files can be actually written and read, etc). Once my system was up and running (well, sort of), I realized my /data is also un-mountable. I tried the same recovery methods on this RAID-5 filesystem but nothing seemed to help (there is an exception with the recovery attempts: the system drive was a small and fast SSD so "chunk-recovery" was a viable option to try but this one consists of huge slow HDDs - so, I tried to run it as a last-resort over-night but I found an unresponsive machine on the morning with the process stuck relatively early in the process). I can always mount it read-only and access files on it, seemingly without errors (I compared some of the contents with backups and it looks good) but as soon as I mount it read-write, all hell breaks loose and it falls into read-only state in no time (with some files seemingly disappearing from the filesystem) and the kernel log is starting to get spammed with various kind of error messages (including missing csums, etc). After mounting it like this: # mount /dev/sdb /data -o rw,noatime,nospace_cache and doing: # btrfs scrub start /data the result is: scrub status for 7d4769d6-2473-4c94-b476-4facce24b425 scrub started at Sat Jul 23 13:50:55 2016 and was aborted after 00:05:30 total bytes scrubbed: 18.99GiB with 16 errors error details: read=16 corrected errors: 0, uncorrectable errors: 16, unverified errors: 0 The relevant dmesg output is: [ 1047.709830] BTRFS info (device sdc): disabling disk space caching [ 1047.709846] BTRFS: has skinny extents [ 1047.895818] BTRFS info (device sdc): bdev /dev/sdc errs: wr 4, rd 0, flush 0, corrupt 0, gen 0 [ 1047.895835] BTRFS info (device sdc): bdev /dev/sdb errs: wr 4, rd 0, flush 0, corrupt 0, gen 0 [ 1065.764352] BTRFS: checking UUID tree [ 1386.423973] BTRFS error (device sdc): parent transid verify failed on 24431936729088 wanted 585936 found 586145 [ 1386.430922] BTRFS error (device sdc): parent transid verify failed on 24431936729088 wanted 585936 found 586145 [ 1411.738955] BTRFS error (device sdc): parent transid verify failed on 24432322764800 wanted 585779 found 586145 [ 1411.948040] BTRFS error (device sdc): parent transid verify failed on 24432322764800 wanted 585779 found 586145 [ 1412.040964] BTRFS error (device sdc): parent transid verify failed on 24432322764800 wanted 585779 found 586145 [ 1412.040980] BTRFS error (device sdc): parent transid verify failed on 24432322764800 wanted 585779 found 586145 [ 1412.041134] BTRFS error (device sdc): parent transid verify failed on 24432322764800 wanted 585779 found 586145 [ 1412.042628] BTRFS error (device sdc): parent transid verify failed on 24432322764800 wanted 585779 found 586145 [ 1412.042748] BTRFS error (device sdc): parent transid verify failed on 24432322764800 wanted 585779 found 586145 [ 1499.45] BTRFS error (device sdc): parent transid verify failed on 24432312270848 wanted 585779 found 586143 [ 1499.230264] BTRFS error (device sdc): parent transid verify failed on 24432312270848 wanted 585779 found 586143 [ 1525.865143] BTRFS error (device sdc): parent transid verify failed on 24432367730688 wanted 585779 found 586144 [ 1525.880537] BTRFS error (device sdc): parent transid verify failed on 24432367730688 wanted 585779 found 586144 [ 1552.434209] BTRFS error (device sdc): parent transid verify failed on 24432415821824 wanted 585781 found 586144 [ 1552.437325] BTRFS error (device sdc): parent transid verify failed on 24432415821824 wanted 585781 found 586144 btrfs check /dev/sdc results in: Checking filesystem on /dev/sdc UUID: 7d4769d6-2473-4c94-b476-4facce24b425 checking extents parent transid verify failed on 24431859855360 wanted 585941 found 586144 parent transid verify failed on 24431859855360 wanted 585941 found 586144 checksum verify failed on 24431859855360 found 3F0C0853 wanted 165308D5 par
Re: Btrfs/RAID5 became unmountable after SATA cable fault
On 6 November 2015 at 10:03, Janos Toth F. wrote: > > Although I updated the firmware of the drives. (I found an IMPORTANT > update when I went there to download SeaTools, although there was no > change log to tell me why this was important). This might changed the > error handling behavior of the drive...? I've had Seagate drives not reporting errors until I updated the firmware. They tended to timeout instead. Got a shitload of SMART errors after I updated, but they still didn't handle errors very well (became unresponsive). -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Btrfs/RAID5 became unmountable after SATA cable fault
I created a fresh RAID-5 mode Btrfs on the same 3 disks (including the faulty one which is still producing numerous random read errors) and Btrfs now seems to work exactly as I would anticipate. I copied some data and verified the checksum. The data is readable and correct regardless of the constant warning messages in the kernel log about the read errors on the single faulty HDD (the bad behavior is confirmed by the SMART logs and I tested it in a different PC as well...). I also ran several scrubs and now it always finishes with X corrected and 0 uncorrected errors. (The errors are supposedly corrected but the faulty HDD keeps randomly corrupting the data...) The last time I saw uncorrected errors during the scrub and not every data was readable. Rather strange... I ran 24 hours of Gimps/Prime95 Blend stresstest without errors on the problematic machine. Although I updated the firmware of the drives. (I found an IMPORTANT update when I went there to download SeaTools, although there was no change log to tell me why this was important). This might changed the error handling behavior of the drive...? -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Btrfs/RAID5 became unmountable after SATA cable fault
Duncan wrote: Austin S Hemmelgarn posted on Wed, 04 Nov 2015 13:45:37 -0500 as excerpted: On 2015-11-04 13:01, Janos Toth F. wrote: But the worst part is that there are some ISO files which were seemingly copied without errors but their external checksums (the one which I can calculate with md5sum and compare to the one supplied by the publisher of the ISO file) don't match! Well... this, I cannot understand. How could these files become corrupt from a single disk failure? And more importantly: how could these files be copied without errors? Why didn't Btrfs gave a read error when the checksums didn't add up? If you can prove that there was a checksum mismatch and BTRFS returned invalid data instead of a read error or going to the other disk, then that is a very serious bug that needs to be fixed. You need to keep in mind also however that it's completely possible that the data was bad before you wrote it to the filesystem, and if that's the case, there's nothing any filesystem can do to fix it for you. As Austin suggests, if btrfs is returning data, and you haven't turned off checksumming with nodatasum or nocow, then it's almost certainly returning the data it was given to write out in the first place. Whether that data it was given to write out was correct, however, is an /entirely/ different matter. If ISOs are failing their external checksums, then something is going on. Had you verified the external checksums when you first got the files? That is, are you sure the files were correct as downloaded and/or ripped? Where were the ISOs stored between original procurement/validation and writing to btrfs? Is it possible you still have some/all of them on that media? Do they still external-checksum-verify there? Basically, assuming btrfs checksums are validating, there's three other likely possibilities for where the corruption could have come from before writing to btrfs. Either the files were bad as downloaded or otherwise procured -- which is why I asked whether you verified them upon receipt -- or you have memory that's going bad, or your temporary storage is going bad, before the files ever got written to btrfs. The memory going bad is a particularly worrying possibility, considering... Now I am really considering to move from Linux to Windows and from Btrfs RAID-5 to Storage Spaces RAID-1 + ReFS (the only limitation is that ReFS is only "self-healing" on RAID-1, not RAID-5, so I need a new motherboard with more native SATA connectors and an extra HDD). That one seemed to actually do what it promises (abort any read operations upon checksum errors [which always happens seamlessly on every read] but look at the redundant data first and seamlessly "self-heal" if possible). The only thing which made Btrfs to look as a better alternative was the RAID-5 support. But I recently experienced two cases of 1 drive failing of 3 and it always tuned out as a smaller or bigger disaster (completely lost data or inconsistent data). Have you considered looking into ZFS? I hate to suggest it as an alternative to BTRFS, but it's a much more mature and well tested technology than ReFS, and has many of the same features as BTRFS (and even has the option for triple parity instead of the double you get with RAID6). If you do consider ZFS, make a point to look at FreeBSD in addition to the Linux version, the BSD one was a much better written port of the original Solaris drivers, and has better performance in many cases (and as much as I hate to admit it, BSD is way more reliable than Linux in most use cases). You should also seriously consider whether the convenience of having a filesystem that fixes internal errors itself with no user intervention is worth the risk of it corrupting your data. Returning correct data whenever possible is one thing, being 'self-healing' is completely different. When you start talking about things that automatically fix internal errors without user intervention is when most seasoned system administrators start to get really nervous. Self correcting systems have just as much chance to make things worse as they do to make things better, and most of them depend on the underlying hardware working correctly to actually provide any guarantee of reliability. I too would point you at ZFS, but there's one VERY BIG caveat, and one related smaller one! The people who have a lot of ZFS experience say it's generally quite reliable, but gobs of **RELIABLE** memory are *absolutely* *critical*! The self-healing works well, *PROVIDED* memory isn't producing errors. Absolutely reliable memory is in fact *so* critical, that running ZFS on non-ECC memory is severely discouraged as a very real risk to your data. Which is why the above hints that your memory may be bad are so worrying. Don't even *THINK* about ZFS, particularly its self-healing features, if you're not absolutely sure your memory is 100% reliable, because apparently, based on the comment's I've seen, if it's not, you WILL
Re: Btrfs/RAID5 became unmountable after SATA cable fault
On 2015-11-04 23:06, Duncan wrote: (Tho I should mention, while not on zfs, I've actually had my own problems with ECC RAM too. In my case, the RAM was certified to run at speeds faster than it was actually reliable at, such that actually stored data, what the ECC protects, was fine, the data was actually getting damaged in transit to/from the RAM. On a lightly loaded system, such as one running many memory tests or under normal desktop usage conditions, the RAM was generally fine, no problems. But on a heavily loaded system, such as when doing parallel builds (I run gentoo, which builds from sources in ordered to get the higher level of option flexibility that comes only when you can toggle build-time options), I'd often have memory faults and my builds would fail. The most common failure, BTW, was on tarball decompression, bunzip2 or the like, since the tarballs contained checksums that were verified on data decompression, and often they'd fail to verify. Once I updated the BIOS to one that would let me set the memory speed instead of using the speed the modules themselves reported, and I declocked the memory just one notch (this was DDR1, IIRC I declocked from the PC3200 it was rated, to PC3000 speeds), not only was the memory then 100% reliable, but I could and did actually reduce the number of wait- states for various operations, and it was STILL 100% reliable. It simply couldn't handle the raw speeds it was certified to run, is all, tho it did handle it well enough, enough of the time, to make the problem far more difficult to diagnose and confirm than it would have been had the problem appeared at low load as well. As it happens, I was running reiserfs at the time, and it handled both that hardware issue, and a number of others I've had, far better than I'd have expected of /any/ filesystem, when the memory feeding it is simply not reliable. Reiserfs metadata, in particular, seems incredibly resilient in the face of hardware issues, and I lost far less data than I might have expected, tho without checksums and with bad memory, I imagine I had occasional undetected bitflip corruption in files here or there, but generally nothing I detected. I still use reiserfs on my spinning rust today, but it's not well suited to SSD, which is where I run btrfs. But the point for this discussion is that just because it's ECC RAM doesn't mean you can't have memory related errors, just that if you do, they're likely to be different errors, "transit errors", that will tend to be undetected by many memory checkers, at least the ones that don't tend to run full out memory bandwidth if they're simply checking that what was stored in a cell can be read back, unchanged.) I've actually seen similar issues with both ECC and non-ECC memory myself. Any time I'm getting RAM for a system that I can afford to over-spec, I get the next higher speed and under-clock it (which in turn means I can lower the timing parameters and usually get a faster system than if I was running it at the rated speed). FWIW, I also make a point of doing multiple memtest86+ runs (at a minimum, one running single core, and one with forced SMP) when I get new RAM, and even have a run-level configured on my Gentoo based home server system where it boots Xen and fires up twice as many VM's running memtest86+ as I have CPU cores, which is usually enough to fully saturate memory bandwidth and check for the type of issues you mentioned having above (although the BOINC client I run usually does a good job of triggering those kind of issues fast, distributed computing apps tend to be memory bound and use a lot of memory bandwidth). smime.p7s Description: S/MIME Cryptographic Signature
Re: Btrfs/RAID5 became unmountable after SATA cable fault
Austin S Hemmelgarn posted on Wed, 04 Nov 2015 13:45:37 -0500 as excerpted: > On 2015-11-04 13:01, Janos Toth F. wrote: >> But the worst part is that there are some ISO files which were >> seemingly copied without errors but their external checksums (the one >> which I can calculate with md5sum and compare to the one supplied by >> the publisher of the ISO file) don't match! >> Well... this, I cannot understand. >> How could these files become corrupt from a single disk failure? And >> more importantly: how could these files be copied without errors? Why >> didn't Btrfs gave a read error when the checksums didn't add up? > If you can prove that there was a checksum mismatch and BTRFS returned > invalid data instead of a read error or going to the other disk, then > that is a very serious bug that needs to be fixed. You need to keep in > mind also however that it's completely possible that the data was bad > before you wrote it to the filesystem, and if that's the case, there's > nothing any filesystem can do to fix it for you. As Austin suggests, if btrfs is returning data, and you haven't turned off checksumming with nodatasum or nocow, then it's almost certainly returning the data it was given to write out in the first place. Whether that data it was given to write out was correct, however, is an /entirely/ different matter. If ISOs are failing their external checksums, then something is going on. Had you verified the external checksums when you first got the files? That is, are you sure the files were correct as downloaded and/or ripped? Where were the ISOs stored between original procurement/validation and writing to btrfs? Is it possible you still have some/all of them on that media? Do they still external-checksum-verify there? Basically, assuming btrfs checksums are validating, there's three other likely possibilities for where the corruption could have come from before writing to btrfs. Either the files were bad as downloaded or otherwise procured -- which is why I asked whether you verified them upon receipt -- or you have memory that's going bad, or your temporary storage is going bad, before the files ever got written to btrfs. The memory going bad is a particularly worrying possibility, considering... >> Now I am really considering to move from Linux to Windows and from >> Btrfs RAID-5 to Storage Spaces RAID-1 + ReFS (the only limitation is >> that ReFS is only "self-healing" on RAID-1, not RAID-5, so I need a new >> motherboard with more native SATA connectors and an extra HDD). That >> one seemed to actually do what it promises (abort any read operations >> upon checksum errors [which always happens seamlessly on every read] >> but look at the redundant data first and seamlessly "self-heal" if >> possible). The only thing which made Btrfs to look as a better >> alternative was the RAID-5 support. But I recently experienced two >> cases of 1 drive failing of 3 and it always tuned out as a smaller or >> bigger disaster (completely lost data or inconsistent data). > Have you considered looking into ZFS? I hate to suggest it as an > alternative to BTRFS, but it's a much more mature and well tested > technology than ReFS, and has many of the same features as BTRFS (and > even has the option for triple parity instead of the double you get with > RAID6). If you do consider ZFS, make a point to look at FreeBSD in > addition to the Linux version, the BSD one was a much better written > port of the original Solaris drivers, and has better performance in many > cases (and as much as I hate to admit it, BSD is way more reliable than > Linux in most use cases). > > You should also seriously consider whether the convenience of having a > filesystem that fixes internal errors itself with no user intervention > is worth the risk of it corrupting your data. Returning correct data > whenever possible is one thing, being 'self-healing' is completely > different. When you start talking about things that automatically fix > internal errors without user intervention is when most seasoned system > administrators start to get really nervous. Self correcting systems > have just as much chance to make things worse as they do to make things > better, and most of them depend on the underlying hardware working > correctly to actually provide any guarantee of reliability. I too would point you at ZFS, but there's one VERY BIG caveat, and one related smaller one! The people who have a lot of ZFS experience say it's generally quite reliable, but gobs of **RELIABLE** memory are *absolutely* *critical*! The self-healing works well, *PROVIDED* memory isn't producing errors. Absolutely reliable memory is in fact *so* critical, that running ZFS on non-ECC memory is severely discouraged as a very real risk to your data. Which is why the above hints that your memory may be bad are so worrying. Don't even *THINK* about ZFS, particularly its self-healing features, if you're not absolu
Re: Btrfs/RAID5 became unmountable after SATA cable fault
On 2015-11-04 13:01, Janos Toth F. wrote: But the worst part is that there are some ISO files which were seemingly copied without errors but their external checksums (the one which I can calculate with md5sum and compare to the one supplied by the publisher of the ISO file) don't match! Well... this, I cannot understand. How could these files become corrupt from a single disk failure? And more importantly: how could these files be copied without errors? Why didn't Btrfs gave a read error when the checksums didn't add up? If you can prove that there was a checksum mismatch and BTRFS returned invalid data instead of a read error or going to the other disk, then that is a very serious bug that needs to be fixed. You need to keep in mind also however that it's completely possible that the data was bad before you wrote it to the filesystem, and if that's the case, there's nothing any filesystem can do to fix it for you. Isn't Btrfs supposed to constantly check the integrity of the file data during any normal read operations and give an error instead of spitting out corrupt data as if it was perfectly legit? I thought that's how it is supposed to work. Assuming that all of your hardware is working exactly like it's supposed to, yes it should work that way. If however, you have something that corrupts the data in RAM before or while BTRFS is computing the checksum prior to writing the data, the it's fully possible for bad data to get written to disk and still have a perfectly correct checksum. Bad RAM may also explain your issues mentioned above with not being able to copy stuff off of the filesystem. Also, if you're using NOCOW files (or just the mount option), those very specifically do not store checksums for the blocks, because there is no way to do it without significant risk of data corruption. What's the point of full data checksuming if only an explicitly requested scrub operation might look for errors? I thought's it's the logical thing to do if checksum verification happens during every single read operation and passing that check is mandatory in order to get any data out of the filesystem (might be excluding the Direct-I/O mode but I never use that on Btrfs - if that's even actually supported, I don't know). Now I am really considering to move from Linux to Windows and from Btrfs RAID-5 to Storage Spaces RAID-1 + ReFS (the only limitation is that ReFS is only "self-healing" on RAID-1, not RAID-5, so I need a new motherboard with more native SATA connectors and an extra HDD). That one seemed to actually do what it promises (abort any read operations upon checksum errors [which always happens seamlessly on every read] but look at the redundant data first and seamlessly "self-heal" if possible). The only thing which made Btrfs to look as a better alternative was the RAID-5 support. But I recently experienced two cases of 1 drive failing of 3 and it always tuned out as a smaller or bigger disaster (completely lost data or inconsistent data). Have you considered looking into ZFS? I hate to suggest it as an alternative to BTRFS, but it's a much more mature and well tested technology than ReFS, and has many of the same features as BTRFS (and even has the option for triple parity instead of the double you get with RAID6). If you do consider ZFS, make a point to look at FreeBSD in addition to the Linux version, the BSD one was a much better written port of the original Solaris drivers, and has better performance in many cases (and as much as I hate to admit it, BSD is way more reliable than Linux in most use cases). You should also seriously consider whether the convenience of having a filesystem that fixes internal errors itself with no user intervention is worth the risk of it corrupting your data. Returning correct data whenever possible is one thing, being 'self-healing' is completely different. When you start talking about things that automatically fix internal errors without user intervention is when most seasoned system administrators start to get really nervous. Self correcting systems have just as much chance to make things worse as they do to make things better, and most of them depend on the underlying hardware working correctly to actually provide any guarantee of reliability. I cannot count the number of stories I've heard of 'self-healing' hardware RAID controllers destroying data. smime.p7s Description: S/MIME Cryptographic Signature
Re: Btrfs/RAID5 became unmountable after SATA cable fault
Well. Now I am really confused about Btrfs RAID-5! So, I replaced all SATA cables (which are explicitly marked for beeing aimed at SATA3 speeds) and all the 3x2Tb WD Red 2.0 drives with 3x4Tb Seagate Contellation ES 3 drives and started from sratch. I secure-erased every drives, created an empty filesystem and ran a "long" SMART self-test on all drivers before I started using the storage space (the tests finished without errors, all drivers looked fine, 0 zero bad sectors, 0 read or SATA CEC errors... all looked perfectly fine at the time...). It didn't take long before I realized that one of the new drives started failing. I started a scrub and it reported both corrected and uncorrectable errors. I looked at the SMART data. 2 drives look perfectly fine and 1 drive seems to be really sick. The latter one has some "reallocated" and several hundreds of "pending" sectors among other error indications in the log. I guess it's not the drive surface but the HDD controller (or may be a head) which is really dying. I figured the uncorrectable errors are write errors which is not surprising given the perceived "health" of the drive according to it's SMART attributes and error logs. That's understandable. Although, I tried to copy data from the filesystem and it failed at various ways. There was a file which couldn't be copied at all. Good question why. I guess it's because the filesystem needs to be repaired to get the checksums and parities sorted out first. That's also understandable (though unexpected, I thought RAID-5 Btrfs is sort-of "self-healing" in these situations, it should theoretically still be able to reconstruct and present the correct data, based on checksums and parities seamlessly and only place error in the kernel log...). But the worst part is that there are some ISO files which were seemingly copied without errors but their external checksums (the one which I can calculate with md5sum and compare to the one supplied by the publisher of the ISO file) don't match! Well... this, I cannot understand. How could these files become corrupt from a single disk failure? And more importantly: how could these files be copied without errors? Why didn't Btrfs gave a read error when the checksums didn't add up? Isn't Btrfs supposed to constantly check the integrity of the file data during any normal read operations and give an error instead of spitting out corrupt data as if it was perfectly legit? I thought that's how it is supposed to work. What's the point of full data checksuming if only an explicitly requested scrub operation might look for errors? I thought's it's the logical thing to do if checksum verification happens during every single read operation and passing that check is mandatory in order to get any data out of the filesystem (might be excluding the Direct-I/O mode but I never use that on Btrfs - if that's even actually supported, I don't know). Now I am really considering to move from Linux to Windows and from Btrfs RAID-5 to Storage Spaces RAID-1 + ReFS (the only limitation is that ReFS is only "self-healing" on RAID-1, not RAID-5, so I need a new motherboard with more native SATA connectors and an extra HDD). That one seemed to actually do what it promises (abort any read operations upon checksum errors [which always happens seamlessly on every read] but look at the redundant data first and seamlessly "self-heal" if possible). The only thing which made Btrfs to look as a better alternative was the RAID-5 support. But I recently experienced two cases of 1 drive failing of 3 and it always tuned out as a smaller or bigger disaster (completely lost data or inconsistent data). Does anybody have ideas what might went wrong in this second scenario? -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Btrfs/RAID5 became unmountable after SATA cable fault
I tried that after every possible combinations of RO mount failed. I used it in the past for an USB attached drive where an USB-SATA adapter had some issues (I plugged it into a standard USB2 port even though it expected USB3 power current, so a high-current or several standard USB2 ports should have been used to prevent it from emergency shutdown during file copy). It worked flawlessly then but in this case it threw the same errors as "btrsf check". It couldn't even find and list alternative roots. Every tools seemed to give up really fast and easy, sometimes with segfaults and tracing messages in the kernel log.On Oct 21, 2015 22:26, Chris Murphy wrote: > > https://btrfs.wiki.kernel.org/index.php/Restore > > This should still be possible with even a degraded/unmounted raid5. It > is a bit tedious to figure out how to use it but if you've got some > things you want off the volume, it's not so difficult to prevent > trying it. > > > Chris Murphy
Re: Btrfs/RAID5 became unmountable after SATA cable fault
https://btrfs.wiki.kernel.org/index.php/Restore This should still be possible with even a degraded/unmounted raid5. It is a bit tedious to figure out how to use it but if you've got some things you want off the volume, it's not so difficult to prevent trying it. Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Btrfs/RAID5 became unmountable after SATA cable fault
I am afraid the filesystem right now is really damaged regardless of it's state upon the unexpected cable failure because I tried some dangerous options after read-only restore/recovery methods all failed (including zero-log, followed by init-csum-tree and even chunk-recovery -> all of them just spit out several kind of errors which suggested they probably didn't even write anything to the disks before they decided that they already failed but they only caused more harm than good if they did write something). Actually, I almost got rid of this data myself intentionally when my new set of drives arrived. I was considering if I should simply start from scratch (may be reviewing and might be saving my "scratchpad" portion of the data but nothing really irreplaceable and/or valuable) but I thought it's a good idea to test the "device replace" function in real life. Even though the replace operation seemed to be successful I am beginning to wonder if it wasn't really. On Wed, Oct 21, 2015 at 7:42 PM, ronnie sahlberg wrote: > Maybe hold off erasing the drives a little in case someone wants to > collect some extra data for diagnosing how/why the filesystem got into > this unrecoverable state. > > A single device having issues should not cause the whole filesystem to > become unrecoverable. > > On Wed, Oct 21, 2015 at 9:09 AM, Janos Toth F. wrote: >> I went through all the recovery options I could find (starting from >> read-only to "extraordinarily dangerous"). Nothing seemed to work. >> >> A Windows based proprietary recovery software (ReclaiMe) could scratch >> the surface but only that (it showed me the whole original folder >> structure after a few minutes of scanning and the "preview" of some >> some plaintext files was promising but most of the bigger files seemed >> to be broken). >> >> I used this as a bulk storage for backups and all the things I didn't >> care to keep in more than one copies but that includes my >> "scratchpad", so I cared enough to use RAID5 mode and to try restoring >> some things. >> >> Any last ideas before I "ata secure erase" and sell/repurpose the disks? >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Btrfs/RAID5 became unmountable after SATA cable fault
I tried several things, including the degraded mount option. One example: # mount /dev/sdb /data -o ro,degraded,nodatasum,notreelog mount: wrong fs type, bad option, bad superblock on /dev/sdb, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so. # cat /proc/kmsg <6>[ 262.616929] BTRFS info (device sdd): allowing degraded mounts <6>[ 262.616943] BTRFS info (device sdd): setting nodatasum <6>[ 262.616949] BTRFS info (device sdd): disk space caching is enabled <6>[ 262.616953] BTRFS: has skinny extents <6>[ 262.652671] BTRFS: bdev (null) errs: wr 858, rd 8057, flush 280, corrupt 0, gen 0 <3>[ 262.697162] BTRFS (device sdd): parent transid verify failed on 38719488 wanted 101765 found 101223 <3>[ 262.697633] BTRFS (device sdd): parent transid verify failed on 38719488 wanted 101765 found 101223 <3>[ 262.697660] BTRFS: Failed to read block groups: -5 <3>[ 262.709885] BTRFS: open_ctree failed <6>[ 267.197365] BTRFS info (device sdd): allowing degraded mounts <6>[ 267.197385] BTRFS info (device sdd): setting nodatasum <6>[ 267.197397] BTRFS info (device sdd): disabling tree log <6>[ 267.197406] BTRFS info (device sdd): disk space caching is enabled <6>[ 267.197412] BTRFS: has skinny extents <6>[ 267.232809] BTRFS: bdev (null) errs: wr 858, rd 8057, flush 280, corrupt 0, gen 0 <3>[ 267.246167] BTRFS (device sdd): parent transid verify failed on 38719488 wanted 101765 found 101223 <3>[ 267.246706] BTRFS (device sdd): parent transid verify failed on 38719488 wanted 101765 found 101223 <3>[ 267.246727] BTRFS: Failed to read block groups: -5 <3>[ 267.261392] BTRFS: open_ctree failed On Wed, Oct 21, 2015 at 6:09 PM, Janos Toth F. wrote: > I went through all the recovery options I could find (starting from > read-only to "extraordinarily dangerous"). Nothing seemed to work. > > A Windows based proprietary recovery software (ReclaiMe) could scratch > the surface but only that (it showed me the whole original folder > structure after a few minutes of scanning and the "preview" of some > some plaintext files was promising but most of the bigger files seemed > to be broken). > > I used this as a bulk storage for backups and all the things I didn't > care to keep in more than one copies but that includes my > "scratchpad", so I cared enough to use RAID5 mode and to try restoring > some things. > > Any last ideas before I "ata secure erase" and sell/repurpose the disks? -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Btrfs/RAID5 became unmountable after SATA cable fault
Maybe hold off erasing the drives a little in case someone wants to collect some extra data for diagnosing how/why the filesystem got into this unrecoverable state. A single device having issues should not cause the whole filesystem to become unrecoverable. On Wed, Oct 21, 2015 at 9:09 AM, Janos Toth F. wrote: > I went through all the recovery options I could find (starting from > read-only to "extraordinarily dangerous"). Nothing seemed to work. > > A Windows based proprietary recovery software (ReclaiMe) could scratch > the surface but only that (it showed me the whole original folder > structure after a few minutes of scanning and the "preview" of some > some plaintext files was promising but most of the bigger files seemed > to be broken). > > I used this as a bulk storage for backups and all the things I didn't > care to keep in more than one copies but that includes my > "scratchpad", so I cared enough to use RAID5 mode and to try restoring > some things. > > Any last ideas before I "ata secure erase" and sell/repurpose the disks? > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Btrfs/RAID5 became unmountable after SATA cable fault
If it is for mostly archival storage, I would suggest you take a look at snapraid. On Wed, Oct 21, 2015 at 9:09 AM, Janos Toth F. wrote: > I went through all the recovery options I could find (starting from > read-only to "extraordinarily dangerous"). Nothing seemed to work. > > A Windows based proprietary recovery software (ReclaiMe) could scratch > the surface but only that (it showed me the whole original folder > structure after a few minutes of scanning and the "preview" of some > some plaintext files was promising but most of the bigger files seemed > to be broken). > > I used this as a bulk storage for backups and all the things I didn't > care to keep in more than one copies but that includes my > "scratchpad", so I cared enough to use RAID5 mode and to try restoring > some things. > > Any last ideas before I "ata secure erase" and sell/repurpose the disks? > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Btrfs/RAID5 became unmountable after SATA cable fault
I went through all the recovery options I could find (starting from read-only to "extraordinarily dangerous"). Nothing seemed to work. A Windows based proprietary recovery software (ReclaiMe) could scratch the surface but only that (it showed me the whole original folder structure after a few minutes of scanning and the "preview" of some some plaintext files was promising but most of the bigger files seemed to be broken). I used this as a bulk storage for backups and all the things I didn't care to keep in more than one copies but that includes my "scratchpad", so I cared enough to use RAID5 mode and to try restoring some things. Any last ideas before I "ata secure erase" and sell/repurpose the disks? -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Btrfs/RAID5 became unmountable after SATA cable fault
Janos Toth F. posted on Mon, 19 Oct 2015 10:39:06 +0200 as excerpted: > I was in the middle of replacing the drives of my NAS one-by-one (I > wished to move to bigger and faster storage at the end), so I used one > more SATA drive + SATA cable than usual. Unfortunately, the extra cable > turned out to be faulty and it looks like it caused some heavy damage to > the file system. > > There was no "devive replace" running at the moment or the disaster. The > first round already got finished hours ago and I planned to start the > next one before going to sleep. So, it was a full RAID-5 setup in normal > state. But one of the active, mounted devices was the first replacment > HDD and it was hanging on the spare SATA cable. > > I tried to save some file to my mounted samba share and I realized the > file system because read-only. I rebooted the machine and saw that my > /data can't be mounted. > According to SmartmonTools, one of the drives was suffering from SATA > communication errors. > > I tried some tirivial recovery methods and I tried to search the mailing > list archives but I didn't really find a solution. I wonder if somebody > can help with this. > > Should I run "btrfs rescue chunk-recover /dev/sda"? > > Here are some raw details: > > # uname -a > Linux F17a_NAS 4.2.3-gentoo #2 SMP Sun Oct 18 17:56:45 CEST 2015 > x86_64 AMD E-350 Processor AuthenticAMD GNU/Linux > > # btrfs --version > btrfs-progs v4.2.2 OK, nice and current kernel and userspace, vital for btrfs raid56 mode especially, as it's so new... > # mount /dev/sda /data -o ro,recovery > mount: wrong fs type, bad option, bad superblock on /dev/sda, ... Did you try mount -o degraded ? The recovery option is for a different problem and you probably simply need degraded at this point. But there's no indication that you tried degraded. If one of the devices is/was on a faulting cable, then it's likely not being detected properly, and btrfs is simply failing to mount it business- as-usual because it's degraded, and btrfs wants to ensure that you know that before it lets you mount, to maximize the chances of fixing the problem before something else goes wrong as well, where the two problems combined really /could/ screw up the filesystem beyond simple repair. Do note that due to a bug with the current kernel, you may get just one writable mount to fix the problem. Attempting a second degraded writable mount will often fail due to the bug, and you can only do degraded,ro after that, which will let you copy off the data to elsewhere but won't let you repair the filesystem, as a writable mount is required for that. So if you don't have current backups and you want to maximize your chance of saving the data, mounting degraded,ro *first* and (assuming it mounts) taking that opportunity to backup, before attempting to mount degraded,rw in ordered to replace the screwed up device, is recommended. Once you have a current backup of anything on it that you consider important[1], then do a btrfs filesystem show and see if it says some devices missing (as it probably will if you had to use the degraded option), after which you can mount degraded,rw and replace/remove the missing device. Meanwhile, should you end up having mounted degraded,rw, and not gotten the missing devices properly deleted/replaced before a second degraded mount, which probably won't let you mount rw due to that bug I mentioned, there's patches available, I believe already applied to the latest kernel 4.3-rc, that should let you mount degraded,rw more than once, if the data and metadata is actually all still available. The bug is that btrfs currently looks only at the number of devices and the chunk-types when deciding whether a filesystem can be mounted degraded,rw, while if new data is written in degraded,rw mode, it may have to fallback new writes to single chunk write mode. That's fine for the first writable mount, but on the second, it sees those single chunks and devices missing, and thinks that single mode with devices missing means data is missing too, even tho as long as no further devices went missing, it could have only been written to the present devices, so no data should actually be missing at all. But current btrfs doesn't know that as it only looks at the number of devices vs what's supposed to be there, along with the chunk type, and if there's any single type chunks with devices missing, it gives up and won't allow writable mount. What the patches make btrfs do instead is look at the individual chunks. If all the chunks are actually available (as they should be in this second degraded writable-mount scenario), then it can still mount degraded,rw despite missing devices, thus allowing you to actually remove/ replace the missing device, since it needs writable mount in ordered to be able to do that remove/replace. --- [1] If you're a regular reader of the list, you'll know that I regularly point to the sysadmin
Btrfs/RAID5 became unmountable after SATA cable fault
I was in the middle of replacing the drives of my NAS one-by-one (I wished to move to bigger and faster storage at the end), so I used one more SATA drive + SATA cable than usual. Unfortunately, the extra cable turned out to be faulty and it looks like it caused some heavy damage to the file system. There was no "devive replace" running at the moment or the disaster. The first round already got finished hours ago and I planned to start the next one before going to sleep. So, it was a full RAID-5 setup in normal state. But one of the active, mounted devices was the first replacment HDD and it was hanging on the spare SATA cable. I tried to save some file to my mounted samba share and I realized the file system because read-only. I rebooted the machine and saw that my /data can't be mounted. According to SmartmonTools, one of the drives was suffering from SATA communication errors. I tried some tirivial recovery methods and I tried to search the mailing list archives but I didn't really find a solution. I wonder if somebody can help with this. Should I run "btrfs rescue chunk-recover /dev/sda"? Here are some raw details: # uname -a Linux F17a_NAS 4.2.3-gentoo #2 SMP Sun Oct 18 17:56:45 CEST 2015 x86_64 AMD E-350 Processor AuthenticAMD GNU/Linux # btrfs --version btrfs-progs v4.2.2 # btrfs check /dev/sda checksum verify failed on 21102592 found 295F0086 wanted checksum verify failed on 21102592 found 295F0086 wanted checksum verify failed on 21102592 found 99D0FC26 wanted B08FFCA0 checksum verify failed on 21102592 found 99D0FC26 wanted B08FFCA0 bytenr mismatch, want=21102592, have=65536 Couldn't read chunk root Couldn't open file system # mount /dev/sda /data -o ro,recovery mount: wrong fs type, bad option, bad superblock on /dev/sda, ... # cat /proc/kmsg <6>[ 1902.033164] BTRFS info (device sdb): enabling auto recovery <6>[ 1902.033184] BTRFS info (device sdb): disk space caching is enabled <6>[ 1902.033191] BTRFS: has skinny extents <3>[ 1902.034931] BTRFS (device sdb): bad tree block start 0 21102592 <3>[ 1902.051259] BTRFS (device sdb): parent transid verify failed on 21147648 wanted 101748 found 101124 <3>[ 1902.111807] BTRFS (device sdb): parent transid verify failed on 44613632 wanted 101770 found 101233 <3>[ 1902.126529] BTRFS (device sdb): parent transid verify failed on 40595456 wanted 101767 found 101232 <6>[ 1902.164667] BTRFS: bdev /dev/sda errs: wr 858, rd 8057, flush 280, corrupt 0, gen 0 <3>[ 1902.165929] BTRFS (device sdb): parent transid verify failed on 44617728 wanted 101770 found 101233 <3>[ 1902.166975] BTRFS (device sdb): parent transid verify failed on 44621824 wanted 101770 found 101233 <3>[ 1902.271296] BTRFS (device sdb): parent transid verify failed on 38621184 wanted 101765 found 101223 <3>[ 1902.380526] BTRFS (device sdb): parent transid verify failed on 38719488 wanted 101765 found 101223 <3>[ 1902.381510] BTRFS (device sdb): parent transid verify failed on 38719488 wanted 101765 found 101223 <3>[ 1902.381549] BTRFS: Failed to read block groups: -5 <3>[ 1902.394835] BTRFS: open_ctree failed <6>[ 1911.202254] BTRFS info (device sdb): enabling auto recovery <6>[ 1911.202270] BTRFS info (device sdb): disk space caching is enabled <6>[ 1911.202275] BTRFS: has skinny extents <3>[ 1911.203611] BTRFS (device sdb): bad tree block start 0 21102592 <3>[ 1911.204803] BTRFS (device sdb): parent transid verify failed on 21147648 wanted 101748 found 101124 <3>[ 1911.246384] BTRFS (device sdb): parent transid verify failed on 44613632 wanted 101770 found 101233 <3>[ 1911.248729] BTRFS (device sdb): parent transid verify failed on 40595456 wanted 101767 found 101232 <6>[ 1911.251658] BTRFS: bdev /dev/sda errs: wr 858, rd 8057, flush 280, corrupt 0, gen 0 <3>[ 1911.252485] BTRFS (device sdb): parent transid verify failed on 44617728 wanted 101770 found 101233 <3>[ 1911.253542] BTRFS (device sdb): parent transid verify failed on 44621824 wanted 101770 found 101233 <3>[ 1911.278414] BTRFS (device sdb): parent transid verify failed on 38621184 wanted 101765 found 101223 <3>[ 1911.283950] BTRFS (device sdb): parent transid verify failed on 38719488 wanted 101765 found 101223 <3>[ 1911.284835] BTRFS (device sdb): parent transid verify failed on 38719488 wanted 101765 found 101223 <3>[ 1911.284873] BTRFS: Failed to read block groups: -5 <3>[ 1911.298783] BTRFS: open_ctree failed # btrfs-show-super /dev/sda superblock: bytenr=65536, device=/dev/sda - csum0xe8789014 [match] bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid2bba7cff-b4bf-4554-bee4-66f69c761ec4 label generation 101480 root37892096 sys_array_size 258 chunk_root_generation 101124 root_level 2 chunk_root 21147648 chunk_root_level1 log_root