Re: Filesystem will remount read-only
On Wed, Sep 21, 2016 at 9:24 AM, Jeffrey Michels wrote: > Hello, > > I've booted into the latest nightly build of Fedora and run btrfs rescue > super-recover -v and also btrfs check. > > Super-recover reports that "All supers are valid, no need to recover." Btrfs > check displays the same errors as before: > > Parent trasid verified failed on wanted found x > Ignoring transid failure > Leaf parent key incorrect xxx > Bad block xxx > Errors found in extend allocation tree or chunk allocation. > > The check eventually fails with Segmentation fault (core dumped.) > > Attempting to mount with -o recovery results in the error "Can't read > superblock." I am able to mount the filesystem read-only however attempting > to copy all of the data off has been unsuccessful. It appears to hit a bad > area and just hangs. > > Is there anything else I can try? Kinda weird that the supers are all valid with one tool but then the kernel says it can't read the superblock? Try btrfs-image -c9 -t4 and optionally with -s, and if the resulting file isn't too big put it somewhere and I can try to iterate some options on the file. You could do the same thing by imaging it, restoring the image to an LVM LV of the same size or bigger, and then see if there's a combination of btrfs check --repair; btrfs check --repair --init-extent-tree btrfs rescue chunk-recover btrfs rescue zero-log That or some other order of things might fix it, and the only way to know is to try it. And unfortunately each thing can make changes that can cause the next thing to fail. Hence doing this on an image. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Filesystem will remount read-only
Hello, I've booted into the latest nightly build of Fedora and run btrfs rescue super-recover -v and also btrfs check. Super-recover reports that "All supers are valid, no need to recover." Btrfs check displays the same errors as before: Parent trasid verified failed on wanted found x Ignoring transid failure Leaf parent key incorrect xxx Bad block xxx Errors found in extend allocation tree or chunk allocation. The check eventually fails with Segmentation fault (core dumped.) Attempting to mount with -o recovery results in the error "Can't read superblock." I am able to mount the filesystem read-only however attempting to copy all of the data off has been unsuccessful. It appears to hit a bad area and just hangs. Is there anything else I can try? Thank you for your insight, Jeff Michels -Original Message- From: ch...@colorremedies.com [mailto:ch...@colorremedies.com] On Behalf Of Chris Murphy Sent: Friday, September 16, 2016 7:08 PM To: Jeffrey Michels Cc: linux-btrfs@vger.kernel.org Subject: Re: Filesystem will remount read-only On Fri, Sep 16, 2016 at 8:57 AM, Jeffrey Michels wrote: > Hello, > > I have a system that has been in production for a few years. The SAN the VM > was running on had a hardware failure about a month ago and now one of the > two btrfs filesystems will remount after boot read-only. Here is the system > information: > > uname -a > > Linux retain 3.0.101-0.47.71-default #1 SMP Thu Nov 12 12:22:22 UTC > 2015 (b5b212e) x86_64 x86_64 x86_64 GNU/Linux > > Btrfs --version > > Btrfs v0.20+ Impressive that it's been running in production this long and with old kernel. I like it! Anyway, you could try mounting with -o recovery and see if that works. That's about the only thing I'd trust with such an old kernel and btrfs-progs. I don't even think it's worth trying the btrfsck on v0.20 just to see what the problems might be, and certainly not for actually using the repair mode. Actually I'm not even sure progs that old even does repairs, it might be the era of notify only. If -o recovery doesn't work, you'll need to use something newer, you could use one of: Fedora Rawhide nightly with 4.8rc6 kernel and btrfs-progs 4.7.2. This is a small netinstall image. dd to a USB stick, choose Troubleshooting option, then the Rescue option, then after startup use the 3 option to get to a shell where you can try to mount normally, or use btrfs-check. Limited tty, no sshd. https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20160914.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20160914.n.0.iso.n.0.iso Or something more official with published hash's for the image and a GUI, Fedora 24 workstation has kernel 4.5.5 and btrfs-progs 4.5.2 https://getfedora.org/en/workstation/download/ -- Chris Murphy iCon 2017 Registration is Now Open! Agents of Innovation March 8 - 10, 2017 TradeWinds Island Resort, St. Pete Beach, Florida Register today at: www.skyward.com/icon PRIVILEGED AND CONFIDENTIAL Skyward Communication This is a transmission from Skyward, Inc. and may contain information which is privileged, confidential, and protected by service work privileges. The response is in direct relationship to the information provided to Skyward. If you are not the addressee, note that any disclosure, copying, distribution, or use of the contents of this message is prohibited. If you have received this transmission in error, please destroy it and notify us immediately at 715-341-9406. N�r��yb�X��ǧv�^�){.n�+{�n�߲)w*jg����ݢj/���z�ޖ��2�ޙ&�)ߡ�a�����G���h��j:+v���w��٥
Re: Filesystem will remount read-only
On Fri, Sep 16, 2016 at 6:08 PM, Chris Murphy wrote: > > If -o recovery doesn't work, you'll need to use something newer, you > could use one of: > > Fedora Rawhide nightly with 4.8rc6 kernel and btrfs-progs 4.7.2. This > is a small netinstall image. dd to a USB stick, choose Troubleshooting > option, then the Rescue option, then after startup use the 3 option to > get to a shell where you can try to mount normally, or use > btrfs-check. Limited tty, no sshd. > https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20160914.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20160914.n.0.iso.n.0.iso > > Or something more official with published hash's for the image and a > GUI, Fedora 24 workstation has kernel 4.5.5 and btrfs-progs 4.5.2 > https://getfedora.org/en/workstation/download/ Just to complete the thought... use these just to boot and have access to something newer. I'm not suggesting install them. First try a normal mount, and if that fails, try -o recovery, if that fails, I'm curious about btrfs rescue super-recover -v btrfs check What I'm after is a way to get it to mount cleanly with a new kernel, and then hoping you can then just reboot with the ancient kernel and it'll be back to normal. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Filesystem will remount read-only
On Fri, Sep 16, 2016 at 8:57 AM, Jeffrey Michels wrote: > Hello, > > I have a system that has been in production for a few years. The SAN the VM > was running on had a hardware failure about a month ago and now one of the > two btrfs filesystems will remount after boot read-only. Here is the system > information: > > uname -a > > Linux retain 3.0.101-0.47.71-default #1 SMP Thu Nov 12 12:22:22 UTC 2015 > (b5b212e) x86_64 x86_64 x86_64 GNU/Linux > > Btrfs --version > > Btrfs v0.20+ Impressive that it's been running in production this long and with old kernel. I like it! Anyway, you could try mounting with -o recovery and see if that works. That's about the only thing I'd trust with such an old kernel and btrfs-progs. I don't even think it's worth trying the btrfsck on v0.20 just to see what the problems might be, and certainly not for actually using the repair mode. Actually I'm not even sure progs that old even does repairs, it might be the era of notify only. If -o recovery doesn't work, you'll need to use something newer, you could use one of: Fedora Rawhide nightly with 4.8rc6 kernel and btrfs-progs 4.7.2. This is a small netinstall image. dd to a USB stick, choose Troubleshooting option, then the Rescue option, then after startup use the 3 option to get to a shell where you can try to mount normally, or use btrfs-check. Limited tty, no sshd. https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20160914.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20160914.n.0.iso.n.0.iso Or something more official with published hash's for the image and a GUI, Fedora 24 workstation has kernel 4.5.5 and btrfs-progs 4.5.2 https://getfedora.org/en/workstation/download/ -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Filesystem will remount read-only
Jeffrey Michels posted on Fri, 16 Sep 2016 14:57:43 + as excerpted: > Hello, > > I have a system that has been in production for a few years. The SAN > the VM was running on had a hardware failure about a month ago and now > one of the two btrfs filesystems will remount after boot read-only. > Here is the system information: > > uname -a > > Linux retain 3.0.101-0.47.71-default #1 SMP Thu Nov 12 12:22:22 UTC 2015 > (b5b212e) x86_64 x86_64 x86_64 GNU/Linux > > Btrfs --version > > Btrfs v0.20+ That is positively /ancient/, both kernel and userspace (btrfs-progs). Keep in mind that btrfs was still considered very experimental back then, with the experimental labels coming off only with 3.14 or there abouts, IIRC (userspace releases got version-synced with kernelspace in 3.12, so 3.14 applies to both). So you have been running an at-the-time still extremely experimental filesystem for years now, and it's only now coming up with problems that need fixed. Pretty remarkable for the experimental state back then, but it doesn't change the fact that it /was/ "may eat your data and burn your kids alive as a sacrifice to appease the filesystem gods" level experimental, with the according warnings, back then. So first thing I'd suggest is to update to kernel 4.4 LTS series, and something similar for btrfs-progs userspace. Then, given the age and experimental nature of the filesystem back then, I'd kill the filesystems and do a fresh mkfs.btrfs, restoring from backups. That way you're starting with a well tested and stable LTS kernel that is both reasonably mature already, and will be supported for some time to come, and eliminate any possibility of long fixed and forgotten bugs coming back to bite you years later. Alternatively, if you're using a long-term support distro, you have the choice of going to them for that support, since unlike this list which focuses on the state going forward, that sort of deep long-term support of long outdated versions is a good part of the reason such distros exist, and a good part of why a lot of people are willing to pay sometimes rather sizable sums of money /for/ that level of support. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html