Sorry, I'm not at all familiar with backtrace. Thanks,
Scotty Edmonds sco...@scottyedmonds.com ________________________________________ From: Qu Wenruo <quwen...@cn.fujitsu.com> Sent: November-17-15 3:35 AM To: Scotty Edmonds; Hugo Mills; Donald Pearson Cc: Btrfs BTRFS Subject: Re: BTRFS Error - Rockstor Scotty Edmonds wrote on 2015/11/17 07:09 +0000: > This was one of the first things I tried actually, this was the result. > > [root@rockstor ~]# btrfs rescue chunk-recover -y /dev/sdg > Scanning: DONE in dev0, DONE in dev1, DONE in dev2, DONE in dev3, DONE in > dev4 Floating > point exception Did you have a gdb backtrace of that FPE? Thanks, QU > [root@rockstor ~]# > > > Thanks, > > Scotty Edmonds > sco...@scottyedmonds.com > > ________________________________________ > From: Qu Wenruo <quwen...@cn.fujitsu.com> > Sent: November-15-15 9:06 PM > To: Scotty Edmonds; Hugo Mills; Donald Pearson > Cc: Btrfs BTRFS > Subject: Re: BTRFS Error - Rockstor > > Oh, before really doing some coding to allow btrfs-find-root to search > under chunk tree broken case, > It occurs to me that we have already a better recovery tool for it. > > You can use 'btrfs rescue chunk-recovery' to recovery your chunk tree. > It may has some risk, but the overall work flow is much the same as what > I am going to do, but with better cross check with extent tree. > > Since your chunk tree is already broken, almost nothing to lose further, > so I'd recommend you to try "btrfs rescue chunk-recovery". > > Thanks, > Qu > > Scotty Edmonds wrote on 2015/11/13 02:23 +0000: >> yes, no problem. I had it powered off as I'm moving the system to a proper >> chassis and won't have it for another two weeks. >> >> Thanks, >> >> Scotty Edmonds >> sco...@scottyedmonds.com >> >> ________________________________________ >> From: Qu Wenruo <quwen...@cn.fujitsu.com> >> Sent: November-12-15 10:21 PM >> To: Scotty Edmonds; Hugo Mills; Donald Pearson >> Cc: Btrfs BTRFS >> Subject: Re: BTRFS Error - Rockstor >> >> Chunk root seems corrupted. >> But that's not a really huge problem, tree root corruption happens, and >> thanks to the full CoW of btrfs metadata, we will always be able to find >> a old version one. >> >> I'd like to do a full disk scan for any earlier version chunk root, but >> I found that btrfs-find-root doesn't support search for chunk root. >> (Hey, who is the bad ass wrote btrfs-find-root and made chunk root >> search unsupported?! Oh, that's myself) >> >> If you can wait, I'll add chunk root search support for you in recent >> days, and then hopes we can find something helpful. >> >> Thanks, >> Qu >> >> Scotty Edmonds wrote on 2015/11/13 01:46 +0000: >>> Thanks for the help, I got the same error on all variants. >>> >>> [root@rockstor ~]# btrfs check --readonly -s 0 /dev/sdh >>> using SB copy 0, bytenr 65536 >>> checksum verify failed on 12060305965056 found 779CCA23 wanted A746C37A >>> checksum verify failed on 12060305965056 found 779CCA23 wanted A746C37A >>> checksum verify failed on 12060305965056 found 1727A198 wanted 231E1577 >>> checksum verify failed on 12060305965056 found 1727A198 wanted 231E1577 >>> bytenr mismatch, want=12060305965056, have=13820656527619066643 >>> Couldn't read chunk tree >>> Couldn't open file system >>> [root@rockstor ~]# >>> >>> >>> Thanks, >>> >>> Scotty Edmonds >>> sco...@scottyedmonds.com >>> >>> ________________________________________ >>> From: Hugo Mills <h...@carfax.org.uk> >>> Sent: November-12-15 6:57 PM >>> To: Donald Pearson >>> Cc: Scotty Edmonds; Btrfs BTRFS >>> Subject: Re: BTRFS Error - Rockstor >>> >>> On IRC earlier, I asked for the btrfs-debug-tree output of the >>> broken tree block (1205030...etc). Since it's also failing, that would >>> kind of indicate that this is pretty badly broken for some reason. >>> >>> It doesn't quite feel like a broken disk to me, but I'm not sure >>> what _has_ happened. Looks like something has stomped on a piece of >>> metadata fairly high up in the data structures. >>> >>> It probably won't show anything different, but could you do >>> >>> $ btrfs check --readonly -s $N /dev/$D >>> >>> for values of $N from 0 to 3, and for all the devices $D? I'm >>> expecting to see the same errors (except for -s3, which is probably >>> out of range), but if by any chance you get something different, that >>> may give us a way into recovery. >>> >>> Hugo. >>> >>> On Thu, Nov 12, 2015 at 04:41:58PM -0600, Donald Pearson wrote: >>>> On Thu, Nov 12, 2015 at 4:24 PM, Scotty Edmonds >>>> <sco...@scottyedmonds.com> wrote: >>>>> Not exactly sure what to look for in dmesg.. If it is a disk fail >>>>> shouldn't I just be able to remove the disk as it's RAID5? >>>>> >>>> >>>> Yes theoretically. >>>> >>>> >>>>> [ 20.323997] BTRFS: device label seagate3x2tb devid 2 transid 2315 >>>>> /dev/sdc >>>>> [ 20.324387] BTRFS: device label seagate3x2tb devid 1 transid 2315 >>>>> /dev/sda >>>>> [ 20.324601] BTRFS: device label seagate3x2tb devid 3 transid 2315 >>>>> /dev/sdd >>>>> [ 20.324698] BTRFS: device label mainNAS devid 1 transid 25209 /dev/sdg >>>>> [ 20.324794] BTRFS: device label mainNAS devid 2 transid 25209 /dev/sdf >>>>> [ 20.324938] BTRFS: device label mainNAS devid 5 transid 25209 /dev/sde >>>>> [ 20.325124] BTRFS: device label mainNAS devid 4 transid 25209 /dev/sdb >>>>> [ 20.325256] BTRFS: device label mainNAS devid 3 transid 25209 /dev/sdh >>>>> [ 105.285746] BTRFS info (device sdh): disk space caching is enabled >>>>> [ 105.285753] BTRFS: has skinny extents >>>>> [ 105.756545] BTRFS (device sdh): bad tree block start >>>>> 13820666663704185619 12060305965056 >>>>> [ 105.758877] BTRFS (device sdh): bad tree block start >>>>> 13820666663704185619 12060305965056 >>>>> [ 105.759154] BTRFS (device sdh): bad tree block start >>>>> 13820666663704185619 12060305965056 >>>>> [ 105.759340] BTRFS (device sdh): bad tree block start >>>>> 13820666663704185619 12060305965056 >>>>> [ 105.759417] BTRFS: failed to read chunk tree on sdh >>>>> [ 105.774774] BTRFS: open_ctree failed >>>>> [ 127.736060] BTRFS info (device sdd): disk space caching is enabled >>>>> [ 127.736066] BTRFS: has skinny extents >>>>> [ 141.887422] BTRFS info (device sdh): disk space caching is enabled >>>>> [ 141.887428] BTRFS: has skinny extents >>>>> [ 141.899666] BTRFS (device sdh): bad tree block start >>>>> 13820666663704185619 12060305965056 >>>>> [ 141.902385] BTRFS (device sdh): bad tree block start >>>>> 13820666663704185619 12060305965056 >>>>> [ 141.902639] BTRFS (device sdh): bad tree block start >>>>> 13820666663704185619 12060305965056 >>>>> [ 141.902795] BTRFS (device sdh): bad tree block start >>>>> 13820666663704185619 12060305965056 >>>>> [ 141.902870] BTRFS: failed to read chunk tree on sdh >>>>> [ 141.915337] BTRFS: open_ctree failed >>>>> [17748.031552] BTRFS info (device sdh): disk space caching is enabled >>>>> [17748.031559] BTRFS: has skinny extents >>>>> [17748.072339] BTRFS (device sdh): bad tree block start >>>>> 13820666663704185619 12060305965056 >>>>> [17748.077023] BTRFS (device sdh): bad tree block start >>>>> 13820666663704185619 12060305965056 >>>>> [17748.077350] BTRFS (device sdh): bad tree block start >>>>> 13820666663704185619 12060305965056 >>>>> [17748.077511] BTRFS (device sdh): bad tree block start >>>>> 13820666663704185619 12060305965056 >>>>> [17748.077587] BTRFS: failed to read chunk tree on sdh >>>>> [17748.088908] BTRFS: open_ctree failed >>>>> [17800.758291] BTRFS info (device sdh): disk space caching is enabled >>>>> [17800.758298] BTRFS: has skinny extents >>>>> [17800.765770] BTRFS (device sdh): bad tree block start >>>>> 13820666663704185619 12060305965056 >>>>> [17800.768816] BTRFS (device sdh): bad tree block start >>>>> 13820666663704185619 12060305965056 >>>>> [17800.769054] BTRFS (device sdh): bad tree block start >>>>> 13820666663704185619 12060305965056 >>>>> [17800.769192] BTRFS (device sdh): bad tree block start >>>>> 13820666663704185619 12060305965056 >>>>> [17800.769264] BTRFS: failed to read chunk tree on sdh >>>>> [17800.784937] BTRFS: open_ctree failed >>>>> [root@rockstor ~]# >>>>> >>>>> and then I get this: >>>>> >>>>> [root@rockstor ~]# btrfs-debug-tree -b 12060305965056 /dev/sdh >>>>> checksum verify failed on 12060305965056 found 779CCA23 wanted A746C37A >>>>> checksum verify failed on 12060305965056 found 779CCA23 wanted A746C37A >>>>> checksum verify failed on 12060305965056 found 1727A198 wanted 231E1577 >>>>> checksum verify failed on 12060305965056 found 1727A198 wanted 231E1577 >>>>> bytenr mismatch, want=12060305965056, have=13820656527619066643 >>>>> Couldn't read chunk tree >>>>> unable to open /dev/sdh >>>>> [root@rockstor ~]# >>>>> >>>>> Thanks, >>>>> >>>>> Scotty Edmonds >>>>> sco...@scottyedmonds.com >>>> >>>> I think we need to see what some of the more experienced users think >>>> on this one. But you can try removing sdh and seeing if you can mount >>>> it *read only* and degraded. Just make sure whatever you do and play >>>> with is done read only. Don't try any fixes or repairs with the tools >>>> unless told to do so by someone who really knows what they're talking >>>> about. >>>> >>>>> >>>>> ________________________________________ >>>>> From: Donald Pearson <donaldwhpear...@gmail.com> >>>>> Sent: November-12-15 6:19 PM >>>>> To: Scotty Edmonds; Btrfs BTRFS >>>>> Subject: Re: BTRFS Error - Rockstor >>>>> >>>>> Anything interesting in dmesg? >>>>> >>>>> That looks similar to the kind of problems I had when I had a disk fail. >>>>> >>>>> On Thu, Nov 12, 2015 at 4:08 PM, Scotty Edmonds >>>>> <sco...@scottyedmonds.com> wrote: >>>>>> I get this: >>>>>> >>>>>> [root@rockstor ~]# btrfs check /dev/sdd >>>>>> checksum verify failed on 12060305965056 found 779CCA23 wanted A746C37A >>>>>> checksum verify failed on 12060305965056 found 779CCA23 wanted A746C37A >>>>>> checksum verify failed on 12060305965056 found 1727A198 wanted 231E1577 >>>>>> checksum verify failed on 12060305965056 found 1727A198 wanted 231E1577 >>>>>> bytenr mismatch, want=12060305965056, have=13820656527619066643 >>>>>> Couldn't read chunk tree >>>>>> Couldn't open file system >>>>>> [root@rockstor ~]# >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Scotty Edmonds >>>>>> sco...@scottyedmonds.com >>>>>> >>>>>> ________________________________________ >>>>>> From: Donald Pearson <donaldwhpear...@gmail.com> >>>>>> Sent: November-12-15 2:55 PM >>>>>> To: Scotty Edmonds >>>>>> Cc: linux-btrfs@vger.kernel.org >>>>>> Subject: Re: BTRFS Error - Rockstor >>>>>> >>>>>> What does btrfs check without any repair options report? >>>>>> >>>>>> btrfs check /dev/sdd >>>>>> >>>>>> On Thu, Nov 12, 2015 at 12:48 PM, Scotty Edmonds >>>>>> <sco...@scottyedmonds.com> wrote: >>>>>>> Rockstor was running great, I ordered a SuperMicro 24-bay Chassis and >>>>>>> decided to power down the machine while I was away. When I turned it >>>>>>> back on I got "Failed to read chunk tree" & "open_ctree failed" error >>>>>>> (http://i.imgur.com/rGk9M57l.jpg) >>>>>>> >>>>>>> I spoke with support at Rockstor and they recommended I seek help via >>>>>>> the mailing list. Here are some details and commands I've run. The >>>>>>> specific array is in RAID5 and the label is mainNAS, seagate3x2tb is >>>>>>> running perfectly. >>>>>>> >>>>>>> [root@rockstor ~]# btrfs device scan >>>>>>> Scanning for Btrfs filesystems >>>>>>> [root@rockstor ~]# >>>>>>> >>>>>>> >>>>>>> >>>>>>> [root@rockstor ~]# /usr/bin/lsblk -P -o >>>>>>> NAME,MODEL,SERIAL,SIZE,TRAN,VENDOR,HCTL,TYPE,FSTYPE,LABEL,UUID >>>>>>> NAME="sda" MODEL="WDC WD30EFRX-68E" SERIAL="WD-WCC4N4KVC39Y" >>>>>>> SIZE="2.7T" TRAN="sas" VENDOR="ATA " HCTL="0:0:0:0" TYPE="disk" >>>>>>> FSTYPE="btrfs" LABEL="mainNAS" >>>>>>> UUID="e8c92d93-fac3-4f83-b3aa-31cb92caafd9" >>>>>>> NAME="sdb" MODEL="WDC WD30EZRX-00M" SERIAL="WD-WCAWZ2551761" >>>>>>> SIZE="2.7T" TRAN="sas" VENDOR="ATA " HCTL="0:0:1:0" TYPE="disk" >>>>>>> FSTYPE="btrfs" LABEL="mainNAS" >>>>>>> UUID="e8c92d93-fac3-4f83-b3aa-31cb92caafd9" >>>>>>> NAME="sdc" MODEL="HGST HDN724030AL" SERIAL="PK2234P9J590GY" SIZE="2.7T" >>>>>>> TRAN="sas" VENDOR="ATA " HCTL="0:0:2:0" TYPE="disk" FSTYPE="btrfs" >>>>>>> LABEL="mainNAS" UUID="e8c92d93-fac3-4f83-b3aa-31cb92caafd9" >>>>>>> NAME="sdd" MODEL="HGST HDN724030AL" SERIAL="PK2234P9J5WA1Y" SIZE="2.7T" >>>>>>> TRAN="sas" VENDOR="ATA " HCTL="0:0:3:0" TYPE="disk" FSTYPE="btrfs" >>>>>>> LABEL="mainNAS" UUID="e8c92d93-fac3-4f83-b3aa-31cb92caafd9" >>>>>>> NAME="sde" MODEL="ST3000DM001-1CH1" SERIAL="Z1F517PH" SIZE="2.7T" >>>>>>> TRAN="sas" VENDOR="ATA " HCTL="0:0:4:0" TYPE="disk" FSTYPE="btrfs" >>>>>>> LABEL="mainNAS" UUID="e8c92d93-fac3-4f83-b3aa-31cb92caafd9" >>>>>>> NAME="sdf" MODEL="ST2000DL003-9VT1" SERIAL="5YD1WK0V" SIZE="1.8T" >>>>>>> TRAN="sas" VENDOR="ATA " HCTL="0:0:5:0" TYPE="disk" FSTYPE="btrfs" >>>>>>> LABEL="seagate3x2tb" UUID="6ef19043-2d83-4ff1-b959-b9f3c425cc69" >>>>>>> NAME="sdg" MODEL="ST2000DL003-9VT1" SERIAL="5YD2EBDA" SIZE="1.8T" >>>>>>> TRAN="sas" VENDOR="ATA " HCTL="0:0:6:0" TYPE="disk" FSTYPE="btrfs" >>>>>>> LABEL="seagate3x2tb" UUID="6ef19043-2d83-4ff1-b959-b9f3c425cc69" >>>>>>> NAME="sdh" MODEL="ST2000DL003-9VT1" SERIAL="5YD2L28Z" SIZE="1.8T" >>>>>>> TRAN="sas" VENDOR="ATA " HCTL="0:0:7:0" TYPE="disk" FSTYPE="btrfs" >>>>>>> LABEL="seagate3x2tb" UUID="6ef19043-2d83-4ff1-b959-b9f3c425cc69" >>>>>>> NAME="sdi" MODEL="INTEL SSDSA2CW08" SERIAL="CVPR1330019Y080BGN" >>>>>>> SIZE="74.5G" TRAN="sata" VENDOR="ATA " HCTL="1:0:0:0" TYPE="disk" >>>>>>> FSTYPE="" LABEL="" UUID="" >>>>>>> NAME="sdi1" MODEL="" SERIAL="" SIZE="500M" TRAN="" VENDOR="" HCTL="" >>>>>>> TYPE="part" FSTYPE="ext4" LABEL="" >>>>>>> UUID="53aabf2f-5e28-4a18-922f-b0767a77a8ec" >>>>>>> NAME="sdi2" MODEL="" SERIAL="" SIZE="7.3G" TRAN="" VENDOR="" HCTL="" >>>>>>> TYPE="part" FSTYPE="swap" LABEL="" >>>>>>> UUID="bf9e72c7-7d72-4a33-a5eb-0a0013033234" >>>>>>> NAME="sdi3" MODEL="" SERIAL="" SIZE="66.8G" TRAN="" VENDOR="" HCTL="" >>>>>>> TYPE="part" FSTYPE="btrfs" LABEL="rockstor_rockstor" >>>>>>> UUID="3533171e-d95b-4491-aa4c-cc956536a1c3" >>>>>>> [root@rockstor ~]# >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> [root@rockstor ~]# btrfs fi show >>>>>>> Label: 'rockstor_rockstor' uuid: 3533171e-d95b-4491-aa4c-cc956536a1c3 >>>>>>> Total devices 1 FS bytes used 2.17GiB >>>>>>> devid 1 size 66.79GiB used 7.02GiB path /dev/sdi3 >>>>>>> >>>>>>> Label: 'seagate3x2tb' uuid: 6ef19043-2d83-4ff1-b959-b9f3c425cc69 >>>>>>> Total devices 3 FS bytes used 1.13TiB >>>>>>> devid 1 size 1.82TiB used 595.03GiB path /dev/sdh >>>>>>> devid 2 size 1.82TiB used 595.01GiB path /dev/sdf >>>>>>> devid 3 size 1.82TiB used 595.01GiB path /dev/sdg >>>>>>> >>>>>>> Label: 'mainNAS' uuid: e8c92d93-fac3-4f83-b3aa-31cb92caafd9 >>>>>>> Total devices 5 FS bytes used 5.43TiB >>>>>>> devid 1 size 2.73TiB used 1.36TiB path /dev/sdd >>>>>>> devid 2 size 2.73TiB used 1.36TiB path /dev/sdc >>>>>>> devid 3 size 2.73TiB used 1.36TiB path /dev/sda >>>>>>> devid 4 size 2.73TiB used 1.36TiB path /dev/sde >>>>>>> devid 5 size 2.73TiB used 1.36TiB path /dev/sdb >>>>>>> >>>>>>> >>>>>>> btrfs-progs v4.2.1 >>>>>>> >>>>>>> I'm unable to mount any of the drives that are in the mainNAS array, >>>>>>> this is the error when I try to mount all of the drives degraded. >>>>>>> >>>>>>> [root@rockstor ~]# mount -v -o degraded /dev/sdd /mnt2/mainNAS >>>>>>> mount: wrong fs type, bad option, bad superblock on /dev/sdd, >>>>>>> missing codepage or helper program, or other error >>>>>>> In some cases useful info is found in syslog - try >>>>>>> dmesg | tail or so. >>>>>>> [root@rockstor ~]# >>>>>>> >>>>>>> I haven't given up hope yet as the "btrfs fi show" gives me all the >>>>>>> correct data and I ran chunk-recover and superblocks all report back as >>>>>>> good. >>>>>>> >>>>>>> >>>>>>> Thanks for your help, let me know if you need any further information. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Scotty Edmonds >>>>>>> sco...@scottyedmonds.com >>> >>> -- >>> Hugo Mills | "How deep will this sub go?" >>> hugo@... carfax.org.uk | "Oh, she'll go all the way to the bottom if we >>> don't >>> http://carfax.org.uk/ | stop her." >>> PGP: E2AB1DE4 | >>> U571 >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>> the body of a message to majord...@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
