Got this: [root@rockstor ~]# btrfs-show-super -f /dev/sdg superblock: bytenr=65536, device=/dev/sdg --------------------------------------------------------- csum 0x793978ef [match] bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid e8c92d93-fac3-4f83-b3aa-31cb92caafd9 label mainNAS generation 25209 root 8640557629440 sys_array_size 225 chunk_root_generation 25196 root_level 1 chunk_root 12060305932288 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 15002964910080 bytes_used 5973553180672 sectorsize 4096 nodesize 16384 leafsize 16384 stripesize 4096 root_dir 6 num_devices 5 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x1e1 ( MIXED_BACKREF | BIG_METADATA | EXTENDED_IREF | RAID56 | SKINNY_METADATA ) csum_type 0 csum_size 4 cache_generation 25209 uuid_tree_generation 25209 dev_item.uuid 286474b1-210d-4db1-b97f-ff07116bf2f9 dev_item.fsid e8c92d93-fac3-4f83-b3aa-31cb92caafd9 [match] dev_item.type 0 dev_item.total_bytes 3000592982016 dev_item.bytes_used 1497618186240 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 1 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 12060305915904) chunk length 67108864 owner 2 stripe_len 65536 type SYSTEM|RAID5 num_stripes 5 stripe 0 devid 4 offset 1074790400 stripe 1 devid 5 offset 1074790400 stripe 2 devid 1 offset 1371537408 stripe 3 devid 3 offset 1351614464 stripe 4 devid 2 offset 1351614464 backup_roots[4]: backup 0: backup_tree_root: 8640557629440 gen: 25209 level: 1 backup_chunk_root: 12060305932288 gen: 25196 level: 1 backup_extent_root: 8640865320960 gen: 25209 level: 2 backup_fs_root: 6680017420288 gen: 25128 level: 0 backup_dev_root: 7742127030272 gen: 25196 level: 1 backup_csum_root: 8640864288768 gen: 25209 level: 3 backup_total_bytes: 15002964910080 backup_bytes_used: 5973553180672 backup_num_devices: 5
backup 1: backup_tree_root: 7742772379648 gen: 25206 level: 1 backup_chunk_root: 12060305932288 gen: 25196 level: 1 backup_extent_root: 7742740856832 gen: 25206 level: 2 backup_fs_root: 6680017420288 gen: 25128 level: 0 backup_dev_root: 7742127030272 gen: 25196 level: 1 backup_csum_root: 7742662393856 gen: 25206 level: 3 backup_total_bytes: 15002964910080 backup_bytes_used: 5972952920064 backup_num_devices: 5 backup 2: backup_tree_root: 7742732894208 gen: 25207 level: 1 backup_chunk_root: 12060305932288 gen: 25196 level: 1 backup_extent_root: 7742674255872 gen: 25207 level: 2 backup_fs_root: 6680017420288 gen: 25128 level: 0 backup_dev_root: 7742127030272 gen: 25196 level: 1 backup_csum_root: 7742665457664 gen: 25207 level: 3 backup_total_bytes: 15002964910080 backup_bytes_used: 5973141712896 backup_num_devices: 5 backup 3: backup_tree_root: 8640836403200 gen: 25208 level: 1 backup_chunk_root: 12060305932288 gen: 25196 level: 1 backup_extent_root: 8640865320960 gen: 25209 level: 2 backup_fs_root: 6680017420288 gen: 25128 level: 0 backup_dev_root: 7742127030272 gen: 25196 level: 1 backup_csum_root: 8640864288768 gen: 25209 level: 3 backup_total_bytes: 15002964910080 backup_bytes_used: 5973372186624 backup_num_devices: 5 [root@rockstor ~]# Thanks, Scotty Edmonds sco...@scottyedmonds.com ________________________________________ From: Qu Wenruo <quwen...@cn.fujitsu.com> Sent: November-12-15 11:00 PM To: Scotty Edmonds; Hugo Mills; Donald Pearson Cc: Btrfs BTRFS Subject: Re: BTRFS Error - Rockstor Although I submitted a patch, that won't work for you. As I found that your whole chunk tree is corrupted, so I'll need to add a new mode for you to handle such case, which will be much time consuming than the patch I submitted... But your superblock should has some glue if we are in good luck. Would you please paste the output of 'btrfs-show-super -f <device>'? If we are in good luck, it has something like: ------ backup 0: backup_tree_root: 30932992 gen: 10 level: 0 backup_chunk_root: 20971520 gen: 9 level: 0 backup_extent_root: 30916608 gen: 10 level: 0 backup_fs_root: 30982144 gen: 10 level: 0 backup_dev_root: 29458432 gen: 8 level: 0 backup_csum_root: 30949376 gen: 10 level: 1 backup_total_bytes: 1073741824 backup_bytes_used: 949682176 backup_num_devices: 1 backup 1: backup_tree_root: 30998528 gen: 11 level: 0 backup_chunk_root: 20971520 gen: 8 level: 0 backup_extent_root: 31014912 gen: 11 level: 0 backup_fs_root: 30982144 gen: 10 level: 0 backup_dev_root: 29458432 gen: 8 level: 0 backup_csum_root: 30949376 gen: 10 level: 1 backup_total_bytes: 1073741824 backup_bytes_used: 949682176 backup_num_devices: 1 ------ And if the back_chunk_root's gen is smaller than 'chunk_root_generation', you're in good luck and can use that chunk root to give it a try. Thanks, Qu Scotty Edmonds wrote on 2015/11/13 02:23 +0000: > yes, no problem. I had it powered off as I'm moving the system to a proper > chassis and won't have it for another two weeks. > > Thanks, > > Scotty Edmonds > sco...@scottyedmonds.com > > ________________________________________ > From: Qu Wenruo <quwen...@cn.fujitsu.com> > Sent: November-12-15 10:21 PM > To: Scotty Edmonds; Hugo Mills; Donald Pearson > Cc: Btrfs BTRFS > Subject: Re: BTRFS Error - Rockstor > > Chunk root seems corrupted. > But that's not a really huge problem, tree root corruption happens, and > thanks to the full CoW of btrfs metadata, we will always be able to find > a old version one. > > I'd like to do a full disk scan for any earlier version chunk root, but > I found that btrfs-find-root doesn't support search for chunk root. > (Hey, who is the bad ass wrote btrfs-find-root and made chunk root > search unsupported?! Oh, that's myself) > > If you can wait, I'll add chunk root search support for you in recent > days, and then hopes we can find something helpful. > > Thanks, > Qu > > Scotty Edmonds wrote on 2015/11/13 01:46 +0000: >> Thanks for the help, I got the same error on all variants. >> >> [root@rockstor ~]# btrfs check --readonly -s 0 /dev/sdh >> using SB copy 0, bytenr 65536 >> checksum verify failed on 12060305965056 found 779CCA23 wanted A746C37A >> checksum verify failed on 12060305965056 found 779CCA23 wanted A746C37A >> checksum verify failed on 12060305965056 found 1727A198 wanted 231E1577 >> checksum verify failed on 12060305965056 found 1727A198 wanted 231E1577 >> bytenr mismatch, want=12060305965056, have=13820656527619066643 >> Couldn't read chunk tree >> Couldn't open file system >> [root@rockstor ~]# >> >> >> Thanks, >> >> Scotty Edmonds >> sco...@scottyedmonds.com >> >> ________________________________________ >> From: Hugo Mills <h...@carfax.org.uk> >> Sent: November-12-15 6:57 PM >> To: Donald Pearson >> Cc: Scotty Edmonds; Btrfs BTRFS >> Subject: Re: BTRFS Error - Rockstor >> >> On IRC earlier, I asked for the btrfs-debug-tree output of the >> broken tree block (1205030...etc). Since it's also failing, that would >> kind of indicate that this is pretty badly broken for some reason. >> >> It doesn't quite feel like a broken disk to me, but I'm not sure >> what _has_ happened. Looks like something has stomped on a piece of >> metadata fairly high up in the data structures. >> >> It probably won't show anything different, but could you do >> >> $ btrfs check --readonly -s $N /dev/$D >> >> for values of $N from 0 to 3, and for all the devices $D? I'm >> expecting to see the same errors (except for -s3, which is probably >> out of range), but if by any chance you get something different, that >> may give us a way into recovery. >> >> Hugo. >> >> On Thu, Nov 12, 2015 at 04:41:58PM -0600, Donald Pearson wrote: >>> On Thu, Nov 12, 2015 at 4:24 PM, Scotty Edmonds >>> <sco...@scottyedmonds.com> wrote: >>>> Not exactly sure what to look for in dmesg.. If it is a disk fail >>>> shouldn't I just be able to remove the disk as it's RAID5? >>>> >>> >>> Yes theoretically. >>> >>> >>>> [ 20.323997] BTRFS: device label seagate3x2tb devid 2 transid 2315 >>>> /dev/sdc >>>> [ 20.324387] BTRFS: device label seagate3x2tb devid 1 transid 2315 >>>> /dev/sda >>>> [ 20.324601] BTRFS: device label seagate3x2tb devid 3 transid 2315 >>>> /dev/sdd >>>> [ 20.324698] BTRFS: device label mainNAS devid 1 transid 25209 /dev/sdg >>>> [ 20.324794] BTRFS: device label mainNAS devid 2 transid 25209 /dev/sdf >>>> [ 20.324938] BTRFS: device label mainNAS devid 5 transid 25209 /dev/sde >>>> [ 20.325124] BTRFS: device label mainNAS devid 4 transid 25209 /dev/sdb >>>> [ 20.325256] BTRFS: device label mainNAS devid 3 transid 25209 /dev/sdh >>>> [ 105.285746] BTRFS info (device sdh): disk space caching is enabled >>>> [ 105.285753] BTRFS: has skinny extents >>>> [ 105.756545] BTRFS (device sdh): bad tree block start >>>> 13820666663704185619 12060305965056 >>>> [ 105.758877] BTRFS (device sdh): bad tree block start >>>> 13820666663704185619 12060305965056 >>>> [ 105.759154] BTRFS (device sdh): bad tree block start >>>> 13820666663704185619 12060305965056 >>>> [ 105.759340] BTRFS (device sdh): bad tree block start >>>> 13820666663704185619 12060305965056 >>>> [ 105.759417] BTRFS: failed to read chunk tree on sdh >>>> [ 105.774774] BTRFS: open_ctree failed >>>> [ 127.736060] BTRFS info (device sdd): disk space caching is enabled >>>> [ 127.736066] BTRFS: has skinny extents >>>> [ 141.887422] BTRFS info (device sdh): disk space caching is enabled >>>> [ 141.887428] BTRFS: has skinny extents >>>> [ 141.899666] BTRFS (device sdh): bad tree block start >>>> 13820666663704185619 12060305965056 >>>> [ 141.902385] BTRFS (device sdh): bad tree block start >>>> 13820666663704185619 12060305965056 >>>> [ 141.902639] BTRFS (device sdh): bad tree block start >>>> 13820666663704185619 12060305965056 >>>> [ 141.902795] BTRFS (device sdh): bad tree block start >>>> 13820666663704185619 12060305965056 >>>> [ 141.902870] BTRFS: failed to read chunk tree on sdh >>>> [ 141.915337] BTRFS: open_ctree failed >>>> [17748.031552] BTRFS info (device sdh): disk space caching is enabled >>>> [17748.031559] BTRFS: has skinny extents >>>> [17748.072339] BTRFS (device sdh): bad tree block start >>>> 13820666663704185619 12060305965056 >>>> [17748.077023] BTRFS (device sdh): bad tree block start >>>> 13820666663704185619 12060305965056 >>>> [17748.077350] BTRFS (device sdh): bad tree block start >>>> 13820666663704185619 12060305965056 >>>> [17748.077511] BTRFS (device sdh): bad tree block start >>>> 13820666663704185619 12060305965056 >>>> [17748.077587] BTRFS: failed to read chunk tree on sdh >>>> [17748.088908] BTRFS: open_ctree failed >>>> [17800.758291] BTRFS info (device sdh): disk space caching is enabled >>>> [17800.758298] BTRFS: has skinny extents >>>> [17800.765770] BTRFS (device sdh): bad tree block start >>>> 13820666663704185619 12060305965056 >>>> [17800.768816] BTRFS (device sdh): bad tree block start >>>> 13820666663704185619 12060305965056 >>>> [17800.769054] BTRFS (device sdh): bad tree block start >>>> 13820666663704185619 12060305965056 >>>> [17800.769192] BTRFS (device sdh): bad tree block start >>>> 13820666663704185619 12060305965056 >>>> [17800.769264] BTRFS: failed to read chunk tree on sdh >>>> [17800.784937] BTRFS: open_ctree failed >>>> [root@rockstor ~]# >>>> >>>> and then I get this: >>>> >>>> [root@rockstor ~]# btrfs-debug-tree -b 12060305965056 /dev/sdh >>>> checksum verify failed on 12060305965056 found 779CCA23 wanted A746C37A >>>> checksum verify failed on 12060305965056 found 779CCA23 wanted A746C37A >>>> checksum verify failed on 12060305965056 found 1727A198 wanted 231E1577 >>>> checksum verify failed on 12060305965056 found 1727A198 wanted 231E1577 >>>> bytenr mismatch, want=12060305965056, have=13820656527619066643 >>>> Couldn't read chunk tree >>>> unable to open /dev/sdh >>>> [root@rockstor ~]# >>>> >>>> Thanks, >>>> >>>> Scotty Edmonds >>>> sco...@scottyedmonds.com >>> >>> I think we need to see what some of the more experienced users think >>> on this one. But you can try removing sdh and seeing if you can mount >>> it *read only* and degraded. Just make sure whatever you do and play >>> with is done read only. Don't try any fixes or repairs with the tools >>> unless told to do so by someone who really knows what they're talking >>> about. >>> >>>> >>>> ________________________________________ >>>> From: Donald Pearson <donaldwhpear...@gmail.com> >>>> Sent: November-12-15 6:19 PM >>>> To: Scotty Edmonds; Btrfs BTRFS >>>> Subject: Re: BTRFS Error - Rockstor >>>> >>>> Anything interesting in dmesg? >>>> >>>> That looks similar to the kind of problems I had when I had a disk fail. >>>> >>>> On Thu, Nov 12, 2015 at 4:08 PM, Scotty Edmonds >>>> <sco...@scottyedmonds.com> wrote: >>>>> I get this: >>>>> >>>>> [root@rockstor ~]# btrfs check /dev/sdd >>>>> checksum verify failed on 12060305965056 found 779CCA23 wanted A746C37A >>>>> checksum verify failed on 12060305965056 found 779CCA23 wanted A746C37A >>>>> checksum verify failed on 12060305965056 found 1727A198 wanted 231E1577 >>>>> checksum verify failed on 12060305965056 found 1727A198 wanted 231E1577 >>>>> bytenr mismatch, want=12060305965056, have=13820656527619066643 >>>>> Couldn't read chunk tree >>>>> Couldn't open file system >>>>> [root@rockstor ~]# >>>>> >>>>> Thanks, >>>>> >>>>> Scotty Edmonds >>>>> sco...@scottyedmonds.com >>>>> >>>>> ________________________________________ >>>>> From: Donald Pearson <donaldwhpear...@gmail.com> >>>>> Sent: November-12-15 2:55 PM >>>>> To: Scotty Edmonds >>>>> Cc: linux-btrfs@vger.kernel.org >>>>> Subject: Re: BTRFS Error - Rockstor >>>>> >>>>> What does btrfs check without any repair options report? >>>>> >>>>> btrfs check /dev/sdd >>>>> >>>>> On Thu, Nov 12, 2015 at 12:48 PM, Scotty Edmonds >>>>> <sco...@scottyedmonds.com> wrote: >>>>>> Rockstor was running great, I ordered a SuperMicro 24-bay Chassis and >>>>>> decided to power down the machine while I was away. When I turned it >>>>>> back on I got "Failed to read chunk tree" & "open_ctree failed" error >>>>>> (http://i.imgur.com/rGk9M57l.jpg) >>>>>> >>>>>> I spoke with support at Rockstor and they recommended I seek help via >>>>>> the mailing list. Here are some details and commands I've run. The >>>>>> specific array is in RAID5 and the label is mainNAS, seagate3x2tb is >>>>>> running perfectly. >>>>>> >>>>>> [root@rockstor ~]# btrfs device scan >>>>>> Scanning for Btrfs filesystems >>>>>> [root@rockstor ~]# >>>>>> >>>>>> >>>>>> >>>>>> [root@rockstor ~]# /usr/bin/lsblk -P -o >>>>>> NAME,MODEL,SERIAL,SIZE,TRAN,VENDOR,HCTL,TYPE,FSTYPE,LABEL,UUID >>>>>> NAME="sda" MODEL="WDC WD30EFRX-68E" SERIAL="WD-WCC4N4KVC39Y" SIZE="2.7T" >>>>>> TRAN="sas" VENDOR="ATA " HCTL="0:0:0:0" TYPE="disk" FSTYPE="btrfs" >>>>>> LABEL="mainNAS" UUID="e8c92d93-fac3-4f83-b3aa-31cb92caafd9" >>>>>> NAME="sdb" MODEL="WDC WD30EZRX-00M" SERIAL="WD-WCAWZ2551761" SIZE="2.7T" >>>>>> TRAN="sas" VENDOR="ATA " HCTL="0:0:1:0" TYPE="disk" FSTYPE="btrfs" >>>>>> LABEL="mainNAS" UUID="e8c92d93-fac3-4f83-b3aa-31cb92caafd9" >>>>>> NAME="sdc" MODEL="HGST HDN724030AL" SERIAL="PK2234P9J590GY" SIZE="2.7T" >>>>>> TRAN="sas" VENDOR="ATA " HCTL="0:0:2:0" TYPE="disk" FSTYPE="btrfs" >>>>>> LABEL="mainNAS" UUID="e8c92d93-fac3-4f83-b3aa-31cb92caafd9" >>>>>> NAME="sdd" MODEL="HGST HDN724030AL" SERIAL="PK2234P9J5WA1Y" SIZE="2.7T" >>>>>> TRAN="sas" VENDOR="ATA " HCTL="0:0:3:0" TYPE="disk" FSTYPE="btrfs" >>>>>> LABEL="mainNAS" UUID="e8c92d93-fac3-4f83-b3aa-31cb92caafd9" >>>>>> NAME="sde" MODEL="ST3000DM001-1CH1" SERIAL="Z1F517PH" SIZE="2.7T" >>>>>> TRAN="sas" VENDOR="ATA " HCTL="0:0:4:0" TYPE="disk" FSTYPE="btrfs" >>>>>> LABEL="mainNAS" UUID="e8c92d93-fac3-4f83-b3aa-31cb92caafd9" >>>>>> NAME="sdf" MODEL="ST2000DL003-9VT1" SERIAL="5YD1WK0V" SIZE="1.8T" >>>>>> TRAN="sas" VENDOR="ATA " HCTL="0:0:5:0" TYPE="disk" FSTYPE="btrfs" >>>>>> LABEL="seagate3x2tb" UUID="6ef19043-2d83-4ff1-b959-b9f3c425cc69" >>>>>> NAME="sdg" MODEL="ST2000DL003-9VT1" SERIAL="5YD2EBDA" SIZE="1.8T" >>>>>> TRAN="sas" VENDOR="ATA " HCTL="0:0:6:0" TYPE="disk" FSTYPE="btrfs" >>>>>> LABEL="seagate3x2tb" UUID="6ef19043-2d83-4ff1-b959-b9f3c425cc69" >>>>>> NAME="sdh" MODEL="ST2000DL003-9VT1" SERIAL="5YD2L28Z" SIZE="1.8T" >>>>>> TRAN="sas" VENDOR="ATA " HCTL="0:0:7:0" TYPE="disk" FSTYPE="btrfs" >>>>>> LABEL="seagate3x2tb" UUID="6ef19043-2d83-4ff1-b959-b9f3c425cc69" >>>>>> NAME="sdi" MODEL="INTEL SSDSA2CW08" SERIAL="CVPR1330019Y080BGN" >>>>>> SIZE="74.5G" TRAN="sata" VENDOR="ATA " HCTL="1:0:0:0" TYPE="disk" >>>>>> FSTYPE="" LABEL="" UUID="" >>>>>> NAME="sdi1" MODEL="" SERIAL="" SIZE="500M" TRAN="" VENDOR="" HCTL="" >>>>>> TYPE="part" FSTYPE="ext4" LABEL="" >>>>>> UUID="53aabf2f-5e28-4a18-922f-b0767a77a8ec" >>>>>> NAME="sdi2" MODEL="" SERIAL="" SIZE="7.3G" TRAN="" VENDOR="" HCTL="" >>>>>> TYPE="part" FSTYPE="swap" LABEL="" >>>>>> UUID="bf9e72c7-7d72-4a33-a5eb-0a0013033234" >>>>>> NAME="sdi3" MODEL="" SERIAL="" SIZE="66.8G" TRAN="" VENDOR="" HCTL="" >>>>>> TYPE="part" FSTYPE="btrfs" LABEL="rockstor_rockstor" >>>>>> UUID="3533171e-d95b-4491-aa4c-cc956536a1c3" >>>>>> [root@rockstor ~]# >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> [root@rockstor ~]# btrfs fi show >>>>>> Label: 'rockstor_rockstor' uuid: 3533171e-d95b-4491-aa4c-cc956536a1c3 >>>>>> Total devices 1 FS bytes used 2.17GiB >>>>>> devid 1 size 66.79GiB used 7.02GiB path /dev/sdi3 >>>>>> >>>>>> Label: 'seagate3x2tb' uuid: 6ef19043-2d83-4ff1-b959-b9f3c425cc69 >>>>>> Total devices 3 FS bytes used 1.13TiB >>>>>> devid 1 size 1.82TiB used 595.03GiB path /dev/sdh >>>>>> devid 2 size 1.82TiB used 595.01GiB path /dev/sdf >>>>>> devid 3 size 1.82TiB used 595.01GiB path /dev/sdg >>>>>> >>>>>> Label: 'mainNAS' uuid: e8c92d93-fac3-4f83-b3aa-31cb92caafd9 >>>>>> Total devices 5 FS bytes used 5.43TiB >>>>>> devid 1 size 2.73TiB used 1.36TiB path /dev/sdd >>>>>> devid 2 size 2.73TiB used 1.36TiB path /dev/sdc >>>>>> devid 3 size 2.73TiB used 1.36TiB path /dev/sda >>>>>> devid 4 size 2.73TiB used 1.36TiB path /dev/sde >>>>>> devid 5 size 2.73TiB used 1.36TiB path /dev/sdb >>>>>> >>>>>> >>>>>> btrfs-progs v4.2.1 >>>>>> >>>>>> I'm unable to mount any of the drives that are in the mainNAS array, >>>>>> this is the error when I try to mount all of the drives degraded. >>>>>> >>>>>> [root@rockstor ~]# mount -v -o degraded /dev/sdd /mnt2/mainNAS >>>>>> mount: wrong fs type, bad option, bad superblock on /dev/sdd, >>>>>> missing codepage or helper program, or other error >>>>>> In some cases useful info is found in syslog - try >>>>>> dmesg | tail or so. >>>>>> [root@rockstor ~]# >>>>>> >>>>>> I haven't given up hope yet as the "btrfs fi show" gives me all the >>>>>> correct data and I ran chunk-recover and superblocks all report back as >>>>>> good. >>>>>> >>>>>> >>>>>> Thanks for your help, let me know if you need any further information. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Scotty Edmonds >>>>>> sco...@scottyedmonds.com >> >> -- >> Hugo Mills | "How deep will this sub go?" >> hugo@... carfax.org.uk | "Oh, she'll go all the way to the bottom if we don't >> http://carfax.org.uk/ | stop her." >> PGP: E2AB1DE4 | >> U571 >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html