So, after experiencing this same issue multiple times (on almost a dozen different kernel versions since 4.0) and ruling out the possibility of it being caused by my hardware (or at least, the RAM, SATA controller and disk drives themselves), I've decided to report it here.
The general symptom is that raid6 profile filesystems that I have are working
fine for multiple weeks, until I either reboot or otherwise try to remount
them, at which point the system refuses to mount them.
I'm currently using btrfs-progs v4.1 with kernel 4.1.2, although I've been
seeing this with versions of both since 4.0.
Output of 'btrfs fi show' for the most recent fs that I had this issue with:
Label: 'altroot' uuid: 86eef6b9-febe-4350-a316-4cb00c40bbc5
Total devices 4 FS bytes used 9.70GiB
devid 1 size 24.00GiB used 6.03GiB path /dev/mapper/vg-altroot.0
devid 2 size 24.00GiB used 6.01GiB path /dev/mapper/vg-altroot.1
devid 3 size 24.00GiB used 6.01GiB path /dev/mapper/vg-altroot.2
devid 4 size 24.00GiB used 6.01GiB path /dev/mapper/vg-altroot.3
btrfs-progs v4.1
Each of the individual LVS that are in the FS is just a flat chunk of space on
a separate disk from the others.
The FS itself passes btrfs check just fine (no reported errors, exit value of
0), but the kernel refuses to mount it with the message 'open_ctree failed'.
I've run btrfs chunk recover and attached the output from that.
Here's a link to an image from 'btrfs image -c9 -w':
https://www.dropbox.com/s/pl7gs305ej65u9q/altroot.btrfs.img?dl=0
(That link will expire in 30 days, let me know if you need access to it beyond
that).
The filesystems in question all see relatively light but consistent usage as
targets for receiving daily incremental snapshots for on-system backups (and
because I know someone will mention it, yes, I do have other backups of the
data, these are just my online backups).
All Devices:
Device: id = 4, name = /dev/mapper/vg-altroot.3
Device: id = 3, name = /dev/mapper/vg-altroot.2
Device: id = 2, name = /dev/mapper/vg-altroot.1
Device: id = 1, name = /dev/vg/altroot.0
DEVICE SCAN RESULT:
Filesystem Information:
sectorsize: 4096
leafsize: 16384
tree root generation: 26
chunk root generation: 11
All Devices:
Device: id = 4, name = /dev/mapper/vg-altroot.3
Device: id = 3, name = /dev/mapper/vg-altroot.2
Device: id = 2, name = /dev/mapper/vg-altroot.1
Device: id = 1, name = /dev/vg/altroot.0
All Block Groups:
Block Group: start = 0, len = 4194304, flag = 2
Block Group: start = 4194304, len = 8388608, flag = 4
Block Group: start = 12582912, len = 8388608, flag = 1
Block Group: start = 20971520, len = 16777216, flag = 102
Block Group: start = 37748736, len = 2147483648, flag = 104
Block Group: start = 2185232384, len = 2147483648, flag = 101
Block Group: start = 4332716032, len = 2147483648, flag = 101
Block Group: start = 6480199680, len = 2147483648, flag = 101
Block Group: start = 8627683328, len = 2147483648, flag = 101
Block Group: start = 10775166976, len = 2147483648, flag = 101
All Chunks:
Chunk: start = 0, len = 4194304, type = 2, num_stripes = 1
Stripes list:
[ 0] Stripe: devid = 1, offset = 0
Chunk: start = 4194304, len = 8388608, type = 4, num_stripes = 1
Stripes list:
[ 0] Stripe: devid = 1, offset = 4194304
Chunk: start = 12582912, len = 8388608, type = 1, num_stripes = 1
Stripes list:
[ 0] Stripe: devid = 1, offset = 12582912
Chunk: start = 20971520, len = 16777216, type = 102, num_stripes = 4
Stripes list:
[ 0] Stripe: devid = 4, offset = 1048576
[ 1] Stripe: devid = 3, offset = 1048576
[ 2] Stripe: devid = 2, offset = 1048576
[ 3] Stripe: devid = 1, offset = 20971520
Chunk: start = 37748736, len = 2147483648, type = 104, num_stripes = 4
Stripes list:
[ 0] Stripe: devid = 4, offset = 9437184
[ 1] Stripe: devid = 3, offset = 9437184
[ 2] Stripe: devid = 2, offset = 9437184
[ 3] Stripe: devid = 1, offset = 29360128
Chunk: start = 2185232384, len = 2147483648, type = 101, num_stripes =
4
Stripes list:
[ 0] Stripe: devid = 4, offset = 1083179008
[ 1] Stripe: devid = 3, offset = 1083179008
[ 2] Stripe: devid = 2, offset = 1083179008
[ 3] Stripe: devid = 1, offset = 1103101952
Chunk: start = 4332716032, len = 2147483648, type = 101, num_stripes =
4
Stripes list:
[ 0] Stripe: devid = 2, offset = 2156920832
[ 1] Stripe: devid = 3, offset = 2156920832
[ 2] Stripe: devid = 4, offset = 2156920832
[ 3] Stripe: devid = 1, offset = 2176843776
Chunk: start = 6480199680, len = 2147483648, type = 101, num_stripes =
4
Stripes list:
[ 0] Stripe: devid = 2, offset = 3230662656
[ 1] Stripe: devid = 3, offset = 3230662656
[ 2] Stripe: devid = 4, offset = 3230662656
[ 3] Stripe: devid = 1, offset = 3250585600
Chunk: start = 8627683328, len = 2147483648, type = 101, num_stripes =
4
Stripes list:
[ 0] Stripe: devid = 2, offset = 4304404480
[ 1] Stripe: devid = 3, offset = 4304404480
[ 2] Stripe: devid = 4, offset = 4304404480
[ 3] Stripe: devid = 1, offset = 4324327424
Chunk: start = 10775166976, len = 2147483648, type = 101, num_stripes
= 4
Stripes list:
[ 0] Stripe: devid = 2, offset = 5378146304
[ 1] Stripe: devid = 3, offset = 5378146304
[ 2] Stripe: devid = 4, offset = 5378146304
[ 3] Stripe: devid = 1, offset = 5398069248
All Device Extents:
Device extent: devid = 1, start = 0, len = 4194304, chunk offset = 0
Device extent: devid = 1, start = 4194304, len = 8388608, chunk offset
= 4194304
Device extent: devid = 1, start = 12582912, len = 8388608, chunk
offset = 12582912
Device extent: devid = 1, start = 20971520, len = 8388608, chunk
offset = 20971520
Device extent: devid = 1, start = 29360128, len = 1073741824, chunk
offset = 37748736
Device extent: devid = 1, start = 1103101952, len = 1073741824, chunk
offset = 2185232384
Device extent: devid = 1, start = 2176843776, len = 1073741824, chunk
offset = 4332716032
Device extent: devid = 1, start = 3250585600, len = 1073741824, chunk
offset = 6480199680
Device extent: devid = 1, start = 4324327424, len = 1073741824, chunk
offset = 8627683328
Device extent: devid = 1, start = 5398069248, len = 1073741824, chunk
offset = 10775166976
Device extent: devid = 2, start = 1048576, len = 8388608, chunk offset
= 20971520
Device extent: devid = 2, start = 9437184, len = 1073741824, chunk
offset = 37748736
Device extent: devid = 2, start = 1083179008, len = 1073741824, chunk
offset = 2185232384
Device extent: devid = 2, start = 2156920832, len = 1073741824, chunk
offset = 4332716032
Device extent: devid = 2, start = 3230662656, len = 1073741824, chunk
offset = 6480199680
Device extent: devid = 2, start = 4304404480, len = 1073741824, chunk
offset = 8627683328
Device extent: devid = 2, start = 5378146304, len = 1073741824, chunk
offset = 10775166976
Device extent: devid = 3, start = 1048576, len = 8388608, chunk offset
= 20971520
Device extent: devid = 3, start = 9437184, len = 1073741824, chunk
offset = 37748736
Device extent: devid = 3, start = 1083179008, len = 1073741824, chunk
offset = 2185232384
Device extent: devid = 3, start = 2156920832, len = 1073741824, chunk
offset = 4332716032
Device extent: devid = 3, start = 3230662656, len = 1073741824, chunk
offset = 6480199680
Device extent: devid = 3, start = 4304404480, len = 1073741824, chunk
offset = 8627683328
Device extent: devid = 3, start = 5378146304, len = 1073741824, chunk
offset = 10775166976
Device extent: devid = 4, start = 1048576, len = 8388608, chunk offset
= 20971520
Device extent: devid = 4, start = 9437184, len = 1073741824, chunk
offset = 37748736
Device extent: devid = 4, start = 1083179008, len = 1073741824, chunk
offset = 2185232384
Device extent: devid = 4, start = 2156920832, len = 1073741824, chunk
offset = 4332716032
Device extent: devid = 4, start = 3230662656, len = 1073741824, chunk
offset = 6480199680
Device extent: devid = 4, start = 4304404480, len = 1073741824, chunk
offset = 8627683328
Device extent: devid = 4, start = 5378146304, len = 1073741824, chunk
offset = 10775166976
CHECK RESULT:
Recoverable Chunks:
Chunk: start = 0, len = 4194304, type = 2, num_stripes = 1
Stripes list:
[ 0] Stripe: devid = 1, offset = 0
Block Group: start = 0, len = 4194304, flag = 2
Device extent list:
[ 0]Device extent: devid = 1, start = 0, len = 4194304, chunk offset
= 0
Chunk: start = 4194304, len = 8388608, type = 4, num_stripes = 1
Stripes list:
[ 0] Stripe: devid = 1, offset = 4194304
Block Group: start = 4194304, len = 8388608, flag = 4
Device extent list:
[ 0]Device extent: devid = 1, start = 4194304, len = 8388608, chunk
offset = 4194304
Chunk: start = 12582912, len = 8388608, type = 1, num_stripes = 1
Stripes list:
[ 0] Stripe: devid = 1, offset = 12582912
Block Group: start = 12582912, len = 8388608, flag = 1
Device extent list:
[ 0]Device extent: devid = 1, start = 12582912, len = 8388608, chunk
offset = 12582912
Chunk: start = 20971520, len = 16777216, type = 102, num_stripes = 4
Stripes list:
[ 0] Stripe: devid = 4, offset = 1048576
[ 1] Stripe: devid = 3, offset = 1048576
[ 2] Stripe: devid = 2, offset = 1048576
[ 3] Stripe: devid = 1, offset = 20971520
Block Group: start = 20971520, len = 16777216, flag = 102
Device extent list:
[ 0]Device extent: devid = 1, start = 20971520, len = 8388608, chunk
offset = 20971520
[ 1]Device extent: devid = 2, start = 1048576, len = 8388608, chunk
offset = 20971520
[ 2]Device extent: devid = 3, start = 1048576, len = 8388608, chunk
offset = 20971520
[ 3]Device extent: devid = 4, start = 1048576, len = 8388608, chunk
offset = 20971520
Chunk: start = 37748736, len = 2147483648, type = 104, num_stripes = 4
Stripes list:
[ 0] Stripe: devid = 4, offset = 9437184
[ 1] Stripe: devid = 3, offset = 9437184
[ 2] Stripe: devid = 2, offset = 9437184
[ 3] Stripe: devid = 1, offset = 29360128
Block Group: start = 37748736, len = 2147483648, flag = 104
Device extent list:
[ 0]Device extent: devid = 1, start = 29360128, len = 1073741824,
chunk offset = 37748736
[ 1]Device extent: devid = 2, start = 9437184, len = 1073741824,
chunk offset = 37748736
[ 2]Device extent: devid = 3, start = 9437184, len = 1073741824,
chunk offset = 37748736
[ 3]Device extent: devid = 4, start = 9437184, len = 1073741824,
chunk offset = 37748736
Chunk: start = 2185232384, len = 2147483648, type = 101, num_stripes = 4
Stripes list:
[ 0] Stripe: devid = 4, offset = 1083179008
[ 1] Stripe: devid = 3, offset = 1083179008
[ 2] Stripe: devid = 2, offset = 1083179008
[ 3] Stripe: devid = 1, offset = 1103101952
Block Group: start = 2185232384, len = 2147483648, flag = 101
Device extent list:
[ 0]Device extent: devid = 1, start = 1103101952, len = 1073741824,
chunk offset = 2185232384
[ 1]Device extent: devid = 2, start = 1083179008, len = 1073741824,
chunk offset = 2185232384
[ 2]Device extent: devid = 3, start = 1083179008, len = 1073741824,
chunk offset = 2185232384
[ 3]Device extent: devid = 4, start = 1083179008, len = 1073741824,
chunk offset = 2185232384
Chunk: start = 4332716032, len = 2147483648, type = 101, num_stripes = 4
Stripes list:
[ 0] Stripe: devid = 2, offset = 2156920832
[ 1] Stripe: devid = 3, offset = 2156920832
[ 2] Stripe: devid = 4, offset = 2156920832
[ 3] Stripe: devid = 1, offset = 2176843776
Block Group: start = 4332716032, len = 2147483648, flag = 101
Device extent list:
[ 0]Device extent: devid = 1, start = 2176843776, len = 1073741824,
chunk offset = 4332716032
[ 1]Device extent: devid = 4, start = 2156920832, len = 1073741824,
chunk offset = 4332716032
[ 2]Device extent: devid = 3, start = 2156920832, len = 1073741824,
chunk offset = 4332716032
[ 3]Device extent: devid = 2, start = 2156920832, len = 1073741824,
chunk offset = 4332716032
Chunk: start = 6480199680, len = 2147483648, type = 101, num_stripes = 4
Stripes list:
[ 0] Stripe: devid = 2, offset = 3230662656
[ 1] Stripe: devid = 3, offset = 3230662656
[ 2] Stripe: devid = 4, offset = 3230662656
[ 3] Stripe: devid = 1, offset = 3250585600
Block Group: start = 6480199680, len = 2147483648, flag = 101
Device extent list:
[ 0]Device extent: devid = 1, start = 3250585600, len = 1073741824,
chunk offset = 6480199680
[ 1]Device extent: devid = 4, start = 3230662656, len = 1073741824,
chunk offset = 6480199680
[ 2]Device extent: devid = 3, start = 3230662656, len = 1073741824,
chunk offset = 6480199680
[ 3]Device extent: devid = 2, start = 3230662656, len = 1073741824,
chunk offset = 6480199680
Chunk: start = 8627683328, len = 2147483648, type = 101, num_stripes = 4
Stripes list:
[ 0] Stripe: devid = 2, offset = 4304404480
[ 1] Stripe: devid = 3, offset = 4304404480
[ 2] Stripe: devid = 4, offset = 4304404480
[ 3] Stripe: devid = 1, offset = 4324327424
Block Group: start = 8627683328, len = 2147483648, flag = 101
Device extent list:
[ 0]Device extent: devid = 1, start = 4324327424, len = 1073741824,
chunk offset = 8627683328
[ 1]Device extent: devid = 4, start = 4304404480, len = 1073741824,
chunk offset = 8627683328
[ 2]Device extent: devid = 3, start = 4304404480, len = 1073741824,
chunk offset = 8627683328
[ 3]Device extent: devid = 2, start = 4304404480, len = 1073741824,
chunk offset = 8627683328
Chunk: start = 10775166976, len = 2147483648, type = 101, num_stripes = 4
Stripes list:
[ 0] Stripe: devid = 2, offset = 5378146304
[ 1] Stripe: devid = 3, offset = 5378146304
[ 2] Stripe: devid = 4, offset = 5378146304
[ 3] Stripe: devid = 1, offset = 5398069248
Block Group: start = 10775166976, len = 2147483648, flag = 101
Device extent list:
[ 0]Device extent: devid = 1, start = 5398069248, len = 1073741824,
chunk offset = 10775166976
[ 1]Device extent: devid = 4, start = 5378146304, len = 1073741824,
chunk offset = 10775166976
[ 2]Device extent: devid = 3, start = 5378146304, len = 1073741824,
chunk offset = 10775166976
[ 3]Device extent: devid = 2, start = 5378146304, len = 1073741824,
chunk offset = 10775166976
Unrecoverable Chunks:
Total Chunks: 10
Recoverable: 10
Unrecoverable: 0
Orphan Block Groups:
Orphan Device Extents:
Check chunks successfully with no orphans
Recover the chunk tree successfully.
smime.p7s
Description: S/MIME Cryptographic Signature
