On 2015-07-21 04:38, Qu Wenruo wrote:
Yes, it's a distributed SAN type system built on top of Ceph. It does allow having multiple systems mount the device.Hi Steve,I checked your binary dump. Previously I was too focused on the assert error, but ignored some even larger bug... As for the btrfs-debug-tree output, subvol 257 and 5 are completely corrupted. Subvol 257 seems to contains a new tree root, and 5 seems to contains a new device tree. ------ fs tree key (FS_TREE ROOT_ITEM 0) leaf 29409280 items 8 free space 15707 generation 9 owner 4 fs uuid 1bb22a03-bc25-466f-b078-c66c6f6a6d28 chunk uuid 11cca6df-e850-45d7-a928-cdff82c5f295 item 0 key (0 DEV_STATS 1) itemoff 16243 itemsize 40 device stats item 1 key (1 DEV_EXTENT 0) itemoff 16195 itemsize 48 dev extent chunk_tree 3 chunk objectid 256 chunk offset 0 length 4194304 item 2 key (1 DEV_EXTENT 4194304) itemoff 16147 itemsize 48 dev extent chunk_tree 3 chunk objectid 256 chunk offset 4194304 length 8388608 item 3 key (1 DEV_EXTENT 12582912) itemoff 16099 itemsize 48 dev extent chunk_tree 3 ...... # DEV_EXTENT should never occur in fs tree. It should only occurs in # dev tree file tree key (257 ROOT_ITEM 0) leaf 29376512 items 13 free space 12844 generation 9 owner 1 fs uuid 1bb22a03-bc25-466f-b078-c66c6f6a6d28 chunk uuid 11cca6df-e850-45d7-a928-cdff82c5f295 item 0 key (EXTENT_TREE ROOT_ITEM 0) itemoff 15844 itemsize 439 root data bytenr 29392896 level 0 dirid 0 refs 1 gen 9 uuid 00000000-0000-0000-0000-000000000000 item 1 key (DEV_TREE ROOT_ITEM 0) itemoff 15405 itemsize 439 root data bytenr 29409280 level 0 dirid 0 refs 1 gen 9 uuid 00000000-0000-0000-0000-000000000000 item 2 key (FS_TREE INODE_REF 6) itemoff 15388 itemsize 17 inode ref index 0 namelen 7 name: default item 3 key (FS_TREE ROOT_ITEM 0) itemoff 14949 itemsize 439 root data bytenr 29360128 level 0 dirid 256 refs 1 gen 4 uuid 00000000-0000-0000-0000-000000000000 item 4 key (ROOT_TREE_DIR INODE_ITEM 0) itemoff 14789 itemsize 160 inode generation 3 transid 0 size 0 nbytes 16384 block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0 flags 0x0 # These things are only in tree root. ------ So the problem is, the kernel you use has some bug (btrfs or rbd related), causing the btrfs write wrong tree blocks into existing tree blocks. For such case, btrfsck won't be able to fix the critical error. And I didn't even have an idea to fix the assert to change it into a normal error. As it's corrupting the whole structure of btrfs... I can't even recall such critical btrfs bug... Not familiar with rbd, but will it allow a block device to be mounted on different systems? Like exporting a device A to system B and system C, and both system B and system C mounting device A at the same time as btrfs?
Ideally, we really should put in some kind of protection against multiple mounts (this would be a significant selling point of BTRFS in my opinion, as the only other Linux native FS that has this is ext4), and make it _very_ obvious that mounting a BTRFS filesystem on multiple nodes concurrently _WILL_ result in pretty much irreparable corruption.
smime.p7s
Description: S/MIME Cryptographic Signature
