On 08/28/2012 09:52 PM, M G Berberich wrote:
Hello,
We had set up a btrfs-fs over 6 hot-plugable SAS-disks for
testing and got it into a state where kernel and btrfs-tools do not
agree any more about the state of the filesystem.
We do not remember exaclty what we did, but roughly it was something
like this (on the running system). THIS IS FROM MEMORY!
(1) pulled out one disk
(2) removed disk from btrfs
(3) rebalanced btrfs
(4) pulled out another disk
(5) removed disk from btrfs
(6) rebalanced btrfs
This went fine sofar.
(7) reinserted disk (and rebooted)
At some point before reboot the first 10 sectors of one disk
were zeroed to test if the disk gets removed from the btrfs.
IIRC the superblock is not placed at the beginning of the disk. On the
basis of [1] it should be near the 64KB (around the sector #128)
[1]
https://btrfs.wiki.kernel.org/index.php/User:Wtachi/On-disk_Format#Superblock
Now btrfs-tools showed:
---------------------------------------------------------------------------
# btrfs fi show
failed to read /dev/sr0
Label: 'BTRFS_RAID' uuid: 807193fd-17de-4088-9a54-3e7cacdc89db
Total devices 6 FS bytes used 3.07GB
devid 4 size 931.00GB used 75.00GB path /dev/sdf
devid 5 size 931.00GB used 324.03GB path /dev/sde
devid 6 size 931.00GB used 83.03GB path /dev/sdd
devid 3 size 931.00GB used 326.03GB path /dev/sdc
devid 2 size 931.00GB used 326.03GB path /dev/sdb
devid 1 size 931.00GB used 324.04GB path /dev/sda
"btrfs filesystem show" shows the content of the disks, which could be
unrelated to the kernel status. Pay attention that if the data is not
flushed to the disk the report of "btrfs fi show" could be unreliable.
I posted few days ago a patch which adds the sysfs support to btrfs.
With this support it is possible to know the real state of the disks.
For example I have a filesystem with 4 disks (note "Total devices 4"):
ghigo@emulato:~$ sudo btrfs fi show
Label: 'btrfs3' uuid: 2a66286d-63e9-4ed5-b347-5af5e4ada814
Total devices 4 FS bytes used 284.00KB
devid 4 size 100.00GB used 8.01GB path /dev/vdj
devid 3 size 100.00GB used 6.04GB path /dev/vdi
devid 5 size 100.00GB used 0.00 path /dev/vdh
devid 1 size 100.00GB used 7.05GB path /dev/vdg
Btrfs Btrfs v0.19
My sysfs interface says that the filesystem is composed by 4 disks:
ghigo@emulato:~$ cat /sys/fs/btrfs/filesystems/2a66286d-
63e9-4ed5b347-5af5e4ada814/fs_devices/open_devices
4
Then I remove 1 disk
ghigo@emulato:~$ sudo btrfs dev del /dev/vdi /mnt/btrfs3/
Now the sysfs interface says:
ghigo@emulato:~$ cat /sys/fs/btrfs/filesystems/2a66286d-
63e9-4ed5b347-5af5e4ada814/fs_devices/open_devices
3
But "btrfs filesystem show" says (note still "Total devices 4"):
ghigo@emulato:~$ sudo btrfs fi show
failed to read /dev/sr0
Label: 'btrfs3' uuid: 2a66286d-63e9-4ed5-b347-5af5e4ada814
Total devices 4 FS bytes used 92.00KB
devid 4 size 100.00GB used 7.00GB path /dev/vdj
devid 3 size 100.00GB used 6.04GB path /dev/vdi
devid 5 size 100.00GB used 5.06GB path /dev/vdh
devid 1 size 100.00GB used 6.08GB path /dev/vdg
Btrfs Btrfs v0.19
Then I do a sync
ghigo@emulato:~$ sync
ghigo@emulato:~$ sudo btrfs fi show
failed to read /dev/sr0
Label: 'btrfs3' uuid: 2a66286d-63e9-4ed5-b347-5af5e4ada814
Total devices 3 FS bytes used 92.00KB
devid 4 size 100.00GB used 7.00GB path /dev/vdj
devid 3 size 100.00GB used 6.04GB path /dev/vdi
devid 5 size 100.00GB used 5.06GB path /dev/vdh
devid 1 size 100.00GB used 6.08GB path /dev/vdg
Btrfs Btrfs v0.19
(note "Total devices 3")
And magically the filesystem is now composed by three disks. However 4
physical devices are show. This because the disk /dev/vdi superblock
says that the disk is still valid (after the "btrfs device del" the disk
is not touched any more)
In the past Hubert posted a patch [2] to clear a btrfs superblock. A
further enhancement of the "btrfs device del" could be to reset
automatically the first superblock (leaving the backup ones unaffected).
[2] http://permalink.gmane.org/gmane.comp.file-systems.btrfs/17065
Btrfs Btrfs v0.19
---------------------------------------------------------------------------
As far as we can tell, only four of the disks are considered part of
the btrfs by kernel. There were only four “btrfs: bdev”-lines in dmesg
and only four disks took part in balancing. “btrfs device scan” says:
unable to scan the device '/dev/sdd' - Device or resource busy
and balance does not balance theses two devices (of 6)
It was neither possible to remove the disk from the btrfs via “btrfs
device delete” nor adding them via “btrfs device add”.
(8) a colleague swaped the two disk
Now btrfs-tools showed:
---------------------------------------------------------------------------
# btrfs fi show
failed to read /dev/sr0
Label: 'BTRFS_RAID' uuid: 807193fd-17de-4088-9a54-3e7cacdc89db
Total devices 5 FS bytes used 3.01GB
devid 6 size 931.00GB used 83.03GB path /dev/sdf
devid 4 size 931.00GB used 75.00GB path /dev/sdd
devid 5 size 931.00GB used 325.03GB path /dev/sde
devid 3 size 931.00GB used 326.03GB path /dev/sdc
devid 2 size 931.00GB used 325.03GB path /dev/sdb
devid 1 size 931.51GB used 326.04GB path /dev/sda
Btrfs Btrfs v0.19
---------------------------------------------------------------------------
Claiming the btrfs has 5 disk, but listing 6 disks out of 5 (6 of 5).
He finally managed to get the btrfs complete again by overwriting the
first 100G of the two disk. After this the btrfs-tools (correctly)
stated a filesystem with 4 disk and it was possible to add the two
disk again.
Assumption:
kernel and btrfs do not share the same view of the filesystem.
In this state commands to repair the filesystem do not work, because
they are either rejected by the tools or by the kernel.
A tool that allows a disk/partition to be marked as not-a-btrfs-part
would be nice.
A “/proc/btrfs” showing the kernels view of the filesystem would be
usefull.
MfG
bmg
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
