Hello, I'm testing Btrfs RAID1 feature on 3 disks of ~10GB. Last one is not exactly 10GB (would be too easy).
About the test machine, it's a kvm vm running an up-to-date archlinux with linux 3.7 and btrfs-progs 0.19.20121005. #uname -a Linux seblu-btrfs-1 3.7.0-1-ARCH #1 SMP PREEMPT Tue Dec 11 15:05:50 CET 2012 x86_64 GNU/Linux Filesystem was created with : # mkfs.btrfs -L test -d raid1 -m raid1 /dev/vd[bcd] I downloaded a lot of linux kernel tarball and untared in into this filesystem until it tell me enough: drwxr-xr-x 1 root root 330 2007-10-09 20:31 linux-2.6.23 -rw-r--r-- 1 root root 44M 2007-10-09 20:48 linux-2.6.23.tar.bz2 drwxr-xr-x 1 root root 344 2008-01-24 22:58 linux-2.6.24 -rw-r--r-- 1 root root 45M 2008-01-24 23:16 linux-2.6.24.tar.bz2 drwxr-xr-x 1 root root 352 2008-04-17 02:49 linux-2.6.25 Some output of btrfs tools # btrfs fi sh Label: 'test' uuid: 7d72c625-4dd7-4db0-b4a2-075e26572b99 Total devices 3 FS bytes used 11.57GB devid 3 size 10.00GB used 9.76GB path /dev/vdd devid 2 size 10.00GB used 10.00GB path /dev/vdc devid 1 size 9.77GB used 9.77GB path /dev/vdb # btrfs fi df . Data, RAID1: total=13.50GB, used=10.56GB Data: total=8.00MB, used=0.00 System, RAID1: total=8.00MB, used=4.00KB System: total=4.00MB, used=0.00 Metadata, RAID1: total=1.25GB, used=1.01GB Metadata: total=8.00MB, used=0.00 I have choosen a _raid1_ filesystem, so I expect every disk be able to die and fs still usuable without data lost. I used chksfv[1] to make checksum of tarball on disk and check my data are safe. I killed the first disk via libvirt. # virsh detach-disk seblu-btrfs-1 vdb Btrfs detect missing disk \o/ # btrfs fi sh Label: 'test' uuid: 7d72c625-4dd7-4db0-b4a2-075e26572b99 Total devices 3 FS bytes used 11.57GB devid 3 size 10.00GB used 8.01GB path /dev/vdd devid 2 size 10.00GB used 9.26GB path /dev/vdc *** Some devices missing Listing all files and cksum on tarballs are good. # cksfv -f ~/checksum Everything OK We have raid1, so we can expect kill one disk (the smaller). Let's go. # virsh detach-disk seblu-btrfs-1 vdd # btrfs fi sh Label: 'test' uuid: 7d72c625-4dd7-4db0-b4a2-075e26572b99 Total devices 3 FS bytes used 11.57GB devid 3 size 10.00GB used 8.01GB path /dev/vdd devid 2 size 10.00GB used 9.26GB path /dev/vdc *** Some devices missing It doesn't see the drive go away... Listing doesn't works. # find /mnt/test . ./pkg ^C # cksfv -f ~/checksum linux-2.6.30.tar.bz2 OK linux-2.6.31.tar.bz2 Input/output error linux-2.6.32.tar.bz2 Input/output error linux-2.6.33.tar.bz2 Input/output error So data and metada seems broken. Filesystem is become read only # touch séb touch: cannot touch ‘séb’: Read-only file system and finally I got a traceback in dmesg [ 804.910405] ------------[ cut here ]------------ [ 804.910491] WARNING: at fs/btrfs/super.c:246 __btrfs_abort_transaction+0xdf/0x100 [btrfs]() [ 804.910499] Hardware name: Bochs [ 804.910501] btrfs: Transaction aborted [ 804.910502] Modules linked in: cirrus ttm i2c_piix4 drm_kms_helper psmouse intel_agp drm evdev intel_gtt serio_raw syscopyarea sysfillrect sysimgblt i2c_core processor button pcspkr microcode btrfs crc32c libcrc32c zlib_deflate ata_generic pata_acpi virtio_balloon virtio_blk virtio_net ata_piix libata uhci_hcd virtio_pci virtio_ring virtio usbcore usb_common scsi_mod floppy [ 804.910575] Pid: 2077, comm: touch Not tainted 3.7.0-1-ARCH #1 [ 804.910577] Call Trace: [ 804.910606] [<ffffffff8105742f>] warn_slowpath_common+0x7f/0xc0 [ 804.910610] [<ffffffff81057526>] warn_slowpath_fmt+0x46/0x50 [ 804.910619] [<ffffffffa015ff6f>] __btrfs_abort_transaction+0xdf/0x100 [btrfs] [ 804.910635] [<ffffffffa01802c0>] ? verify_parent_transid+0x170/0x170 [btrfs] [ 804.910648] [<ffffffffa016402a>] __btrfs_cow_block+0x48a/0x510 [btrfs] [ 804.910661] [<ffffffffa0164227>] btrfs_cow_block+0xf7/0x230 [btrfs] [ 804.910672] [<ffffffffa016752b>] push_leaf_right+0x11b/0x190 [btrfs] [ 804.910681] [<ffffffffa0167cb1>] split_leaf+0x621/0x740 [btrfs] [ 804.910690] [<ffffffffa0160d52>] ? leaf_space_used+0xd2/0x110 [btrfs] [ 804.910710] [<ffffffffa01b9455>] ? btrfs_set_lock_blocking_rw+0xb5/0x120 [btrfs] [ 804.910720] [<ffffffffa016894c>] btrfs_search_slot+0x89c/0x900 [btrfs] [ 804.910730] [<ffffffffa0169fcc>] btrfs_insert_empty_items+0x7c/0xe0 [btrfs] [ 804.910742] [<ffffffffa017c313>] insert_with_overflow+0x43/0x120 [btrfs] [ 804.910753] [<ffffffffa017c4ac>] btrfs_insert_dir_item+0xbc/0x200 [btrfs] [ 804.910767] [<ffffffffa0195c6b>] btrfs_add_link+0xeb/0x300 [btrfs] [ 804.910781] [<ffffffffa0196f64>] btrfs_create+0x1a4/0x210 [btrfs] [ 804.910799] [<ffffffff81217a0c>] ? security_inode_permission+0x1c/0x30 [ 804.910813] [<ffffffff8118fd56>] vfs_create+0xb6/0x120 [ 804.910833] [<ffffffff8119281d>] do_last+0xd5d/0xeb0 [ 804.910838] [<ffffffff8118ed68>] ? link_path_walk+0x78/0x950 [ 804.910841] [<ffffffff81192a2c>] path_openat+0xbc/0x530 [ 804.910855] [<ffffffff8114b551>] ? handle_mm_fault+0x261/0x330 [ 804.910871] [<ffffffff814a6314>] ? __do_page_fault+0x2d4/0x590 [ 804.910875] [<ffffffff81193211>] do_filp_open+0x41/0xa0 [ 804.910878] [<ffffffff811a0119>] ? __alloc_fd+0xe9/0x140 [ 804.910887] [<ffffffff81182fc4>] do_sys_open+0xf4/0x1e0 [ 804.910890] [<ffffffff814a65de>] ? do_page_fault+0xe/0x10 [ 804.910895] [<ffffffff811830d1>] sys_open+0x21/0x30 [ 804.910899] [<ffffffff814aa51d>] system_call_fastpath+0x1a/0x1f [ 804.910904] ---[ end trace b713d23404f7cc1c ]--- [ 804.910911] BTRFS error (device vdc) in __btrfs_cow_block:1014: IO failure [ 804.912156] btrfs is forced readonly I get the feeling that RAID1 only allow one disk removing. Which is more a RAID5 feature. I'm afraid Btrfs raid1 will not be working before the end of the world. Sad day. Cheers, [1] https://www.archlinux.org/packages/community/i686/cksfv/ -- Sebastien "Seblu" Luttringer Research & Engineering Manager - Smartjog - TDF 27 bd Hippolyte Marques, 94200 Ivry sur Seine, France Phone: +33.1.58.68.62.32 / Fax: +33.1.58.68.62.05 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html