Re: what is the best way to monitor raid1 drive failures?
On Oct 14, 2014, at 2:11 PM, Suman C schakr...@gmail.com wrote: After the reboot step, where I indicated that I mounted ro, I was unable to mount rw or rw,degraded. I get the mount: wrong fs type, bad option, bad superblock…. error if I try to mount it rw. I would test with kernel 3.17 at least, but better to use 3.18rc1 now. If you can reproduce it, then I'd file a bug. I'm a bit haphazard in my bug filing. It seems more effective to start a new subject with the word bug in brackets, and subject something like can't mount rw,degraded and then go through the reproduce steps being clear how the fs was created and all that. Also note not only what does happen, but what you expect to happen instead. Seems obvious but it should be done that way. I'd also file it as a bug the way we're asked to on the wiki which is bugzilla.kernel.org. Chris Murphy-- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: what is the best way to monitor raid1 drive failures?
Hi, Here's a simple raid1 recovery experiment that's not working as expected. kernel: 3.17, latest mainline progs: 3.16.1 I started with a simple raid1 mirror of 2 drives (sda and sdb). The filesystem is functional, I created one subvol, put some data, read/write tested etc.. yanked the sdb out. (this is physical/hardware). btrfs fi show prints drive missing, as expected. powered the machine down. removed the bad(yanked out sdb) drive and replaced it with a new drive. Powered up the machine. The new drive shows up as sdb. btrfs fi show still prints drive missing. mounted the filesystem with ro,degraded tried adding the new sdb drive which results in the following error. (-f because the new drive has a fs from past) # btrfs device add -f /dev/sdb /mnt2/raid1pool /dev/sdb is mounted Unless I am missing something, this looks like a bug. Let me know, I can retest. Thanks Suman On Mon, Oct 13, 2014 at 7:13 PM, Anand Jain anand.j...@oracle.com wrote: On 10/14/14 03:50, Suman C wrote: I had progs 3.12 and updated to the latest from git(3.16). With this update, btrfs fi show reports there is a missing device immediately after i pull it out. Thanks! I am using virtualbox to test this. So, I am detaching the drive like so: vboxmanage storageattach vm --storagectl controller --port port --device device --medium none Next I am going to try and test a more realistic scenario where a harddrive is not pulled out, but is damaged. Can/does btrfs mark a filesystem(say, 2 drive raid1) degraded or unhealthy automatically when one drive is damaged badly enough that it cannot be written to or read from reliably? There are some gaps as directly compared to an enterprise volume manager, which is being fixed. but pls do report what you find. Thanks, Anand Suman On Sun, Oct 12, 2014 at 7:21 PM, Anand Jain anand.j...@oracle.com wrote: Suman, To simulate the failure, I detached one of the drives from the system. After that, I see no sign of a problem except for these errors: Are you physically pulling out the device ? I wonder if lsblk or blkid shows the error ? reporting device missing logic is in the progs (so have that latest) and it works provided user script such as blkid/lsblk also reports the problem. OR for soft-detach tests you could use devmgt at http://github.com/anajain/devmgt Also I am trying to get the device management framework for the btrfs with a more better device management and reporting. Thanks, Anand On 10/13/14 07:50, Suman C wrote: Hi, I am testing some disk failure scenarios in a 2 drive raid1 mirror. They are 4GB each, virtual SATA drives inside virtualbox. To simulate the failure, I detached one of the drives from the system. After that, I see no sign of a problem except for these errors: Oct 12 15:37:14 rock-dev kernel: btrfs: bdev /dev/sdb errs: wr 0, rd 0, flush 1, corrupt 0, gen 0 Oct 12 15:37:14 rock-dev kernel: lost page write due to I/O error on /dev/sdb /dev/sdb is gone from the system, but btrfs fi show still lists it. Label: raid1pool uuid: 4e5d8b43-1d34-4672-8057-99c51649b7c6 Total devices 2 FS bytes used 1.46GiB devid1 size 4.00GiB used 2.45GiB path /dev/sdb devid2 size 4.00GiB used 2.43GiB path /dev/sdc I am able to read and write just fine, but do see the above errors in dmesg. What is the best way to find out that one of the drives has gone bad? Suman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: what is the best way to monitor raid1 drive failures?
On Tue, Oct 14, 2014 at 10:48 AM, Suman C schakr...@gmail.com wrote: The new drive shows up as sdb. btrfs fi show still prints drive missing. mounted the filesystem with ro,degraded tried adding the new sdb drive which results in the following error. (-f because the new drive has a fs from past) # btrfs device add -f /dev/sdb /mnt2/raid1pool /dev/sdb is mounted Unless I am missing something, this looks like a bug. You need to first run btrfs device delete missing /mnt2/raid1pool I believe (missing is a keyword for a missing device in the array - if the device were still present you could specify it by /dev/sdX). -- Rich -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: what is the best way to monitor raid1 drive failures?
I cannot delete that way because it would mean going below the minimum number of devices and it fails, as explained in the wiki. The solution from wiki is to add a new device and then delete the old one, but the problem here may be due to the new device appearing with the same name(sdb)? Suman On Tue, Oct 14, 2014 at 7:52 AM, Rich Freeman r-bt...@thefreemanclan.net wrote: On Tue, Oct 14, 2014 at 10:48 AM, Suman C schakr...@gmail.com wrote: The new drive shows up as sdb. btrfs fi show still prints drive missing. mounted the filesystem with ro,degraded tried adding the new sdb drive which results in the following error. (-f because the new drive has a fs from past) # btrfs device add -f /dev/sdb /mnt2/raid1pool /dev/sdb is mounted Unless I am missing something, this looks like a bug. You need to first run btrfs device delete missing /mnt2/raid1pool I believe (missing is a keyword for a missing device in the array - if the device were still present you could specify it by /dev/sdX). -- Rich -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: what is the best way to monitor raid1 drive failures?
On Oct 14, 2014, at 10:48 AM, Suman C schakr...@gmail.com wrote: mounted the filesystem with ro,degraded tried adding the new sdb drive which results in the following error. (-f because the new drive has a fs from past) # btrfs device add -f /dev/sdb /mnt2/raid1pool /dev/sdb is mounted Unless I am missing something, this looks like a bug. Strange message. I expect a device can't be added to a volume mounted ro. If the device add command works on a volume mounted rw, then the bug is the message '/dev/sdb is mounted' when adding device to ro mounted volume. Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: what is the best way to monitor raid1 drive failures?
After the reboot step, where I indicated that I mounted ro, I was unable to mount rw or rw,degraded. I get the mount: wrong fs type, bad option, bad superblock error if I try to mount it rw. What might be the reason for that? Suman On Tue, Oct 14, 2014 at 12:15 PM, Chris Murphy li...@colorremedies.com wrote: On Oct 14, 2014, at 10:48 AM, Suman C schakr...@gmail.com wrote: mounted the filesystem with ro,degraded tried adding the new sdb drive which results in the following error. (-f because the new drive has a fs from past) # btrfs device add -f /dev/sdb /mnt2/raid1pool /dev/sdb is mounted Unless I am missing something, this looks like a bug. Strange message. I expect a device can't be added to a volume mounted ro. If the device add command works on a volume mounted rw, then the bug is the message '/dev/sdb is mounted' when adding device to ro mounted volume. Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: what is the best way to monitor raid1 drive failures?
Suman C posted on Tue, 14 Oct 2014 07:48:01 -0700 as excerpted: Here's a simple raid1 recovery experiment that's not working as expected. kernel: 3.17, latest mainline progs: 3.16.1 I started with a simple raid1 mirror of 2 drives (sda and sdb). The filesystem is functional, I created one subvol, put some data, read/write tested etc.. yanked the sdb out. (this is physical/hardware). btrfs fi show prints drive missing, as expected. powered the machine down. removed the bad(yanked out sdb) drive and replaced it with a new drive. Powered up the machine. The new drive shows up as sdb. btrfs fi show still prints drive missing. mounted the filesystem with ro,degraded tried adding the new sdb drive which results in the following error. (-f because the new drive has a fs from past) # btrfs device add -f /dev/sdb /mnt2/raid1pool /dev/sdb is mounted While I'm not sure it'll get you past the error, did you try... # btrfs replace ... That's the new way to /replace/ a missing device, adding a new one and deleting the old one (which can be missing) at the same time. See the btrfs-replace manpage. While the btrfs-replace manpage says that you have to use the devid format if the device is missing, it isn't particularly helpful in telling what that format actually is. Do a btrfs fi show and use the appropriate devid /number/ from there. =:^) Please report back as I'm using btrfs raid1 as well, but my own tests are rather stale by this point and I'd have to figure it out as I went. So I'm highly interested in your results. =:^) (FWIW, personally I'd have made that btrfs device replace, instead of btrfs replace, to keep it grouped with the other device operations, but whatever, it's its own top-level command, now. Tho at least the btrfs-device manpage mentions btrfs replace and its manpage as well. But I still think having replace as its own top-level command is confusing.) -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: what is the best way to monitor raid1 drive failures?
On 10/14/14 22:48, Suman C wrote: Hi, Here's a simple raid1 recovery experiment that's not working as expected. kernel: 3.17, latest mainline progs: 3.16.1 I started with a simple raid1 mirror of 2 drives (sda and sdb). The filesystem is functional, I created one subvol, put some data, read/write tested etc.. yanked the sdb out. (this is physical/hardware). btrfs fi show prints drive missing, as expected. powered the machine down. removed the bad(yanked out sdb) drive and replaced it with a new drive. Powered up the machine. Does new drive at sdb contain some stale btrfs SB ? and was that FS mounted during boot ? Or simply unmount; wipefs -a /dev/sdb; reboot. That will help to achieve your test case. The new drive shows up as sdb. btrfs fi show still prints drive missing. good in fact. mounted the filesystem with ro,degraded ro wasn't required. And now you should be able to replace using devid/missing string. tried adding the new sdb drive which results in the following error. (-f because the new drive has a fs from past) # btrfs device add -f /dev/sdb /mnt2/raid1pool /dev/sdb is mounted Unless I am missing something, this looks like a bug. Our progs check_mounted() is not too strong, there might be bug. But as of now some info as above is missing. If you could have experimental patch: [PATCH RFC] btrfs: introduce procfs interface for the device list And read /proc/fs/btrfs/devlist That will tell you the real status of devices inside the kernel. progs tries to act intelligent at times not required. Anand Let me know, I can retest. Thanks Suman On Mon, Oct 13, 2014 at 7:13 PM, Anand Jain anand.j...@oracle.com wrote: On 10/14/14 03:50, Suman C wrote: I had progs 3.12 and updated to the latest from git(3.16). With this update, btrfs fi show reports there is a missing device immediately after i pull it out. Thanks! I am using virtualbox to test this. So, I am detaching the drive like so: vboxmanage storageattach vm --storagectl controller --port port --device device --medium none Next I am going to try and test a more realistic scenario where a harddrive is not pulled out, but is damaged. Can/does btrfs mark a filesystem(say, 2 drive raid1) degraded or unhealthy automatically when one drive is damaged badly enough that it cannot be written to or read from reliably? There are some gaps as directly compared to an enterprise volume manager, which is being fixed. but pls do report what you find. Thanks, Anand Suman On Sun, Oct 12, 2014 at 7:21 PM, Anand Jain anand.j...@oracle.com wrote: Suman, To simulate the failure, I detached one of the drives from the system. After that, I see no sign of a problem except for these errors: Are you physically pulling out the device ? I wonder if lsblk or blkid shows the error ? reporting device missing logic is in the progs (so have that latest) and it works provided user script such as blkid/lsblk also reports the problem. OR for soft-detach tests you could use devmgt at http://github.com/anajain/devmgt Also I am trying to get the device management framework for the btrfs with a more better device management and reporting. Thanks, Anand On 10/13/14 07:50, Suman C wrote: Hi, I am testing some disk failure scenarios in a 2 drive raid1 mirror. They are 4GB each, virtual SATA drives inside virtualbox. To simulate the failure, I detached one of the drives from the system. After that, I see no sign of a problem except for these errors: Oct 12 15:37:14 rock-dev kernel: btrfs: bdev /dev/sdb errs: wr 0, rd 0, flush 1, corrupt 0, gen 0 Oct 12 15:37:14 rock-dev kernel: lost page write due to I/O error on /dev/sdb /dev/sdb is gone from the system, but btrfs fi show still lists it. Label: raid1pool uuid: 4e5d8b43-1d34-4672-8057-99c51649b7c6 Total devices 2 FS bytes used 1.46GiB devid1 size 4.00GiB used 2.45GiB path /dev/sdb devid2 size 4.00GiB used 2.43GiB path /dev/sdc I am able to read and write just fine, but do see the above errors in dmesg. What is the best way to find out that one of the drives has gone bad? Suman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: what is the best way to monitor raid1 drive failures?
I had progs 3.12 and updated to the latest from git(3.16). With this update, btrfs fi show reports there is a missing device immediately after i pull it out. Thanks! I am using virtualbox to test this. So, I am detaching the drive like so: vboxmanage storageattach vm --storagectl controller --port port --device device --medium none Next I am going to try and test a more realistic scenario where a harddrive is not pulled out, but is damaged. Can/does btrfs mark a filesystem(say, 2 drive raid1) degraded or unhealthy automatically when one drive is damaged badly enough that it cannot be written to or read from reliably? Suman On Sun, Oct 12, 2014 at 7:21 PM, Anand Jain anand.j...@oracle.com wrote: Suman, To simulate the failure, I detached one of the drives from the system. After that, I see no sign of a problem except for these errors: Are you physically pulling out the device ? I wonder if lsblk or blkid shows the error ? reporting device missing logic is in the progs (so have that latest) and it works provided user script such as blkid/lsblk also reports the problem. OR for soft-detach tests you could use devmgt at http://github.com/anajain/devmgt Also I am trying to get the device management framework for the btrfs with a more better device management and reporting. Thanks, Anand On 10/13/14 07:50, Suman C wrote: Hi, I am testing some disk failure scenarios in a 2 drive raid1 mirror. They are 4GB each, virtual SATA drives inside virtualbox. To simulate the failure, I detached one of the drives from the system. After that, I see no sign of a problem except for these errors: Oct 12 15:37:14 rock-dev kernel: btrfs: bdev /dev/sdb errs: wr 0, rd 0, flush 1, corrupt 0, gen 0 Oct 12 15:37:14 rock-dev kernel: lost page write due to I/O error on /dev/sdb /dev/sdb is gone from the system, but btrfs fi show still lists it. Label: raid1pool uuid: 4e5d8b43-1d34-4672-8057-99c51649b7c6 Total devices 2 FS bytes used 1.46GiB devid1 size 4.00GiB used 2.45GiB path /dev/sdb devid2 size 4.00GiB used 2.43GiB path /dev/sdc I am able to read and write just fine, but do see the above errors in dmesg. What is the best way to find out that one of the drives has gone bad? Suman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: what is the best way to monitor raid1 drive failures?
On 10/14/14 03:50, Suman C wrote: I had progs 3.12 and updated to the latest from git(3.16). With this update, btrfs fi show reports there is a missing device immediately after i pull it out. Thanks! I am using virtualbox to test this. So, I am detaching the drive like so: vboxmanage storageattach vm --storagectl controller --port port --device device --medium none Next I am going to try and test a more realistic scenario where a harddrive is not pulled out, but is damaged. Can/does btrfs mark a filesystem(say, 2 drive raid1) degraded or unhealthy automatically when one drive is damaged badly enough that it cannot be written to or read from reliably? There are some gaps as directly compared to an enterprise volume manager, which is being fixed. but pls do report what you find. Thanks, Anand Suman On Sun, Oct 12, 2014 at 7:21 PM, Anand Jain anand.j...@oracle.com wrote: Suman, To simulate the failure, I detached one of the drives from the system. After that, I see no sign of a problem except for these errors: Are you physically pulling out the device ? I wonder if lsblk or blkid shows the error ? reporting device missing logic is in the progs (so have that latest) and it works provided user script such as blkid/lsblk also reports the problem. OR for soft-detach tests you could use devmgt at http://github.com/anajain/devmgt Also I am trying to get the device management framework for the btrfs with a more better device management and reporting. Thanks, Anand On 10/13/14 07:50, Suman C wrote: Hi, I am testing some disk failure scenarios in a 2 drive raid1 mirror. They are 4GB each, virtual SATA drives inside virtualbox. To simulate the failure, I detached one of the drives from the system. After that, I see no sign of a problem except for these errors: Oct 12 15:37:14 rock-dev kernel: btrfs: bdev /dev/sdb errs: wr 0, rd 0, flush 1, corrupt 0, gen 0 Oct 12 15:37:14 rock-dev kernel: lost page write due to I/O error on /dev/sdb /dev/sdb is gone from the system, but btrfs fi show still lists it. Label: raid1pool uuid: 4e5d8b43-1d34-4672-8057-99c51649b7c6 Total devices 2 FS bytes used 1.46GiB devid1 size 4.00GiB used 2.45GiB path /dev/sdb devid2 size 4.00GiB used 2.43GiB path /dev/sdc I am able to read and write just fine, but do see the above errors in dmesg. What is the best way to find out that one of the drives has gone bad? Suman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
what is the best way to monitor raid1 drive failures?
Hi, I am testing some disk failure scenarios in a 2 drive raid1 mirror. They are 4GB each, virtual SATA drives inside virtualbox. To simulate the failure, I detached one of the drives from the system. After that, I see no sign of a problem except for these errors: Oct 12 15:37:14 rock-dev kernel: btrfs: bdev /dev/sdb errs: wr 0, rd 0, flush 1, corrupt 0, gen 0 Oct 12 15:37:14 rock-dev kernel: lost page write due to I/O error on /dev/sdb /dev/sdb is gone from the system, but btrfs fi show still lists it. Label: raid1pool uuid: 4e5d8b43-1d34-4672-8057-99c51649b7c6 Total devices 2 FS bytes used 1.46GiB devid1 size 4.00GiB used 2.45GiB path /dev/sdb devid2 size 4.00GiB used 2.43GiB path /dev/sdc I am able to read and write just fine, but do see the above errors in dmesg. What is the best way to find out that one of the drives has gone bad? Suman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: what is the best way to monitor raid1 drive failures?
Suman, To simulate the failure, I detached one of the drives from the system. After that, I see no sign of a problem except for these errors: Are you physically pulling out the device ? I wonder if lsblk or blkid shows the error ? reporting device missing logic is in the progs (so have that latest) and it works provided user script such as blkid/lsblk also reports the problem. OR for soft-detach tests you could use devmgt at http://github.com/anajain/devmgt Also I am trying to get the device management framework for the btrfs with a more better device management and reporting. Thanks, Anand On 10/13/14 07:50, Suman C wrote: Hi, I am testing some disk failure scenarios in a 2 drive raid1 mirror. They are 4GB each, virtual SATA drives inside virtualbox. To simulate the failure, I detached one of the drives from the system. After that, I see no sign of a problem except for these errors: Oct 12 15:37:14 rock-dev kernel: btrfs: bdev /dev/sdb errs: wr 0, rd 0, flush 1, corrupt 0, gen 0 Oct 12 15:37:14 rock-dev kernel: lost page write due to I/O error on /dev/sdb /dev/sdb is gone from the system, but btrfs fi show still lists it. Label: raid1pool uuid: 4e5d8b43-1d34-4672-8057-99c51649b7c6 Total devices 2 FS bytes used 1.46GiB devid1 size 4.00GiB used 2.45GiB path /dev/sdb devid2 size 4.00GiB used 2.43GiB path /dev/sdc I am able to read and write just fine, but do see the above errors in dmesg. What is the best way to find out that one of the drives has gone bad? Suman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
