Re: zfs, cam sticking on failed disk
On Thu, May 07, 2015 at 09:41:43AM +0100, Steven Hartland wrote: On 07/05/2015 09:07, Slawa Olhovchenkov wrote: I have zpool of 12 vdev (zmirrors). One disk in one vdev out of service and stop serving reuquest: dT: 1.036s w: 1.000s L(q) ops/sr/s kBps ms/rw/s kBps ms/w %busy Name 0 0 0 00.0 0 00.00.0| ada0 0 0 0 00.0 0 00.00.0| ada1 1 0 0 00.0 0 00.00.0| ada2 0 0 0 00.0 0 00.00.0| ada3 0 0 0 00.0 0 00.00.0| da0 0 0 0 00.0 0 00.00.0| da1 0 0 0 00.0 0 00.00.0| da2 0 0 0 00.0 0 00.00.0| da3 0 0 0 00.0 0 00.00.0| da4 0 0 0 00.0 0 00.00.0| da5 0 0 0 00.0 0 00.00.0| da6 0 0 0 00.0 0 00.00.0| da7 0 0 0 00.0 0 00.00.0| da8 0 0 0 00.0 0 00.00.0| da9 0 0 0 00.0 0 00.00.0| da10 0 0 0 00.0 0 00.00.0| da11 0 0 0 00.0 0 00.00.0| da12 0 0 0 00.0 0 00.00.0| da13 0 0 0 00.0 0 00.00.0| da14 0 0 0 00.0 0 00.00.0| da15 0 0 0 00.0 0 00.00.0| da16 0 0 0 00.0 0 00.00.0| da17 0 0 0 00.0 0 00.00.0| da18 24 0 0 00.0 0 00.00.0| da19 0 0 0 00.0 0 00.00.0| da20 0 0 0 00.0 0 00.00.0| da21 0 0 0 00.0 0 00.00.0| da22 0 0 0 00.0 0 00.00.0| da23 0 0 0 00.0 0 00.00.0| da24 0 0 0 00.0 0 00.00.0| da25 0 0 0 00.0 0 00.00.0| da26 0 0 0 00.0 0 00.00.0| da27 As result zfs operation on this pool stoped too. `zpool list -v` don't worked. `zpool detach tank da19` don't worked. Application worked with this pool sticking in `zfs` wchan and don't killed. # camcontrol tags da19 -v (pass19:isci0:0:3:0): dev_openings 7 (pass19:isci0:0:3:0): dev_active25 (pass19:isci0:0:3:0): allocated 25 (pass19:isci0:0:3:0): queued0 (pass19:isci0:0:3:0): held 0 (pass19:isci0:0:3:0): mintags 2 (pass19:isci0:0:3:0): maxtags 255 How I can cancel this 24 requst? Why this requests don't timeout (3 hours already)? How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). Why ZFS (or geom) don't timeout on request and don't rerouted to da18? If they are in mirrors, in theory you can just pull the disk, isci will report to cam and cam will report to ZFS which should all recover. Yes, zmirror with da18. I am surprise that ZFS don't use da18. All zpool fully stuck. With regards to not timing out this could be a default issue, but having I am understand, no universal acceptable timeout for all cases: good disk, good saturated disk, tape, tape library, failed disk, etc. In my case -- failed disk. This model already failed (other specimen) with same symptoms). May be exist some tricks for cancel/aborting all request in queue and removing disk from system? a very quick look that's not obvious in the code as isci_io_request_construct etc do indeed set a timeout when CAM_TIME_INFINITY hasn't been requested. The sysctl hw.isci.debug_level may be able to provide more information, but be aware this can be spammy. I am already have this situation, what command interesting after setting hw.isci.debug_level? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs, cam sticking on failed disk
On 07/05/2015 09:07, Slawa Olhovchenkov wrote: I have zpool of 12 vdev (zmirrors). One disk in one vdev out of service and stop serving reuquest: dT: 1.036s w: 1.000s L(q) ops/sr/s kBps ms/rw/s kBps ms/w %busy Name 0 0 0 00.0 0 00.00.0| ada0 0 0 0 00.0 0 00.00.0| ada1 1 0 0 00.0 0 00.00.0| ada2 0 0 0 00.0 0 00.00.0| ada3 0 0 0 00.0 0 00.00.0| da0 0 0 0 00.0 0 00.00.0| da1 0 0 0 00.0 0 00.00.0| da2 0 0 0 00.0 0 00.00.0| da3 0 0 0 00.0 0 00.00.0| da4 0 0 0 00.0 0 00.00.0| da5 0 0 0 00.0 0 00.00.0| da6 0 0 0 00.0 0 00.00.0| da7 0 0 0 00.0 0 00.00.0| da8 0 0 0 00.0 0 00.00.0| da9 0 0 0 00.0 0 00.00.0| da10 0 0 0 00.0 0 00.00.0| da11 0 0 0 00.0 0 00.00.0| da12 0 0 0 00.0 0 00.00.0| da13 0 0 0 00.0 0 00.00.0| da14 0 0 0 00.0 0 00.00.0| da15 0 0 0 00.0 0 00.00.0| da16 0 0 0 00.0 0 00.00.0| da17 0 0 0 00.0 0 00.00.0| da18 24 0 0 00.0 0 00.00.0| da19 0 0 0 00.0 0 00.00.0| da20 0 0 0 00.0 0 00.00.0| da21 0 0 0 00.0 0 00.00.0| da22 0 0 0 00.0 0 00.00.0| da23 0 0 0 00.0 0 00.00.0| da24 0 0 0 00.0 0 00.00.0| da25 0 0 0 00.0 0 00.00.0| da26 0 0 0 00.0 0 00.00.0| da27 As result zfs operation on this pool stoped too. `zpool list -v` don't worked. `zpool detach tank da19` don't worked. Application worked with this pool sticking in `zfs` wchan and don't killed. # camcontrol tags da19 -v (pass19:isci0:0:3:0): dev_openings 7 (pass19:isci0:0:3:0): dev_active25 (pass19:isci0:0:3:0): allocated 25 (pass19:isci0:0:3:0): queued0 (pass19:isci0:0:3:0): held 0 (pass19:isci0:0:3:0): mintags 2 (pass19:isci0:0:3:0): maxtags 255 How I can cancel this 24 requst? Why this requests don't timeout (3 hours already)? How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). Why ZFS (or geom) don't timeout on request and don't rerouted to da18? If they are in mirrors, in theory you can just pull the disk, isci will report to cam and cam will report to ZFS which should all recover. With regards to not timing out this could be a default issue, but having a very quick look that's not obvious in the code as isci_io_request_construct etc do indeed set a timeout when CAM_TIME_INFINITY hasn't been requested. The sysctl hw.isci.debug_level may be able to provide more information, but be aware this can be spammy. Regards Steve ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs, cam sticking on failed disk
On 07/05/2015 15:28, Matthew Seaman wrote: On 05/07/15 14:32, Steven Hartland wrote: I wouldn't have thought so, I would expect that to only have an effect on removal media such as CDROM drives, but no harm in trying ;-) zpool offline -t zroot da19 That might work but it also might just wedge waiting for the outstanding IO to complete as I thought that was a nice off-line instead of a I don't care one. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
zfs, cam sticking on failed disk
I have zpool of 12 vdev (zmirrors). One disk in one vdev out of service and stop serving reuquest: dT: 1.036s w: 1.000s L(q) ops/sr/s kBps ms/rw/s kBps ms/w %busy Name 0 0 0 00.0 0 00.00.0| ada0 0 0 0 00.0 0 00.00.0| ada1 1 0 0 00.0 0 00.00.0| ada2 0 0 0 00.0 0 00.00.0| ada3 0 0 0 00.0 0 00.00.0| da0 0 0 0 00.0 0 00.00.0| da1 0 0 0 00.0 0 00.00.0| da2 0 0 0 00.0 0 00.00.0| da3 0 0 0 00.0 0 00.00.0| da4 0 0 0 00.0 0 00.00.0| da5 0 0 0 00.0 0 00.00.0| da6 0 0 0 00.0 0 00.00.0| da7 0 0 0 00.0 0 00.00.0| da8 0 0 0 00.0 0 00.00.0| da9 0 0 0 00.0 0 00.00.0| da10 0 0 0 00.0 0 00.00.0| da11 0 0 0 00.0 0 00.00.0| da12 0 0 0 00.0 0 00.00.0| da13 0 0 0 00.0 0 00.00.0| da14 0 0 0 00.0 0 00.00.0| da15 0 0 0 00.0 0 00.00.0| da16 0 0 0 00.0 0 00.00.0| da17 0 0 0 00.0 0 00.00.0| da18 24 0 0 00.0 0 00.00.0| da19 0 0 0 00.0 0 00.00.0| da20 0 0 0 00.0 0 00.00.0| da21 0 0 0 00.0 0 00.00.0| da22 0 0 0 00.0 0 00.00.0| da23 0 0 0 00.0 0 00.00.0| da24 0 0 0 00.0 0 00.00.0| da25 0 0 0 00.0 0 00.00.0| da26 0 0 0 00.0 0 00.00.0| da27 As result zfs operation on this pool stoped too. `zpool list -v` don't worked. `zpool detach tank da19` don't worked. Application worked with this pool sticking in `zfs` wchan and don't killed. # camcontrol tags da19 -v (pass19:isci0:0:3:0): dev_openings 7 (pass19:isci0:0:3:0): dev_active25 (pass19:isci0:0:3:0): allocated 25 (pass19:isci0:0:3:0): queued0 (pass19:isci0:0:3:0): held 0 (pass19:isci0:0:3:0): mintags 2 (pass19:isci0:0:3:0): maxtags 255 How I can cancel this 24 requst? Why this requests don't timeout (3 hours already)? How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). Why ZFS (or geom) don't timeout on request and don't rerouted to da18? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs, cam sticking on failed disk
On 07/05/2015 13:51, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 01:46:40PM +0100, Steven Hartland wrote: Yes in theory new requests should go to the other vdev, but there could be some dependency issues preventing that such as a syncing TXG. Currenly this pool must not have write activity (from application). What about go to the other (mirror) device in the same vdev? Same dependency? Yes, if there's an outstanding TXG, then I believe all IO will stall. Where this TXG released? When all devices in all vdevs report 'completed'? When at the least one device in all vdevs report 'completed'? When at the least one device in at least one vdev report 'completed'? When all devices have report completed or failed. Hence if you pull the disk things should continue as normal, with the failed device being marked as such. Regards Steve ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs, cam sticking on failed disk
On Thu, May 07, 2015 at 02:05:11PM +0100, Steven Hartland wrote: On 07/05/2015 13:51, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 01:46:40PM +0100, Steven Hartland wrote: Yes in theory new requests should go to the other vdev, but there could be some dependency issues preventing that such as a syncing TXG. Currenly this pool must not have write activity (from application). What about go to the other (mirror) device in the same vdev? Same dependency? Yes, if there's an outstanding TXG, then I believe all IO will stall. Where this TXG released? When all devices in all vdevs report 'completed'? When at the least one device in all vdevs report 'completed'? When at the least one device in at least one vdev report 'completed'? When all devices have report completed or failed. Thanks for explained. Hence if you pull the disk things should continue as normal, with the failed device being marked as such. I am have trouble to phisical access. May be someone can be suggest software method to forced detach device from system. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs, cam sticking on failed disk
On Thu, May 07, 2015 at 01:35:05PM +0100, Steven Hartland wrote: On 07/05/2015 13:05, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 01:00:40PM +0100, Steven Hartland wrote: On 07/05/2015 11:46, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 11:38:46AM +0100, Steven Hartland wrote: How I can cancel this 24 requst? Why this requests don't timeout (3 hours already)? How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). Why ZFS (or geom) don't timeout on request and don't rerouted to da18? If they are in mirrors, in theory you can just pull the disk, isci will report to cam and cam will report to ZFS which should all recover. Yes, zmirror with da18. I am surprise that ZFS don't use da18. All zpool fully stuck. A single low level request can only be handled by one device, if that device returns an error then ZFS will use the other device, but not until. Why next requests don't routed to da18? Current request stuck on da19 (unlikely, but understund), but why stuck all pool? Its still waiting for the request from the failed device to complete. As far as ZFS currently knows there is nothing wrong with the device as its had no failures. Can you explain some more? One requst waiting, understand. I am do next request. Some information need from vdev with failed disk. Failed disk more busy (queue long), why don't routed to mirror disk? Or, for metadata, to less busy vdev? As no error has been reported to ZFS, due to the stalled IO, there is no failed vdev. I see that device isn't failed (for both OS and ZFS). I am don't talk 'failed vdev'. I am talk 'busy vdev' or 'busy device'. Yes in theory new requests should go to the other vdev, but there could be some dependency issues preventing that such as a syncing TXG. Currenly this pool must not have write activity (from application). What about go to the other (mirror) device in the same vdev? Same dependency? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs, cam sticking on failed disk
On Thu, May 07, 2015 at 01:46:40PM +0100, Steven Hartland wrote: Yes in theory new requests should go to the other vdev, but there could be some dependency issues preventing that such as a syncing TXG. Currenly this pool must not have write activity (from application). What about go to the other (mirror) device in the same vdev? Same dependency? Yes, if there's an outstanding TXG, then I believe all IO will stall. Where this TXG released? When all devices in all vdevs report 'completed'? When at the least one device in all vdevs report 'completed'? When at the least one device in at least one vdev report 'completed'? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
panic: pmap active 0xfffff8001b7154b8
Hi, We’ve been seeing (seemingly) random reboots on 10.1-RELEASE virtual machines (KVM virtualisation) on our production servers. In an attempt to determine what was causing this we’ve switched to running a kernel with INVARIANTS enabled. This resulted for us in the following panic: Unread portion of the kernel message buffer: panic: pmap active 0xf8001b7154b8 cpuid = 3 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe03dd1493a0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfe03dd149450 vpanic() at vpanic+0x126/frame 0xfe03dd149490 kassert_panic() at kassert_panic+0x139/frame 0xfe03dd149500 pmap_remove_pages() at pmap_remove_pages+0x8c/frame 0xfe03dd1495f0 exec_new_vmspace() at exec_new_vmspace+0x16a/frame 0xfe03dd149650 exec_elf64_imgact() at exec_elf64_imgact+0x658/frame 0xfe03dd149720 kern_execve() at kern_execve+0x5e4/frame 0xfe03dd149a80 sys_execve() at sys_execve+0x37/frame 0xfe03dd149ae0 amd64_syscall() at amd64_syscall+0x25a/frame 0xfe03dd149bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfe03dd149bf0 --- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x80158af1a, rsp = 0x7fffac38, rbp = 0x7fffad40 --- I’ve only come across one other report here (without result unfortunate): https://lists.freebsd.org/pipermail/freebsd-current/2014-June/050827.html https://lists.freebsd.org/pipermail/freebsd-current/2014-June/050827.html Are other people aware of this issue or working on this? I can provide access to a VM with a kernel dump and the kernel build for extra information if needed. - Johan ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs, cam sticking on failed disk
On 07/05/2015 13:44, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 01:35:05PM +0100, Steven Hartland wrote: On 07/05/2015 13:05, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 01:00:40PM +0100, Steven Hartland wrote: On 07/05/2015 11:46, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 11:38:46AM +0100, Steven Hartland wrote: How I can cancel this 24 requst? Why this requests don't timeout (3 hours already)? How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). Why ZFS (or geom) don't timeout on request and don't rerouted to da18? If they are in mirrors, in theory you can just pull the disk, isci will report to cam and cam will report to ZFS which should all recover. Yes, zmirror with da18. I am surprise that ZFS don't use da18. All zpool fully stuck. A single low level request can only be handled by one device, if that device returns an error then ZFS will use the other device, but not until. Why next requests don't routed to da18? Current request stuck on da19 (unlikely, but understund), but why stuck all pool? Its still waiting for the request from the failed device to complete. As far as ZFS currently knows there is nothing wrong with the device as its had no failures. Can you explain some more? One requst waiting, understand. I am do next request. Some information need from vdev with failed disk. Failed disk more busy (queue long), why don't routed to mirror disk? Or, for metadata, to less busy vdev? As no error has been reported to ZFS, due to the stalled IO, there is no failed vdev. I see that device isn't failed (for both OS and ZFS). I am don't talk 'failed vdev'. I am talk 'busy vdev' or 'busy device'. Yes in theory new requests should go to the other vdev, but there could be some dependency issues preventing that such as a syncing TXG. Currenly this pool must not have write activity (from application). What about go to the other (mirror) device in the same vdev? Same dependency? Yes, if there's an outstanding TXG, then I believe all IO will stall. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs, cam sticking on failed disk
On Thu, May 07, 2015 at 01:00:40PM +0100, Steven Hartland wrote: On 07/05/2015 11:46, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 11:38:46AM +0100, Steven Hartland wrote: How I can cancel this 24 requst? Why this requests don't timeout (3 hours already)? How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). Why ZFS (or geom) don't timeout on request and don't rerouted to da18? If they are in mirrors, in theory you can just pull the disk, isci will report to cam and cam will report to ZFS which should all recover. Yes, zmirror with da18. I am surprise that ZFS don't use da18. All zpool fully stuck. A single low level request can only be handled by one device, if that device returns an error then ZFS will use the other device, but not until. Why next requests don't routed to da18? Current request stuck on da19 (unlikely, but understund), but why stuck all pool? Its still waiting for the request from the failed device to complete. As far as ZFS currently knows there is nothing wrong with the device as its had no failures. Can you explain some more? One requst waiting, understand. I am do next request. Some information need from vdev with failed disk. Failed disk more busy (queue long), why don't routed to mirror disk? Or, for metadata, to less busy vdev? You didn't say which FreeBSD version you where running? 10-STABLE, r281264. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs, cam sticking on failed disk
On 07/05/2015 13:05, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 01:00:40PM +0100, Steven Hartland wrote: On 07/05/2015 11:46, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 11:38:46AM +0100, Steven Hartland wrote: How I can cancel this 24 requst? Why this requests don't timeout (3 hours already)? How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). Why ZFS (or geom) don't timeout on request and don't rerouted to da18? If they are in mirrors, in theory you can just pull the disk, isci will report to cam and cam will report to ZFS which should all recover. Yes, zmirror with da18. I am surprise that ZFS don't use da18. All zpool fully stuck. A single low level request can only be handled by one device, if that device returns an error then ZFS will use the other device, but not until. Why next requests don't routed to da18? Current request stuck on da19 (unlikely, but understund), but why stuck all pool? Its still waiting for the request from the failed device to complete. As far as ZFS currently knows there is nothing wrong with the device as its had no failures. Can you explain some more? One requst waiting, understand. I am do next request. Some information need from vdev with failed disk. Failed disk more busy (queue long), why don't routed to mirror disk? Or, for metadata, to less busy vdev? As no error has been reported to ZFS, due to the stalled IO, there is no failed vdev. Yes in theory new requests should go to the other vdev, but there could be some dependency issues preventing that such as a syncing TXG. Regards Steve ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs, cam sticking on failed disk
On Thu, May 07, 2015 at 03:29:20PM +0200, Ronald Klop wrote: On Thu, 07 May 2015 15:23:58 +0200, Steven Hartland kill...@multiplay.co.uk wrote: On 07/05/2015 14:10, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 02:05:11PM +0100, Steven Hartland wrote: On 07/05/2015 13:51, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 01:46:40PM +0100, Steven Hartland wrote: Yes in theory new requests should go to the other vdev, but there could be some dependency issues preventing that such as a syncing TXG. Currenly this pool must not have write activity (from application). What about go to the other (mirror) device in the same vdev? Same dependency? Yes, if there's an outstanding TXG, then I believe all IO will stall. Where this TXG released? When all devices in all vdevs report 'completed'? When at the least one device in all vdevs report 'completed'? When at the least one device in at least one vdev report 'completed'? When all devices have report completed or failed. Thanks for explained. Hence if you pull the disk things should continue as normal, with the failed device being marked as such. I am have trouble to phisical access. May be someone can be suggest software method to forced detach device from system. In 11 that should be possible with devctl, but under 10 I'm not aware of anything that wouldn't involve some custom kernel level code I'm afraid. Maybe I'm talking BS here, but does 'camcontrol eject' do something on a disk? I am don't try 'camcontrol eject' but 'camcontrol identify' already stalled. Need control on adapter layer. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs, cam sticking on failed disk
On May 7, 2015, at 8:00 AM, Steven Hartland kill...@multiplay.co.uk wrote: On 07/05/2015 11:46, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 11:38:46AM +0100, Steven Hartland wrote: How I can cancel this 24 requst? Why this requests don't timeout (3 hours already)? How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). Why ZFS (or geom) don't timeout on request and don't rerouted to da18? If they are in mirrors, in theory you can just pull the disk, isci will report to cam and cam will report to ZFS which should all recover. Yes, zmirror with da18. I am surprise that ZFS don't use da18. All zpool fully stuck. A single low level request can only be handled by one device, if that device returns an error then ZFS will use the other device, but not until. Why next requests don't routed to da18? Current request stuck on da19 (unlikely, but understund), but why stuck all pool? Its still waiting for the request from the failed device to complete. As far as ZFS currently knows there is nothing wrong with the device as its had no failures. Maybe related to this, but if the drive stalls indefinitely, is it what leads to the panic: I/O to pool 'poolname' appears to be hung on vdev guid GUID-ID at '/dev/somedevice'.? I have a 6-disk RAIDZ2 pool that is used for nightly rsync backups from various systems. I believe one of the drives is a bit temperamental. Very occasionally, I discover the backup has failed and the machine actually paniced because of this drive, with a panic message like the above. The panic backtrace includes references to vdev_deadman, which sounds like some sort of dead man's switch/watchdog. It's a bit counter-intuitive that a single drive wandering off into la-la land can not only cause an entire ZFS pool to wedge, but, worse still, panic the whole machine. If I'm understanding this thread correctly, part of the problem is that an I/O never completing is not the same as a failure to ZFS, and hence ZFS can't call upon various resources in the pool and mechanisms at its disposal to correct for that. Is that accurate? I would think that never-ending I/O requests would be a type of failure that ZFS could sustain. It seems from the hung on vdev panic that it does detect this situation, though the resolution (panic) is not ideal. :-) Cheers, Paul. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs, cam sticking on failed disk
On 07/05/2015 14:10, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 02:05:11PM +0100, Steven Hartland wrote: On 07/05/2015 13:51, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 01:46:40PM +0100, Steven Hartland wrote: Yes in theory new requests should go to the other vdev, but there could be some dependency issues preventing that such as a syncing TXG. Currenly this pool must not have write activity (from application). What about go to the other (mirror) device in the same vdev? Same dependency? Yes, if there's an outstanding TXG, then I believe all IO will stall. Where this TXG released? When all devices in all vdevs report 'completed'? When at the least one device in all vdevs report 'completed'? When at the least one device in at least one vdev report 'completed'? When all devices have report completed or failed. Thanks for explained. Hence if you pull the disk things should continue as normal, with the failed device being marked as such. I am have trouble to phisical access. May be someone can be suggest software method to forced detach device from system. In 11 that should be possible with devctl, but under 10 I'm not aware of anything that wouldn't involve some custom kernel level code I'm afraid. Regards Steve ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs, cam sticking on failed disk
On 07/05/2015 14:29, Ronald Klop wrote: On Thu, 07 May 2015 15:23:58 +0200, Steven Hartland kill...@multiplay.co.uk wrote: On 07/05/2015 14:10, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 02:05:11PM +0100, Steven Hartland wrote: On 07/05/2015 13:51, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 01:46:40PM +0100, Steven Hartland wrote: Yes in theory new requests should go to the other vdev, but there could be some dependency issues preventing that such as a syncing TXG. Currenly this pool must not have write activity (from application). What about go to the other (mirror) device in the same vdev? Same dependency? Yes, if there's an outstanding TXG, then I believe all IO will stall. Where this TXG released? When all devices in all vdevs report 'completed'? When at the least one device in all vdevs report 'completed'? When at the least one device in at least one vdev report 'completed'? When all devices have report completed or failed. Thanks for explained. Hence if you pull the disk things should continue as normal, with the failed device being marked as such. I am have trouble to phisical access. May be someone can be suggest software method to forced detach device from system. In 11 that should be possible with devctl, but under 10 I'm not aware of anything that wouldn't involve some custom kernel level code I'm afraid. Maybe I'm talking BS here, but does 'camcontrol eject' do something on a disk? I wouldn't have thought so, I would expect that to only have an effect on removal media such as CDROM drives, but no harm in trying ;-) Regards Steve ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs, cam sticking on failed disk
On Thu, 07 May 2015 15:23:58 +0200, Steven Hartland kill...@multiplay.co.uk wrote: On 07/05/2015 14:10, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 02:05:11PM +0100, Steven Hartland wrote: On 07/05/2015 13:51, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 01:46:40PM +0100, Steven Hartland wrote: Yes in theory new requests should go to the other vdev, but there could be some dependency issues preventing that such as a syncing TXG. Currenly this pool must not have write activity (from application). What about go to the other (mirror) device in the same vdev? Same dependency? Yes, if there's an outstanding TXG, then I believe all IO will stall. Where this TXG released? When all devices in all vdevs report 'completed'? When at the least one device in all vdevs report 'completed'? When at the least one device in at least one vdev report 'completed'? When all devices have report completed or failed. Thanks for explained. Hence if you pull the disk things should continue as normal, with the failed device being marked as such. I am have trouble to phisical access. May be someone can be suggest software method to forced detach device from system. In 11 that should be possible with devctl, but under 10 I'm not aware of anything that wouldn't involve some custom kernel level code I'm afraid. Maybe I'm talking BS here, but does 'camcontrol eject' do something on a disk? Ronald. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: panic: pmap active 0xfffff8001b7154b8
On 5/7/2015 7:08 AM, Johan Schuijt-Li wrote: Hi, We’ve been seeing (seemingly) random reboots on 10.1-RELEASE virtual machines (KVM virtualisation) on our production servers. In an attempt to determine what was causing this we’ve switched to running a kernel with INVARIANTS enabled. This resulted for us in the following panic: Unread portion of the kernel message buffer: panic: pmap active 0xf8001b7154b8 cpuid = 3 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe03dd1493a0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfe03dd149450 vpanic() at vpanic+0x126/frame 0xfe03dd149490 kassert_panic() at kassert_panic+0x139/frame 0xfe03dd149500 pmap_remove_pages() at pmap_remove_pages+0x8c/frame 0xfe03dd1495f0 exec_new_vmspace() at exec_new_vmspace+0x16a/frame 0xfe03dd149650 exec_elf64_imgact() at exec_elf64_imgact+0x658/frame 0xfe03dd149720 kern_execve() at kern_execve+0x5e4/frame 0xfe03dd149a80 sys_execve() at sys_execve+0x37/frame 0xfe03dd149ae0 amd64_syscall() at amd64_syscall+0x25a/frame 0xfe03dd149bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfe03dd149bf0 --- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x80158af1a, rsp = 0x7fffac38, rbp = 0x7fffad40 --- I’ve only come across one other report here (without result unfortunate): https://lists.freebsd.org/pipermail/freebsd-current/2014-June/050827.html https://lists.freebsd.org/pipermail/freebsd-current/2014-June/050827.html I looked around for the conclusion of that thread but could not find it. I was reproducing so often I'm sure this case was fixed. I may have privately contacted one of the VM maintainers to fix it. However lacking evidence I think it just stopped happening for me and I never reported anything useful. Are other people aware of this issue or working on this? I can provide access to a VM with a kernel dump and the kernel build for extra information if needed. What we really need is a full core dump (minidump) and backtrace. This will let us inspect the pmap state. https://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html https://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug-gdb.html -- Regards, Bryan Drewery signature.asc Description: OpenPGP digital signature
Re: panic: pmap active 0xfffff8001b7154b8
On 5/7/2015 10:06 AM, Bryan Drewery wrote: On 5/7/2015 7:08 AM, Johan Schuijt-Li wrote: Hi, We’ve been seeing (seemingly) random reboots on 10.1-RELEASE virtual machines (KVM virtualisation) on our production servers. In an attempt to determine what was causing this we’ve switched to running a kernel with INVARIANTS enabled. This resulted for us in the following panic: Unread portion of the kernel message buffer: panic: pmap active 0xf8001b7154b8 cpuid = 3 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe03dd1493a0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfe03dd149450 vpanic() at vpanic+0x126/frame 0xfe03dd149490 kassert_panic() at kassert_panic+0x139/frame 0xfe03dd149500 pmap_remove_pages() at pmap_remove_pages+0x8c/frame 0xfe03dd1495f0 exec_new_vmspace() at exec_new_vmspace+0x16a/frame 0xfe03dd149650 exec_elf64_imgact() at exec_elf64_imgact+0x658/frame 0xfe03dd149720 kern_execve() at kern_execve+0x5e4/frame 0xfe03dd149a80 sys_execve() at sys_execve+0x37/frame 0xfe03dd149ae0 amd64_syscall() at amd64_syscall+0x25a/frame 0xfe03dd149bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfe03dd149bf0 --- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x80158af1a, rsp = 0x7fffac38, rbp = 0x7fffad40 --- I’ve only come across one other report here (without result unfortunate): https://lists.freebsd.org/pipermail/freebsd-current/2014-June/050827.html https://lists.freebsd.org/pipermail/freebsd-current/2014-June/050827.html I looked around for the conclusion of that thread but could not find it. Found it. https://svnweb.freebsd.org/base?view=revisionrevision=271000 This fix is in 10.1-RELEASE, so yours must be a little different. -- Regards, Bryan Drewery signature.asc Description: OpenPGP digital signature
Re: zfs, cam sticking on failed disk
On 05/07/15 14:32, Steven Hartland wrote: On 07/05/2015 14:29, Ronald Klop wrote: On Thu, 07 May 2015 15:23:58 +0200, Steven Hartland kill...@multiplay.co.uk wrote: On 07/05/2015 14:10, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 02:05:11PM +0100, Steven Hartland wrote: On 07/05/2015 13:51, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 01:46:40PM +0100, Steven Hartland wrote: Yes in theory new requests should go to the other vdev, but there could be some dependency issues preventing that such as a syncing TXG. Currenly this pool must not have write activity (from application). What about go to the other (mirror) device in the same vdev? Same dependency? Yes, if there's an outstanding TXG, then I believe all IO will stall. Where this TXG released? When all devices in all vdevs report 'completed'? When at the least one device in all vdevs report 'completed'? When at the least one device in at least one vdev report 'completed'? When all devices have report completed or failed. Thanks for explained. Hence if you pull the disk things should continue as normal, with the failed device being marked as such. I am have trouble to phisical access. May be someone can be suggest software method to forced detach device from system. In 11 that should be possible with devctl, but under 10 I'm not aware of anything that wouldn't involve some custom kernel level code I'm afraid. Maybe I'm talking BS here, but does 'camcontrol eject' do something on a disk? I wouldn't have thought so, I would expect that to only have an effect on removal media such as CDROM drives, but no harm in trying ;-) zpool offline -t zroot da19 Cheers, Matthew signature.asc Description: OpenPGP digital signature
Re: panic: pmap active 0xfffff8001b7154b8
What we really need is a full core dump (minidump) and backtrace. This will let us inspect the pmap state. https://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html https://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html https://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug-gdb.html https://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug-gdb.html Sorry if my words were a bit unclear, but this is what I have. If you could e-mail me a preferred username with public key I can give you access to this. - Johan ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs, cam sticking on failed disk
On Thu, May 07, 2015 at 11:38:46AM +0100, Steven Hartland wrote: How I can cancel this 24 requst? Why this requests don't timeout (3 hours already)? How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). Why ZFS (or geom) don't timeout on request and don't rerouted to da18? If they are in mirrors, in theory you can just pull the disk, isci will report to cam and cam will report to ZFS which should all recover. Yes, zmirror with da18. I am surprise that ZFS don't use da18. All zpool fully stuck. A single low level request can only be handled by one device, if that device returns an error then ZFS will use the other device, but not until. Why next requests don't routed to da18? Current request stuck on da19 (unlikely, but understund), but why stuck all pool? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs, cam sticking on failed disk
On 07/05/2015 11:46, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 11:38:46AM +0100, Steven Hartland wrote: How I can cancel this 24 requst? Why this requests don't timeout (3 hours already)? How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). Why ZFS (or geom) don't timeout on request and don't rerouted to da18? If they are in mirrors, in theory you can just pull the disk, isci will report to cam and cam will report to ZFS which should all recover. Yes, zmirror with da18. I am surprise that ZFS don't use da18. All zpool fully stuck. A single low level request can only be handled by one device, if that device returns an error then ZFS will use the other device, but not until. Why next requests don't routed to da18? Current request stuck on da19 (unlikely, but understund), but why stuck all pool? Its still waiting for the request from the failed device to complete. As far as ZFS currently knows there is nothing wrong with the device as its had no failures. You didn't say which FreeBSD version you where running? Regards Steve ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: zfs, cam sticking on failed disk
On 07/05/2015 10:50, Slawa Olhovchenkov wrote: On Thu, May 07, 2015 at 09:41:43AM +0100, Steven Hartland wrote: On 07/05/2015 09:07, Slawa Olhovchenkov wrote: I have zpool of 12 vdev (zmirrors). One disk in one vdev out of service and stop serving reuquest: dT: 1.036s w: 1.000s L(q) ops/sr/s kBps ms/rw/s kBps ms/w %busy Name 0 0 0 00.0 0 00.00.0| ada0 0 0 0 00.0 0 00.00.0| ada1 1 0 0 00.0 0 00.00.0| ada2 0 0 0 00.0 0 00.00.0| ada3 0 0 0 00.0 0 00.00.0| da0 0 0 0 00.0 0 00.00.0| da1 0 0 0 00.0 0 00.00.0| da2 0 0 0 00.0 0 00.00.0| da3 0 0 0 00.0 0 00.00.0| da4 0 0 0 00.0 0 00.00.0| da5 0 0 0 00.0 0 00.00.0| da6 0 0 0 00.0 0 00.00.0| da7 0 0 0 00.0 0 00.00.0| da8 0 0 0 00.0 0 00.00.0| da9 0 0 0 00.0 0 00.00.0| da10 0 0 0 00.0 0 00.00.0| da11 0 0 0 00.0 0 00.00.0| da12 0 0 0 00.0 0 00.00.0| da13 0 0 0 00.0 0 00.00.0| da14 0 0 0 00.0 0 00.00.0| da15 0 0 0 00.0 0 00.00.0| da16 0 0 0 00.0 0 00.00.0| da17 0 0 0 00.0 0 00.00.0| da18 24 0 0 00.0 0 00.00.0| da19 0 0 0 00.0 0 00.00.0| da20 0 0 0 00.0 0 00.00.0| da21 0 0 0 00.0 0 00.00.0| da22 0 0 0 00.0 0 00.00.0| da23 0 0 0 00.0 0 00.00.0| da24 0 0 0 00.0 0 00.00.0| da25 0 0 0 00.0 0 00.00.0| da26 0 0 0 00.0 0 00.00.0| da27 As result zfs operation on this pool stoped too. `zpool list -v` don't worked. `zpool detach tank da19` don't worked. Application worked with this pool sticking in `zfs` wchan and don't killed. # camcontrol tags da19 -v (pass19:isci0:0:3:0): dev_openings 7 (pass19:isci0:0:3:0): dev_active25 (pass19:isci0:0:3:0): allocated 25 (pass19:isci0:0:3:0): queued0 (pass19:isci0:0:3:0): held 0 (pass19:isci0:0:3:0): mintags 2 (pass19:isci0:0:3:0): maxtags 255 How I can cancel this 24 requst? Why this requests don't timeout (3 hours already)? How I can forced detach this disk? (I am lready try `camcontrol reset`, `camconrol rescan`). Why ZFS (or geom) don't timeout on request and don't rerouted to da18? If they are in mirrors, in theory you can just pull the disk, isci will report to cam and cam will report to ZFS which should all recover. Yes, zmirror with da18. I am surprise that ZFS don't use da18. All zpool fully stuck. A single low level request can only be handled by one device, if that device returns an error then ZFS will use the other device, but not until. With regards to not timing out this could be a default issue, but having I am understand, no universal acceptable timeout for all cases: good disk, good saturated disk, tape, tape library, failed disk, etc. In my case -- failed disk. This model already failed (other specimen) with same symptoms). May be exist some tricks for cancel/aborting all request in queue and removing disk from system? Unlikely tbh, pulling the disk however should. a very quick look that's not obvious in the code as isci_io_request_construct etc do indeed set a timeout when CAM_TIME_INFINITY hasn't been requested. The sysctl hw.isci.debug_level may be able to provide more information, but be aware this can be spammy. I am already have this situation, what command interesting after setting hw.isci.debug_level? I'm afraid I'm not familiar isci I'm afraid possibly someone else who is can chime in. Regards Steve ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
