[zfs-discuss] FW: Solaris panic
Hi all I have only just seen this, and thought someone may be able to help. On heavy IO activity, my Solaris 11 Express box hosting a ZFS data pool crashes. It seems to show page faults in several things, including nfsd, sched, zpool-tank and automountd. I get the following in the logs: Mar 17 15:28:17 zippy unix: [ID 836849 kern.notice] Mar 17 15:28:17 zippy ^Mpanic[cpu1]/thread=ff01d017f400: Mar 17 15:28:17 zippy genunix: [ID 335743 kern.notice] BAD TRAP: type=e (#pf Page fault) rp=ff0008925c40 addr=f701e470feb0 occurred in module genunix due to an illegal access to a user address Mar 17 15:28:17 zippy unix: [ID 10 kern.notice] Mar 17 15:28:17 zippy unix: [ID 839527 kern.notice] automountd: Mar 17 15:28:17 zippy unix: [ID 753105 kern.notice] #pf Page fault Mar 17 15:28:17 zippy unix: [ID 532287 kern.notice] Bad kernel fault at addr=0xf701e470feb0 Mar 17 15:28:17 zippy unix: [ID 243837 kern.notice] pid=598, pc=0xfba0f3e4, sp=0xff0008925d30, eflags=0x10297 Mar 17 15:28:17 zippy unix: [ID 211416 kern.notice] cr0: 8005003bpg,wp,ne,et,ts,mp,pe cr4: 6f8xmme,fxsr,pge,mce,pae,pse,de Mar 17 15:28:17 zippy unix: [ID 624947 kern.notice] cr2: f701e470feb0 Mar 17 15:28:17 zippy unix: [ID 625075 kern.notice] cr3: 13fefb000 Mar 17 15:28:17 zippy unix: [ID 625715 kern.notice] cr8: c Mar 17 15:28:17 zippy unix: [ID 10 kern.notice] Mar 17 15:28:17 zippy unix: [ID 592667 kern.notice] rdi: ff01cd1d5608 rsi:0 rdx: ff01d017f400 Mar 17 15:28:17 zippy unix: [ID 592667 kern.notice] rcx: 1 r8: ff01f4d0b040 r9: ff01f4da7c40 Mar 17 15:28:17 zippy unix: [ID 592667 kern.notice] rax: 0 rbx: ff01cd1d5608 rbp: ff0008925dc0 Mar 17 15:28:17 zippy unix: [ID 592667 kern.notice] r10: ff01cb227588 r11: ff01de01aa40 r12: ff01cd1d55f8 Mar 17 15:28:17 zippy unix: [ID 592667 kern.notice] r13: f701e470feb0 r14: ff01e7177ec0 r15: f701e470feb0 Mar 17 15:28:17 zippy unix: [ID 592667 kern.notice] fsb: 0 gsb: ff01cecc4080 ds: 4b Mar 17 15:28:17 zippy unix: [ID 592667 kern.notice] es: 4b fs:0 gs: 1c3 Mar 17 15:28:17 zippy unix: [ID 592667 kern.notice] trp: e err:0 rip: fba0f3e4 Mar 17 15:28:17 zippy unix: [ID 592667 kern.notice] cs: 30 rfl:10297 rsp: ff0008925d30 Mar 17 15:28:17 zippy unix: [ID 266532 kern.notice] ss: 38 Mar 17 15:28:17 zippy unix: [ID 10 kern.notice] Mar 17 15:28:17 zippy genunix: [ID 655072 kern.notice] ff0008925b20 unix:die+dd () Mar 17 15:28:17 zippy genunix: [ID 655072 kern.notice] ff0008925c30 unix:trap+1799 () Mar 17 15:28:17 zippy genunix: [ID 655072 kern.notice] ff0008925c40 unix:cmntrap+e6 () Mar 17 15:28:17 zippy genunix: [ID 655072 kern.notice] ff0008925dc0 genunix:dnlc_purge_vfsp+8c () Mar 17 15:28:17 zippy genunix: [ID 655072 kern.notice] ff0008925e10 genunix:dounmount+35 () Mar 17 15:28:17 zippy genunix: [ID 655072 kern.notice] ff0008925e60 genunix:umount2_engine+5c () Mar 17 15:28:17 zippy genunix: [ID 655072 kern.notice] ff0008925ec0 genunix:umount2+145 () Mar 17 15:28:17 zippy genunix: [ID 655072 kern.notice] ff0008925f10 unix:brand_sys_syscall32+17a () Mar 17 15:28:17 zippy unix: [ID 10 kern.notice] Mar 17 15:28:17 zippy genunix: [ID 672855 kern.notice] syncing file systems... Mar 17 15:28:17 zippy genunix: [ID 904073 kern.notice] done Mar 17 15:28:18 zippy genunix: [ID 111219 kern.notice] dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel Mar 17 15:28:18 zippy nv_sata: [ID 517869 kern.info] NOTICE: inst0 port0: nv_reset: reason: sata_reset serr 0x0 nvp_state: 0x8 Mar 17 15:28:18 zippy nv_sata: [ID 517869 kern.info] NOTICE: inst0 port1: nv_reset: reason: sata_reset serr 0x0 nvp_state: 0x8 Mar 17 15:28:48 zippy genunix: [ID 10 kern.notice] Mar 17 15:28:48 zippy genunix: [ID 665016 kern.notice] ^M100% done: 263844 pages dumped, Mar 17 15:28:48 zippy genunix: [ID 851671 kern.notice] dump succeeded Mar 17 15:28:51 zippy genunix: [ID 540533 kern.notice] ^MSunOS Release 5.11 Version snv_151a 64-bit Mar 17 15:28:51 zippy genunix: [ID 877030 kern.notice] Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved. Can anyone help? Regards Karl ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] FW: Solaris panic
On 03/18/11 04:46 AM, Karl Wagner wrote: Hi all I have only just seen this, and thought someone may be able to help. On heavy IO activity, my Solaris 11 Express box hosting a ZFS data pool crashes. It seems to show page faults in several things, including nfsd, sched, zpool-tank and automountd. If you see apparently random crashes, a hardware problem is the likely cause. Run a full diagnostic on the system's memory. -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] FW: Solaris panic
On Thu, Mar 17, 2011 at 10:46 AM, Karl Wagner k...@mouse-hole.com wrote: Hi all I have only just seen this, and thought someone may be able to help. On heavy IO activity, my Solaris 11 Express box hosting a ZFS data pool crashes. It seems to show page faults in several things, including nfsd, sched, zpool-tank and automountd. The stack trace in your panic looks a lot like that found in 6935256 Corrupted DNLC caused panic in dnlc_purge_vfsp() http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6935256 It's marked as a dup of another bug that was fixed before Solaris 11 Express. In a few seconds of looking at 6935256 and 6923355, I don't understand how they are related. Assuming you have a support contract, I would advise opening a support case to get someone to look into it. I get the following in the logs: Mar 17 15:28:17 zippy unix: [ID 836849 kern.notice] Mar 17 15:28:17 zippy ^Mpanic[cpu1]/thread=ff01d017f400: Mar 17 15:28:17 zippy genunix: [ID 335743 kern.notice] BAD TRAP: type=e (#pf Page fault) rp=ff0008925c40 addr=f701e470feb0 occurred in module genunix due to an illegal access to a user address Mar 17 15:28:17 zippy unix: [ID 10 kern.notice] Mar 17 15:28:17 zippy unix: [ID 839527 kern.notice] automountd: Mar 17 15:28:17 zippy unix: [ID 753105 kern.notice] #pf Page fault Mar 17 15:28:17 zippy unix: [ID 532287 kern.notice] Bad kernel fault at addr=0xf701e470feb0 Mar 17 15:28:17 zippy unix: [ID 243837 kern.notice] pid=598, pc=0xfba0f3e4, sp=0xff0008925d30, eflags=0x10297 Mar 17 15:28:17 zippy unix: [ID 211416 kern.notice] cr0: 8005003bpg,wp,ne,et,ts,mp,pe cr4: 6f8xmme,fxsr,pge,mce,pae,pse,de Mar 17 15:28:17 zippy unix: [ID 624947 kern.notice] cr2: f701e470feb0 Mar 17 15:28:17 zippy unix: [ID 625075 kern.notice] cr3: 13fefb000 Mar 17 15:28:17 zippy unix: [ID 625715 kern.notice] cr8: c Mar 17 15:28:17 zippy unix: [ID 10 kern.notice] Mar 17 15:28:17 zippy unix: [ID 592667 kern.notice] rdi: ff01cd1d5608 rsi: 0 rdx: ff01d017f400 Mar 17 15:28:17 zippy unix: [ID 592667 kern.notice] rcx: 1 r8: ff01f4d0b040 r9: ff01f4da7c40 Mar 17 15:28:17 zippy unix: [ID 592667 kern.notice] rax: 0 rbx: ff01cd1d5608 rbp: ff0008925dc0 Mar 17 15:28:17 zippy unix: [ID 592667 kern.notice] r10: ff01cb227588 r11: ff01de01aa40 r12: ff01cd1d55f8 Mar 17 15:28:17 zippy unix: [ID 592667 kern.notice] r13: f701e470feb0 r14: ff01e7177ec0 r15: f701e470feb0 Mar 17 15:28:17 zippy unix: [ID 592667 kern.notice] fsb: 0 gsb: ff01cecc4080 ds: 4b Mar 17 15:28:17 zippy unix: [ID 592667 kern.notice] es: 4b fs: 0 gs: 1c3 Mar 17 15:28:17 zippy unix: [ID 592667 kern.notice] trp: e err: 0 rip: fba0f3e4 Mar 17 15:28:17 zippy unix: [ID 592667 kern.notice] cs: 30 rfl: 10297 rsp: ff0008925d30 Mar 17 15:28:17 zippy unix: [ID 266532 kern.notice] ss: 38 Mar 17 15:28:17 zippy unix: [ID 10 kern.notice] Mar 17 15:28:17 zippy genunix: [ID 655072 kern.notice] ff0008925b20 unix:die+dd () Mar 17 15:28:17 zippy genunix: [ID 655072 kern.notice] ff0008925c30 unix:trap+1799 () Mar 17 15:28:17 zippy genunix: [ID 655072 kern.notice] ff0008925c40 unix:cmntrap+e6 () Mar 17 15:28:17 zippy genunix: [ID 655072 kern.notice] ff0008925dc0 genunix:dnlc_purge_vfsp+8c () Mar 17 15:28:17 zippy genunix: [ID 655072 kern.notice] ff0008925e10 genunix:dounmount+35 () Mar 17 15:28:17 zippy genunix: [ID 655072 kern.notice] ff0008925e60 genunix:umount2_engine+5c () Mar 17 15:28:17 zippy genunix: [ID 655072 kern.notice] ff0008925ec0 genunix:umount2+145 () Mar 17 15:28:17 zippy genunix: [ID 655072 kern.notice] ff0008925f10 unix:brand_sys_syscall32+17a () Mar 17 15:28:17 zippy unix: [ID 10 kern.notice] Mar 17 15:28:17 zippy genunix: [ID 672855 kern.notice] syncing file systems... Mar 17 15:28:17 zippy genunix: [ID 904073 kern.notice] done Mar 17 15:28:18 zippy genunix: [ID 111219 kern.notice] dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel Mar 17 15:28:18 zippy nv_sata: [ID 517869 kern.info] NOTICE: inst0 port0: nv_reset: reason: sata_reset serr 0x0 nvp_state: 0x8 Mar 17 15:28:18 zippy nv_sata: [ID 517869 kern.info] NOTICE: inst0 port1: nv_reset: reason: sata_reset serr 0x0 nvp_state: 0x8 Mar 17 15:28:48 zippy genunix: [ID 10 kern.notice] Mar 17 15:28:48 zippy genunix: [ID 665016 kern.notice] ^M100% done: 263844 pages dumped, Mar 17 15:28:48 zippy genunix: [ID 851671 kern.notice] dump succeeded Mar 17 15:28:51 zippy genunix: [ID 540533 kern.notice] ^MSunOS Release 5.11 Version snv_151a 64-bit Mar 17 15:28:51 zippy genunix: [ID 877030 kern.notice] Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved. Can anyone help? Regards Karl ___
