My pool panic'd while updating to Lucid Lynx hosted inside an iSCSI LUN. And
now it won't come back up. I have dedup and compression on.
These are my current findings:
* iostat -En won't list 8 of my disks
* zdb lists all my disks except my cache device
* The following commands panics the box in single-user mode: format, zfs, zpool
and zdb -l. Multi-user panics before reading ZFS config.
* Unplugging all devices belonging to the pool brings up the host to multi-user
mode and lists my pool as UNAVAIL.
I've scavenged the net for extracting useful information that might be of use.
I suspect it has something to do with the DDT table.
Best Regards
Michael
zdb output:
rpool:
version: 22
name: 'rpool'
state: 0
txg: 10643295
pool_guid: 16751367988873007995
hostid: 13336047
hostname: ''
vdev_children: 1
vdev_tree:
type: 'root'
id: 0
guid: 16751367988873007995
children[0]:
type: 'mirror'
id: 0
guid: 6639969804249231424
whole_disk: 0
metaslab_array: 23
metaslab_shift: 31
ashift: 9
asize: 250956742656
is_log: 0
children[0]:
type: 'disk'
id: 0
guid: 14476065696483338328
path: '/dev/dsk/c14d0s0'
devid: 'id1,c...@awdc_wd2500yd-01nvb1=_____wd-wcank4006148/a'
phys_path:
'/p...@0,0/pci10de,7...@8/pci-...@9/i...@0/c...@0,0:a'
whole_disk: 0
DTL: 78
children[1]:
type: 'disk'
id: 1
guid: 10422182008705867883
path: '/dev/dsk/c16d0s0'
devid: 'id1,c...@awdc_wd2500yd-01nvb1=_____wd-wcank5135915/a'
phys_path:
'/p...@0,0/pci10de,7...@8/pci-...@9/i...@1/c...@0,0:a'
whole_disk: 0
DTL: 173
tank:
version: 22
name: 'tank'
state: 0
txg: 36636297
pool_guid: 10904371515657913150
hostid: 13336047
hostname: 'zen'
vdev_children: 3
vdev_tree:
type: 'root'
id: 0
guid: 10904371515657913150
children[0]:
type: 'raidz'
id: 0
guid: 4940983256616168565
nparity: 1
metaslab_array: 23
metaslab_shift: 32
ashift: 9
asize: 2560443285504
is_log: 0
children[0]:
type: 'disk'
id: 0
guid: 7633768960477747795
path: '/dev/dsk/c13t4d0s0'
devid:
'id1,s...@sata_____wdc_wd6400aacs-0_____wd-wcauf0933938/a'
phys_path:
'/p...@0,0/pci10de,7...@13/pci1033,1...@0/pci11ab,1...@1/d...@4,0:a'
whole_disk: 1
DTL: 4268
children[1]:
type: 'disk'
id: 1
guid: 12141479741527311128
path: '/dev/dsk/c13t5d0s0'
devid:
'id1,s...@sata_____wdc_wd6400aacs-0_____wd-wcauf0934597/a'
phys_path:
'/p...@0,0/pci10de,7...@13/pci1033,1...@0/pci11ab,1...@1/d...@5,0:a'
whole_disk: 1
DTL: 4267
children[2]:
type: 'disk'
id: 2
guid: 7952488001712683172
path: '/dev/dsk/c13t6d0s0'
devid:
'id1,s...@sata_____wdc_wd6400aacs-0_____wd-wcauf0934679/a'
phys_path:
'/p...@0,0/pci10de,7...@13/pci1033,1...@0/pci11ab,1...@1/d...@6,0:a'
whole_disk: 1
DTL: 4266
children[3]:
type: 'disk'
id: 3
guid: 535039729687145914
path: '/dev/dsk/c13t7d0s0'
devid:
'id1,s...@sata_____wdc_wd6400aacs-0_____wd-wcauf0931654/a'
phys_path:
'/p...@0,0/pci10de,7...@13/pci1033,1...@0/pci11ab,1...@1/d...@7,0:a'
whole_disk: 1
DTL: 4265
children[1]:
type: 'raidz'
id: 1
guid: 6936009139020911476
nparity: 1
metaslab_array: 4097
metaslab_shift: 34
ashift: 9
asize: 2000373678080
is_log: 0
children[0]:
type: 'disk'
id: 0
guid: 4043674464412192471
path: '/dev/dsk/c13t3d0s0'
devid:
'id1,s...@sata_____samsung_hd103si_______s1vsj90sc22045/a'
phys_path:
'/p...@0,0/pci10de,7...@13/pci1033,1...@0/pci11ab,1...@1/d...@3,0:a'
whole_disk: 1
DTL: 8198
children[1]:
type: 'disk'
id: 1
guid: 7230587084054299877
path: '/dev/dsk/c13t1d0s0'
devid:
'id1,s...@sata_____wdc_wd5001aals-0_____wd-wmasy3260051/a'
phys_path:
'/p...@0,0/pci10de,7...@13/pci1033,1...@0/pci11ab,1...@1/d...@1,0:a'
whole_disk: 1
DTL: 4263
children[2]:
type: 'disk'
id: 2
guid: 10560603583403897619
path: '/dev/dsk/c13t2d0s0'
devid:
'id1,s...@sata_____samsung_hd103si_______s1vsj90sc22634/a'
phys_path:
'/p...@0,0/pci10de,7...@13/pci1033,1...@0/pci11ab,1...@1/d...@2,0:a'
whole_disk: 1
DTL: 12327
children[3]:
type: 'disk'
id: 3
guid: 1310727864203033402
path: '/dev/dsk/c13t0d0s0'
devid:
'id1,s...@sata_____wdc_wd5001aals-0_____wd-wmasy3508706/a'
phys_path:
'/p...@0,0/pci10de,7...@13/pci1033,1...@0/pci11ab,1...@1/d...@0,0:a'
whole_disk: 1
DTL: 4261
children[2]:
type: 'disk'
id: 2
guid: 14323860655899304907
path: '/dev/dsk/c8t0d0s0'
devid: 'id1,s...@sata_____intel_ssdsa2m080__cvpo003401vt080bgn/a'
phys_path: '/p...@0,0/pci1043,8...@9/d...@0,0:a'
whole_disk: 1
metaslab_array: 933
metaslab_shift: 29
ashift: 9
asize: 80012902400
is_log: 1
DTL: 12330
create_txg: 36514714
Kernel debug output: (Raw typescript, sorry)
Script started on May 4, 2010 06:22:58 PM CEST
r...@zen:~/coredir/foo# mdb -k unix.0 vmcore.0
(B)0Loading modules: [ unix genunix specfs mac cpu.generic uppc pcplusmp
scsi_vhci zfs sata sd sockfs ip hook neti sctp arp usba uhci s1394 qlc fctl
stmf md lofs ]
> ::stt atus
debugging crash dump vmcore.0 (64-bit) from zen
operating system: 5.11 snv_134 (i86pc)
panic message:
BAD TRAP: type=e (#pf Page fault) rp=ffffff000fd16950 addr=30 occurred in module
"zfs" due to a NULL pointer dereference
dump content: kernel pages only
> stack ::stack
ddt_phys_decref+0xc(0)
zio_ddt_free+0x55(ffffff02d9d1d660)
zio_execute+0x8d(ffffff02d9d1d660)
taskq_thread+0x248(ffffff02c97eb368)
thread_start+8()
> ::msgbuf
[4mMESSAGE [m
48-bit LBA, DMA, Native Command Queueing, SMART, SMART self-test
SATA Gen2 signaling speed (3.0Gbps)
Supported queue depth 32
capacity = 1250263728 sectors
sd17 at marvell88sx0: target 4 lun 0
sd17 is /p...@0,0/pci10de,7...@13/pci1033,1...@0/pci11ab,1...@1/d...@4,0
/p...@0,0/pci10de,7...@13/pci1033,1...@0/pci11ab,1...@1/d...@4,0 (sd17) online
/p...@0,0/pci10de,7...@13/pci1033,1...@0/pci11ab,1...@1 :
SATA disk device at port 5
model WDC WD6400AACS-00G8B0
firmware 05.04C05
serial number WD-WCAUF0934597
supported features:
48-bit LBA, DMA, Native Command Queueing, SMART, SMART self-test
SATA Gen2 signaling speed (3.0Gbps)
Supported queue depth 32
capacity = 1250263728 sectors
sd18 at marvell88sx0: target 5 lun 0
sd18 is /p...@0,0/pci10de,7...@13/pci1033,1...@0/pci11ab,1...@1/d...@5,0
/p...@0,0/pci10de,7...@13/pci1033,1...@0/pci11ab,1...@1/d...@5,0 (sd18) online
/p...@0,0/pci10de,7...@13/pci1033,1...@0/pci11ab,1...@1 :
SATA disk device at port 6
model WDC WD6400AACS-00G8B0
>> More [<space>, <cr>, q, n, c, a] ?
>>
>> firmware 05.04C05
serial number WD-WCAUF0934679
supported features:
48-bit LBA, DMA, Native Command Queueing, SMART, SMART self-test
SATA Gen2 signaling speed (3.0Gbps)
Supported queue depth 32
capacity = 1250263728 sectors
sd19 at marvell88sx0: target 6 lun 0
sd19 is /p...@0,0/pci10de,7...@13/pci1033,1...@0/pci11ab,1...@1/d...@6,0
/p...@0,0/pci10de,7...@13/pci1033,1...@0/pci11ab,1...@1/d...@6,0 (sd19) online
/p...@0,0/pci10de,7...@13/pci1033,1...@0/pci11ab,1...@1 :
SATA disk device at port 7
model WDC WD6400AACS-00G8B0
firmware 05.04C05
serial number WD-WCAUF0931654
supported features:
48-bit LBA, DMA, Native Command Queueing, SMART, SMART self-test
SATA Gen2 signaling speed (3.0Gbps)
Supported queue depth 32
capacity = 1250263728 sectors
sd20 at marvell88sx0: target 7 lun 0
sd20 is /p...@0,0/pci10de,7...@13/pci1033,1...@0/pci11ab,1...@1/d...@7,0
/p...@0,0/pci10de,7...@13/pci1033,1...@0/pci11ab,1...@1/d...@7,0 (sd20) online
/p...@0,0/pci1043,8...@4,1/h...@2/dev...@2/keybo...@0 (hid6) offline
>> More [<space>, <cr>, q, n, c, a] ?
>>
>> /p...@0,0/pci1043,8...@4,1/h...@2/dev...@2/in...@1
>> (hid7) offline
/p...@0,0/pci1043,8...@4,1/h...@2/dev...@2/keybo...@0 (hid6) offline
/p...@0,0/pci1043,8...@4,1/h...@2/dev...@2/in...@1 (hid7) offline
/p...@0,0/pci1043,8...@4,1/h...@2/mo...@1 (hid5) removed
/p...@0,0/pci1043,8...@4,1/h...@2/dev...@2 (usb_mid2) removed
/p...@0,0/pci1043,8...@4,1/h...@2 (hubd1) removed
USB 1.10 device (usb557,2404) operating at low speed (USB 1.x) on USB 1.10 root
hub: dev...@2, usb_mid1 at bus address 3
ATEN USB 2.0 Switch (4-port)
usb_mid1 is /p...@0,0/pci1043,8...@4/dev...@2
/p...@0,0/pci1043,8...@4/dev...@2 (usb_mid1) online
USB 1.10 interface (usbif557,2404.config1.0) operating at low speed (USB 1.x) on
USB 1.10 root hub: in...@0, hid3 at bus address 3
ATEN USB 2.0 Switch (4-port)
hid3 is /p...@0,0/pci1043,8...@4/dev...@2/in...@0
/p...@0,0/pci1043,8...@4/dev...@2/in...@0 (hid3) online
USB 1.10 interface (usbif557,2404.config1.1) operating at low speed (USB 1.x) on
USB 1.10 root hub: in...@1, hid4 at bus address 3
ATEN USB 2.0 Switch (4-port)
hid4 is /p...@0,0/pci1043,8...@4/dev...@2/in...@1
/p...@0,0/pci1043,8...@4/dev...@2/in...@1 (hid4) online
/p...@0,0/pci1043,8...@4/dev...@2/in...@0 (hid3) offline
/p...@0,0/pci1043,8...@4/dev...@2/in...@1 (hid4) offline
/p...@0,0/pci1043,8...@4/dev...@2/in...@0 (hid3) offline
>> More [<space>, <cr>, q, n, c, a] ?
>>
>> /p...@0,0/pci1043,8...@4/dev...@2/in...@1
>> (hid4) offline
/p...@0,0/pci1043,8...@4/dev...@2 (usb_mid1) removed
USB 2.0 device (usb424,2514) operating at hi speed (USB 2.x) on USB 2.0 root hub
: h...@2, hubd1 at bus address 2
hubd1 is /p...@0,0/pci1043,8...@4,1/h...@2
/p...@0,0/pci1043,8...@4,1/h...@2 (hubd1) online
USB 2.0 device (usb46d,c025) operating at low speed (USB 1.x) on USB 2.0 externa
l hub: mo...@1, hid5 at bus address 3
B16_b_02 USB-PS/2 Optical Mouse
hid5 is /p...@0,0/pci1043,8...@4,1/h...@2/mo...@1
/p...@0,0/pci1043,8...@4,1/h...@2/mo...@1 (hid5) online
USB 1.10 device (usb46d,c30e) operating at low speed (USB 1.x) on USB 2.0 extern
al hub: dev...@2, usb_mid2 at bus address 4
Logitech HID compliant keyboard
usb_mid2 is /p...@0,0/pci1043,8...@4,1/h...@2/dev...@2
/p...@0,0/pci1043,8...@4,1/h...@2/dev...@2 (usb_mid2) online
USB 1.10 interface (usbif46d,c30e.config1.0) operating at low speed (USB 1.x) on
USB 2.0 external hub: keybo...@0, hid6 at bus address 4
Logitech HID compliant keyboard
hid6 is /p...@0,0/pci1043,8...@4,1/h...@2/dev...@2/keybo...@0
/p...@0,0/pci1043,8...@4,1/h...@2/dev...@2/keybo...@0 (hid6) online
USB 1.10 interface (usbif46d,c30e.config1.1) operating at low speed (USB 1.x) on
USB 2.0 external hub: in...@1, hid7 at bus address 4
Logitech HID compliant keyboard
>> More [<space>, <cr>, q, n, c, a] ?
>> hid7
>> is /p...@0,0/pci1043,8...@4,1/h...@2/dev...@2/in...@1
/p...@0,0/pci1043,8...@4,1/h...@2/dev...@2/in...@1 (hid7) online
panic[cpu1]/thread=ffffff000fd16c60:
BAD TRAP: type=e (#pf Page fault) rp=ffffff000fd16950 addr=30 occurred in module
"zfs" due to a NULL pointer dereference
zpool-tank:
#pf Page fault
Bad kernel fault at addr=0x30
pid=225, pc=0xfffffffff795abe4, sp=0xffffff000fd16a48, eflags=0x10296
cr0: 8005003b<pg,wp,ne,et,ts,mp,pe> cr4: 6f8<xmme,fxsr,pge,mce,pae,pse,de>
cr2: 30
cr3: 4000000
cr8: c
rdi: 0 rsi: ffffff02d9d1d6c0 rdx: ffffffffffffffff
rcx: 144 r8: 70fb497da6 r9: 3ba11c96
rax: 0 rbx: 200 rbp: ffffff000fd16a50
r10: ffffff02dd30a0d0 r11: ffffff02dd30a098 r12: ffffff02d9d1d6c0
r13: ffffff02dd308000 r14: ffffff02c97eb388 r15: ffffff02c97eb390
fsb: 0 gsb: ffffff02c874c080 ds: 4b
es: 4b fs: 0 gs: 1c3
>> More [<space>, <cr>, q, n, c, a] ?
>>
>> trp: e err: 2 rip: fffffffff795abe4
cs: 30 rfl: 10296 rsp: ffffff000fd16a48
ss: 38
ffffff000fd16830 unix:die+dd ()
ffffff000fd16940 unix:trap+177b ()
ffffff000fd16950 unix:cmntrap+e6 ()
ffffff000fd16a50 zfs:ddt_phys_decref+c ()
ffffff000fd16a80 zfs:zio_ddt_free+55 ()
ffffff000fd16ab0 zfs:zio_execute+8d ()
ffffff000fd16b50 genunix:taskq_thread+248 ()
ffffff000fd16b60 unix:thread_start+8 ()
syncing file systems...
done
dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel
> ::panicinfo
cpu 1
thread ffffff000fd16c60
message
BAD TRAP: type=e (#pf Page fault) rp=ffffff000fd16950 addr=30 occurred in module
"zfs" due to a NULL pointer dereference
rdi 0
rsi ffffff02d9d1d6c0
rdx ffffffffffffffff
rcx 144
r8 70fb497da6
r9 3ba11c96
rax 0
rbx 200
rbp ffffff000fd16a50
r10 ffffff02dd30a0d0
r10 ffffff02dd30a0d0
r11 ffffff02dd30a098
r12 ffffff02d9d1d6c0
r13 ffffff02dd308000
r14 ffffff02c97eb388
r15 ffffff02c97eb390
fsbase 0
gsbase ffffff02c874c080
ds 4b
>> More [<space>, <cr>, q, n, c, a] ?
>>
>> es 4b
fs 0
gs 1c3
trapno e
err 2
rip fffffffff795abe4
cs 30
rflags 10296
rsp ffffff000fd16a48
ss 38
gdt_hi 0
gdt_lo 1ef
idt_hi 0
idt_lo d0000fff
ldt 0
task 70
cr0 8005003b
cr2 30
cr3 4000000
cr4 6f8
> :: ps -z
[4mS PID PPID PGID SID UID FLAGS ADDR NAME[m
R 0 0 0 0 0 0x00000001 fffffffffbc2dbb0 sched
R 225 0 0 0 0 0x00020001 ffffff02c6e4ac70 zpool-tank
R 3 0 0 0 0 0x00020001 ffffff02c6e4de10 fsflush
R 2 0 0 0 0 0x00020001 ffffff02c6e4ea78 pageout
R 1 0 0 0 0 0x4a004000 ffffff02c6e4f6e0 init
R 224 1 224 224 0 0x42000000 ffffff02d4b116f0 syseventconfd
R 233 224 224 224 0 0x4a004000 ffffff02d897fe28 zfsdle
R 232 224 224 224 0 0x4a004000 ffffff02d8980a90 zfsdle
R 231 224 224 224 0 0x4a004000 ffffff02d89816f8 zfsdle
R 230 224 224 224 0 0x4a004000 ffffff02c9d0c010 zfsdle
R 229 224 224 224 0 0x4a004000 ffffff02c9d10a80 zfsdle
R 228 224 224 224 0 0x4a004000 ffffff02c9d0fe18 zfsdle
R 227 224 224 224 0 0x4a004000 ffffff02d4b0cc80 zfsdle
R 226 224 224 224 0 0x4a004000 ffffff02d4b0c018 zfsdle
R 136 1 136 136 0 0x42000000 ffffff02c9d0e548 rcm_daemon
R 134 1 134 134 0 0x42000000 ffffff02c9d12350 devfsadm
R 111 1 111 111 0 0x42010000 ffffff02d4b12358 syseventd
R 76 1 76 76 1 0x42000000 ffffff02c9d0d8e0 kcfd
R 16 1 16 16 15 0x52000000 ffffff02c9d116e8 dlmgmtd
R 11 1 11 11 0 0x42000000 ffffff02c6e4d1a8 svc.configd
R 9 1 9 9 0 0x42000000 ffffff02c6e4c540 svc.startd
R 197 9 197 197 0 0x4a014000 ffffff02d4b0d8e8 bash
R 5 0 0 0 0 0x00020001 ffffff02c6e50348 zpool-rpool
> ::quit
[mr...@zen:~/coredir/foo# ls
debug.txt unix.0 vmcore.0
script done on May 4, 2010 06:26:46 PM CEST
--
This message posted from opensolaris.org
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
