Hi Victor,
Thanks for the prompt reply. Here are the results from your suggestions.
> Panic stack would be useful.
I'm sorry I don't have this available and I don't want to cause another panic :)
>
> It is apparently blocked somewhere in kernel. Try to do something like this
> from another window to get better idea:
>
> echo "<pid of zpool>::pid2proc|::walk thread|::findtsack -v" | mdb -k
> echo "::threadlist -v" | mdb -k
>
window 1:
bash-3.2# zpool import -f data1
window 2:
bash-3.2# ps -fel | grep zpool
0 S root 897 874 0 40 20 ? 1262 ? 15:03:23
pts/3 0:00 zpool import -f data1
bash-3.2# echo "0t897::pid2proc|::walk thread|::findstack -v" | mdb -k
stack pointer for thread ffffff01b57ab840: ffffff00083709f0
[ ffffff00083709f0 _resume_from_idle+0xf1() ]
ffffff0008370a30 swtch+0x17f()
ffffff0008370a60 cv_wait+0x61(ffffff01b3bd71ca, ffffff01b3bd7188)
ffffff0008370ab0 txg_wait_synced+0x81(ffffff01b3bd7000, 299ee597)
ffffff0008370b10 spa_config_update_common+0x79(ffffff01b42a8a80, 0, 0)
ffffff0008370bb0 spa_import_common+0x36e(ffffff01b5ad4000, ffffff01b5325310, 0
, 0, 0)
ffffff0008370be0 spa_import+0x1e(ffffff01b5ad4000, ffffff01b5325310, 0)
ffffff0008370c30 zfs_ioc_pool_import+0xad(ffffff01b5ad4000)
ffffff0008370cb0 zfsdev_ioctl+0x10d(b600000000, 5a02, 80424f0, 100003,
ffffff01b3f181a0, ffffff0008370e9c)
ffffff0008370cf0 cdev_ioctl+0x48(b600000000, 5a02, 80424f0, 100003,
ffffff01b3f181a0, ffffff0008370e9c)
ffffff0008370d30 spec_ioctl+0x86(ffffff01adf41d00, 5a02, 80424f0, 100003,
ffffff01b3f181a0, ffffff0008370e9c, 0)
ffffff0008370db0 fop_ioctl+0x7b(ffffff01adf41d00, 5a02, 80424f0, 100003,
ffffff01b3f181a0, ffffff0008370e9c, 0)
ffffff0008370ec0 ioctl+0x174(3, 5a02, 80424f0)
ffffff0008370f10 _sys_sysenter_post_swapgs+0x14b()
bash-3.2# echo "::threadlist -v" | mdb -k
Output a bit too long to post here. Is there anything in particular i
should look for in this output?
>
> Some useful information may be logged in FMA, try to see what is available
> there with
>
> fmdump -eV
- I get a few ereport.fs.zfs.checksum reports like this
bash-3.2# fmdump -eV
Aug 22 2008 15:03:23.203687016 ereport.fs.zfs.checksum
nvlist version: 0
class = ereport.fs.zfs.checksum
ena = 0x1a77b8287a00001
detector = (embedded nvlist)
nvlist version: 0
version = 0x0
scheme = zfs
pool = 0xe2bba51ab8c26b53
vdev = 0x79871af1e1f39de1
(end detector)
pool = data1
pool_guid = 0xe2bba51ab8c26b53
pool_context = 0
pool_failmode = wait
vdev_guid = 0x79871af1e1f39de1
vdev_type = disk
vdev_path = /dev/dsk/c2t2d0s0
vdev_devid = id1,[EMAIL PROTECTED]/a
parent_guid = 0xe2bba51ab8c26b53
parent_type = root
zio_err = 50
zio_offset = 0x1e800416c00
zio_size = 0x400
zio_objset = 0x0
zio_object = 0xb
zio_level = 0
zio_blkid = 0x0
__ttl = 0x1
Aug 22 2008 15:03:23.203687247 ereport.fs.zfs.data
nvlist version: 0
class = ereport.fs.zfs.data
ena = 0x1a77b8287a00001
detector = (embedded nvlist)
nvlist version: 0
version = 0x0
scheme = zfs
pool = 0xe2bba51ab8c26b53
(end detector)
pool = data1
pool_guid = 0xe2bba51ab8c26b53
pool_context = 0
pool_failmode = wait
zio_err = 50
zio_objset = 0x0
zio_object = 0xb
zio_level = 0
zio_blkid = 0x0
__ttl = 0x1
__tod = 0x48aeb91b 0xc24054f
Aug 22 2008 15:03:23.207225717 ereport.fs.zfs.io_failure
nvlist version: 0
class = ereport.fs.zfs.io_failure
ena = 0x1a77ee27cb00001
detector = (embedded nvlist)
nvlist version: 0
version = 0x0
scheme = zfs
pool = 0xe2bba51ab8c26b53
(end detector)
pool = data1
pool_guid = 0xe2bba51ab8c26b53
pool_context = 0
pool_failmode = wait
__ttl = 0x1
__tod = 0x48aeb91b 0xc5a0375
>
> On Nevada you can try the following to (same option repeated several times
> increases verbosity):
>
> zdb -e -bb data1
> zdb -e -dddd data1
>
bash-3.2# zdb -e -bb data1
Traversing all blocks to verify nothing leaked ...
out of memory -- generating core dump
Abort
Seems i need to get a machine with more ram to do this :) This can be
arranged on monday.
bash-3.2# zdb -e -dddd data1
Dataset mos [META], ID 0, cr_txg 4, 210M, 189 objects, rootbp [L0 DMU
objset] 400L/200P DVA[0]=<0:56800000200:200> DVA[1]=<0:3000020200:200>
DVA[2]=<0:48800001800:200> fletcher4 lzjb LE contiguous
birth=698279317 fill=189
cksum=89744d6d8:36e7cf71f81:b1d06b2acd36:1850b4cc5621f3
Object lvl iblk dblk lsize asize type
0 2 16K 16K 96.0K 94.5K DMU dnode
Object lvl iblk dblk lsize asize type
1 1 16K 512 512 1K object directory
microzap: 512 bytes, 6 entries
sync_bplist = 12
config = 11
root_dataset = 2
errlog_scrub = 0
deflate = 1
errlog_last = 0
Object lvl iblk dblk lsize asize type
2 1 16K 512 512 0 DSL directory
256 bonus DSL directory
creation_time = Tue Mar 18 12:03:05 2008
head_dataset_obj = 5
parent_dir_obj = 0
origin_obj = 0
child_dir_zapobj = 4
used_bytes = 4.41T
compressed_bytes = 4.40T
uncompressed_bytes = 4.40T
quota = 0
reserved = 0
props_zapobj = 3
deleg_zapobj = 0
Object lvl iblk dblk lsize asize type
3 1 16K 16K 32K 7.00K DSL props
Fat ZAP stats:
Pointer table:
1024 elements
zt_blk: 0
zt_numblks: 0
zt_shift: 10
zt_blks_copied: 0
zt_nextblk: 0
ZAP entries: 3
Leaf blocks: 1
Total blocks: 2
zap_block_type: 0x8000000000000001
zap_magic: 0x2f52ab2ab
zap_salt: 0x1d0e3ebe7
Leafs with 2^n pointers:
9: 1 *
Blocks with n*5 entries:
0: 1 *
Blocks n/10 full:
1: 1 *
Entries with n chunks:
3: 3 ***
Buckets with n entries:
0: 509 ****************************************
1: 3 *
atime = 0
exec = 0
mountpoint = legacy
Object lvl iblk dblk lsize asize type
4 1 16K 512 512 1K DSL directory child map
microzap: 512 bytes, 1 entries
$MOS = 8
Object lvl iblk dblk lsize asize type
5 1 16K 512 512 0 DSL dataset
320 bonus DSL dataset
dir_obj = 2
prev_snap_obj = 0
prev_snap_txg = 0
next_snap_obj = 0
snapnames_zapobj = 6
num_children = 0
creation_time = Tue Mar 18 12:03:05 2008
creation_txg = 4
deadlist_obj = 7
used_bytes = 4.41T
compressed_bytes = 4.40T
uncompressed_bytes = 4.40T
unique = 4.41T
fsid_guid = 63153422916496006
guid = 3076616381815835747
flags = 0
next_clones_obj = 0
bp = [L0 DMU objset] 400L/200P DVA[0]=<0:56800000000:200>
DVA[1]=<0:3000020000:200> DVA[2]=<0:48800000a00:200> fletcher4 lzjb LE
contiguous birth=698279317 fill=6040564
cksum=f9e53509d:5f89f3d9404:12e97014634a8:294b54e7bf25c2
Object lvl iblk dblk lsize asize type
6 1 16K 512 512 1K DSL dataset snap map
microzap: 512 bytes, 0 entries
Object lvl iblk dblk lsize asize type
7 1 16K 128K 128K 0 bplist
32 bonus bplist header
Object lvl iblk dblk lsize asize type
8 1 16K 512 512 0 DSL directory
256 bonus DSL directory
creation_time = Tue Mar 18 12:03:05 2008
head_dataset_obj = 0
parent_dir_obj = 2
origin_obj = 0
child_dir_zapobj = 10
used_bytes = 210M
compressed_bytes = 104M
uncompressed_bytes = 104M
quota = 0
reserved = 0
props_zapobj = 9
deleg_zapobj = 0
Object lvl iblk dblk lsize asize type
9 1 16K 512 512 1K DSL props
microzap: 512 bytes, 0 entries
Object lvl iblk dblk lsize asize type
10 1 16K 512 512 1K DSL directory child map
microzap: 512 bytes, 0 entries
Object lvl iblk dblk lsize asize type
11 1 16K 16K 16K 2K packed nvlist
8 bonus packed nvlist size
Assertion failed: 0 == dmu_read(os, object, 0, nvsize, packed), file
../zdb.c, line 216, function dump_packed_nvlist
> Btw, why does timestamp on your uberblock show July 1?
Well, this is about the time when the crash happened. The clock on the
server is correct.
Once again, thank you for taking the time to look into this.
cheers
Erik
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
