I am debugging something similar and opened https://illumos.org/issues/6779 yesterday. It looks somewhat related, as it is also a panic in zap_leaf_lookup_closest(), though not during zfs_unlinked_drain() but from NFS. Second, I also encountered an all-zero zap_phys_t along the way.
I cannot do zdb inspection, as the corruption could be mitigated by slightly increasing the dataset's quota. You could try the same, depending on your goals. -Jan On 21.03.2016 19:14, Dan McDonald wrote: > >> On Mar 21, 2016, at 2:03 PM, Dan McDonald <dan...@omniti.com> wrote: >> >> I would appreciate any insight into this corruption. I'll be continuing to >> dig on my own. > > It appears the zap_phys_t is all zeroes. > > Using the zdb panic, I found this: > > # mdb core.zdb > Loading modules: [ libumem.so.1 libc.so.1 libzpool.so.1 libavl.so.1 > libcmdutils.so.1 libnvpair.so.1 libtopo.so.1 ld.so.1 ] >> $c > libc.so.1`_lwp_kill+0xa() > libc.so.1`_assfail+0x168(ffffdf7fffdfea20, ffffdf7ffda347b0, 241) > libc.so.1`assfail3+0xe6(ffffdf7ffda34e88, 0, ffffdf7ffda39f6c, 2f52ab2ab, > ffffdf7ffda347b0, 241) > libzpool.so.1`zap_deref_leaf+0x7b(3dceb500, 0, 0, 0, ffffdf7fffdff200) > libzpool.so.1`fzap_cursor_retrieve+0x140(3dceb500, ffffdf7fffdff1f0, > ffffdf7fffdff050) > libzpool.so.1`zap_cursor_retrieve+0x67(ffffdf7fffdff1f0, ffffdf7fffdff050) > dump_zpldir+0xc3(3904d300, e107ad5, 0, 0) > dump_object+0x31f(3904d300, e107ad5, 4, ffffdf7fffdff8bc) > dump_dir+0x1e8(3904d300) > main+0x4b9(4, ffffdf7fffdff9e8) > _start+0x6c() >> 3dceb500::print -ta zap_t > 3dceb500 zap_t { > 3dceb500 dmu_buf_user_t zap_dbu = { > 3dceb500 taskq_ent_t dbu_tqent = { > 3dceb500 struct taskq_ent *tqent_next = 0 > 3dceb508 struct taskq_ent *tqent_prev = 0 > 3dceb510 task_func_t *tqent_func = 0 > 3dceb518 void *tqent_arg = 0 > 3dceb520 uintptr_t tqent_flags = 0 > } > 3dceb528 dmu_buf_evict_func_t *dbu_evict_func = > libzpool.so.1`zap_evict > 3dceb530 dmu_buf_t **dbu_clear_on_evict_dbufp = 0x3dceb548 > } > 3dceb538 objset_t *zap_objset = 0x3904d300 > 3dceb540 uint64_t zap_object = 0xe107ad5 > 3dceb548 struct dmu_buf *zap_dbuf = 0x3de46460 > 3dceb550 krwlock_t zap_rwlock = { > 3dceb550 void *rw_owner = 1 > 3dceb558 boolean_t initialized = 0x1 (B_TRUE) > 3dceb560 rwlock_t rw_lock = { > 3dceb560 int32_t readers = 0x1 > 3dceb564 uint16_t type = 0 > 3dceb566 uint16_t magic = 0x5257 > 3dceb568 mutex_t mutex = { > <SNIP!> >> 0x3de46460::print "struct dmu_buf" > { > db_object = 0xe107ad5 > db_offset = 0 > db_size = 0x200 > db_data = 0x3de6d400 > } >> 0x3de6d400::print zap_phys_t > { > zap_block_type = 0 > zap_magic = 0 > zap_ptrtbl = { > zt_blk = 0 > zt_numblks = 0 > zt_shift = 0 > zt_nextblk = 0 > zt_blks_copied = 0 > } > zap_freeblk = 0 > zap_num_leafs = 0 > zap_num_entries = 0 > zap_salt = 0 > zap_normflags = 0 > zap_flags = 0 > } >> > > > Is there any way I can correct this? Or otherwise help my customer against > this clear on-disk corruption? > > Dan > ------------------------------------------- openzfs-developer Archives: https://www.listbox.com/member/archive/274414/=now RSS Feed: https://www.listbox.com/member/archive/rss/274414/28015062-cce53afa Modify Your Subscription: https://www.listbox.com/member/?member_id=28015062&id_secret=28015062-f966d51c Powered by Listbox: http://www.listbox.com