Per Alex suggestion to see where ZFS is at during the hang period: THREAD STATE SOBJ COUNT ffffff007a8b3c40 SLEEP CV 3 swtch+0x145 cv_timedwait_hires+0xe0 cv_timedwait+0x5a txg_thread_wait+0x7c txg_sync_thread+0x118 thread_start+8 ffffff007a292c40 SLEEP CV 3 swtch+0x145 cv_wait+0x61 spa_thread+0x225 thread_start+8 ffffff007a8aac40 SLEEP CV 3 swtch+0x145 cv_wait+0x61 txg_thread_wait+0x5f txg_quiesce_thread+0x94 thread_start+8 ffffff007a1bbc40 SLEEP CV 1 swtch+0x145 cv_timedwait_hires+0xe0 cv_timedwait+0x5a arc_reclaim_thread+0x13d thread_start+8 ffffff007a1c1c40 SLEEP CV 1 swtch+0x145 cv_timedwait_hires+0xe0 cv_timedwait+0x5a l2arc_feed_thread+0xa1 thread_start+8 ffffff11bde0f4a0 ONPROC <NONE> 1 mutex_exit dbuf_hold_impl+0x81 dnode_next_offset_level+0xee dnode_next_offset+0xa2 dmu_object_next+0x54 restore_freeobjects+0x7e dmu_recv_stream+0x7f1 zfs_ioc_recv+0x416 zfsdev_ioctl+0x347 cdev_ioctl+0x45 spec_ioctl+0x5a fop_ioctl+0x7b ioctl+0x18e _sys_sysenter_post_swapgs+0x149 echo "::stacks -m zfs" |mdb -k THREAD STATE SOBJ COUNT ffffff007a8b3c40 SLEEP CV 3 swtch+0x145 cv_timedwait_hires+0xe0 cv_timedwait+0x5a txg_thread_wait+0x7c txg_sync_thread+0x118 thread_start+8 ffffff007a292c40 SLEEP CV 3 swtch+0x145 cv_wait+0x61 spa_thread+0x225 thread_start+8 ffffff007a8aac40 SLEEP CV 3 swtch+0x145 cv_wait+0x61 txg_thread_wait+0x5f txg_quiesce_thread+0x94 thread_start+8 ffffff007a1bbc40 SLEEP CV 1 swtch+0x145 cv_timedwait_hires+0xe0 cv_timedwait+0x5a arc_reclaim_thread+0x13d thread_start+8 ffffff007a1c1c40 SLEEP CV 1 swtch+0x145 cv_timedwait_hires+0xe0 cv_timedwait+0x5a l2arc_feed_thread+0xa1 thread_start+8 ffffff11bde0f4a0 ONPROC <NONE> 1 dbuf_hash+0xdc 0xffffff11ca05c460 dbuf_hold_impl+0x59 dnode_next_offset_level+0xee dnode_next_offset+0xa2 dmu_object_next+0x54 restore_freeobjects+0x7e dmu_recv_stream+0x7f1 zfs_ioc_recv+0x416 zfsdev_ioctl+0x347 cdev_ioctl+0x45 spec_ioctl+0x5a fop_ioctl+0x7b ioctl+0x18e _sys_sysenter_post_swapgs+0x149 Where the action was: zfs recv -v dpool01/wtf <test-15-out receiving full stream of dpool01/test@now into dpool01/wtf@now test-15-out is zfs send dpool01/test@now >test-15-out and then sent to the node It’s only about 48k in size; no filesystem data (though, problem exists when I have a filesystem with data). I’ve created a few identical filesystems on a few nodes and done some hex compares with them; but nothing extensive beyond “I differences”. Thanks Alex, Joe On 11/2/16, 11:15 AM, "Hetrick, Joseph P" <joseph-hetr...@uiowa.edu> wrote: Hi folks, We’ve run into an odd issue that seems concerning. Our shop runs OpenIndiana and we’ve got several versions in play. Recently while testing a new system which is much more recent (bleeding edge OI Hipster release) we discovered that zfs sends to older systems caused hangs. By older, we’re talking same zfs/zpool versions of 5/28 and no visible properties differences. Can provide more info if told what is useful; but the gist is that: Zfs send of a vanilla dataset (no properties defined other than defaults) to any “older” system causes the recv to hang, eventually the host will crash. Truss’ing the recvr process doesn’t seem to give a lot of info as to the cause. Filesystem snapshot is received; and then that’s it. No fancy send or recv args in play (zfs send dataset via netcat or mbuffer or ssh to a recv –v <dest>. A close comparision of zfs and pool properties shows no difference. On a whim we even created pools and datasets that were downversioned below the senders. We’ve seen this in hosts a bit later than: illumos-a7317ce but not before (and certainly a bit later); and where we are now: illumos-2816291. Oddly illumos-a7317ce systems appear to be able to receive these datasets just fine…and we’ve had no problems with systems of that vintage sending to older systems. Any ideas and instruction is most welcome, Joe
------------------------------------------- illumos-discuss Archives: https://www.listbox.com/member/archive/182180/=now RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175430-2e6923be Modify Your Subscription: https://www.listbox.com/member/?member_id=21175430&id_secret=21175430-6a77cda4 Powered by Listbox: http://www.listbox.com