For a small amount of background, I’ve been trying to help nail down some of the reported problems from users trying out the zfs crypto PR on illumos distros. I have one instance here I’ve been digging into, and have some questions I’m hoping those who’ve spent far more time in this code than I have (so far) might be able to shed some insight.
Specifically, so far it appears there is something surrounding zfs send/recv that is causing the checksums to become corrupt. I’m still digging into this further, but the short version is, multiple zfs scrubs on the source show no issues, but when sending an incremental stream from an encrypted zvol to another system is fairly easily cause a panic during reception: panic[cpu1]/thread=fffffe000fe5cc20: assertion failed: (db = dbuf_hold(dn, blkid, FTAG)) != NULL, file: ../../common/fs/zfs/dmu.c, line: 1706 fffffe000fe5caa0 genunix:process_type+16c14c () fffffe000fe5cb10 zfs:dmu_assign_arcbuf_dnode+198 () fffffe000fe5cb80 zfs:receive_write+140 () fffffe000fe5cbc0 zfs:receive_process_record+123 () fffffe000fe5cc00 zfs:receive_writer_thread+88 () fffffe000fe5cc10 unix:thread_start+8 () While not obvious from that panic message (it took a bit of dtrace and a few laps), that panic is because of the following scenario: dmu_assign_arcbuf_dnode dbuf_hold (returns NULL — as seen in the VERIFY violation) dbuf_hold_level (returns EIO) dbuf_hold_impl (returns EIO) dbuf_hold_impl (returns EIO) dbuf_hold_impl (returns EIO) dbuf_hold_impl (returns EIO) dbuf_findbp (returns EIO) dbuf_read (returns EIO) dbuf_read_verify_dnode_crypt (returns EIO) arc_untransform (returns EIO) arc_buf_fill (returns ECKSUM) arc_fill_hdr_crypt (returns ECKSUM) Looking at the ZoL code , it would appear it would also panic if it tried to receive the same stream (the call stack for ZoL would be slightly different since the dbuf_hold args are heap allocated there instead of on the stack, but same general sequence). It seems like zfs recv should be more resilient here — the receive should just error out instead of leaving a corrupted zvol on the destination — zfs scrub on the _destination_ will show errors that it cannot fix, removing the last sent snapshot clears the errors (both systems have ECC ram, no errors there). Is this already a known issue? I didn’t see anything obvious that looked like this, but then I might not be looking in the correct places. I’m still digging more into the actual corruption itself, but I wanted to at least mention the above issue in the meantime. Of course if the zfs send/recv issue is ringing bells with anyone, I’d love to know that as well. Interestingly enough, it’s always the same blkid in the panic. ------------------------------------------ openzfs: openzfs-developer Permalink: https://openzfs.topicbox.com/groups/developer/Ta7b88998bf99fe05-M364b5079b4f0f5172d4c080c Delivery options: https://openzfs.topicbox.com/groups/developer/subscription
