Your analysis makes sense to me. Having two versions of the resume state (in the filesystem itself, and then also in the %recv clone) seems unnecessary and confusing.
If this was not a new filesystem, I think that the 2nd "zfs recv -F" would fail because there is resume state present (see first check in recv_begin_check_existing_impl()). You would need to "zfs recv -A" to discard it. It seems reasonable for receiving a new filesystem to work similarly, by would making the 2nd "zfs recv -F" fail with EBUSY due to the resumable state (i.e. the presence of DS_FIELD_RESUME_*). FWIW, another way to fix this would be to discard the resume state (just the DS_FIELD_RESUME_*) in the filesystem when we create a %recv with resume state. That way we return the on-disk state to something that could exist "normally" (outside of this issue). But I think that it would be preferable to have "zfs recv -F" fail when there is resumable state. --matt On Fri, Aug 23, 2019 at 7:36 AM Andriy Gapon <[email protected]> wrote: > > I have got a report, for FreeBSD, about a filesystem getting into a > weird state with respect to the resumable receive. > The filesystem has non-empty 'receive_resume_token' property. > But a send stream generated with that token cannot be received. > Also, the receive state cannot be cleared with zfs recv -A. > The filesystem's 'inconsistent' property is zero. > From the above I conclude that there is no %recv child. > > I was able to reproduce that condition -- again, on FreeBSD -- using the > following steps. > > - create a full send stream and make a truncated copy of it (e.g. 50%) > - zfs recv -F -s the truncated copy into a new destination dataset > - this creates the destination dataset > - marks the dataset inconsistent > - receives as much data as available in the truncated stream > - sets up a partial receive state for the dataset (in its ZAP) > - repeat the same command with the same truncated stream again > - because of -F, the command is allowed to proceed despite the > existing destination > - because the destination exists, the stream is received into %recv > - a partial receive state is set up on %recv > - create a resume stream with zfs send -t using the token from the > previous step > - zfs recv -F -s the resume stream > - this finds %recv and the resume state recorded in it > - this completes the receive into %recv > - the clone swap is done to move the received objset into the parent > dataset > - %recv is destroyed > - the parent dataset is marked consistent > > So, in the end we have a fully received dataset in a consistent state, > but it still has the partial receive state from the first receive. > > I think that maybe dmu_recv_end_sync() should clear the resume state on > origin_head in the !drc->drc_newfs case like it does for ds in the other > case. > > Or maybe, if we start a new full receive and the destination already > exists and it is inconsistent and it has the resume state, then we > should assert that the destination has no snapshots and then re-create > it, so that we can receive into it directly (without creating %recv). > I see two ways of re-creating the destination. Either, we can destroy > and create it. Or we can somehow discard all its data, e.g., roll back > to @$ORIGIN. > > Maybe there are other options. > > What do you think? > > Thank you! > > P.S. > I have not tried illumos or ZoL yet. > Could anyone please try to reproduce? > Thanks again. > > -- > Andriy Gapon > > ------------------------------------------ > openzfs: openzfs-developer > Permalink: > https://openzfs.topicbox.com/groups/developer/Tbb598d5972318c77-M2aaa54b879588765828c633d > Delivery options: > https://openzfs.topicbox.com/groups/developer/subscription > ------------------------------------------ openzfs: openzfs-developer Permalink: https://openzfs.topicbox.com/groups/developer/Tbb598d5972318c77-M856b51923a34571ca98d0837 Delivery options: https://openzfs.topicbox.com/groups/developer/subscription
