[
https://issues.apache.org/jira/browse/HDDS-16755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16755:
------------------------------
Priority: Critical (was: Major)
> Uncommitted block cleanup can delete a block that a snapshot of an hsync'ed
> key still references
> ------------------------------------------------------------------------------------------------
>
> Key: HDDS-16755
> URL: https://issues.apache.org/jira/browse/HDDS-16755
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager, Snapshot
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Critical
> Labels: pull-request-available
>
> h3. Mechanism
> When a key commit leaves out a block that was allocated to the open key,
> {{OMKeyRequest.wrapUncommittedBlocksAsPseudoKey}}
> ({{OMKeyRequest.java:1227}}) wraps the block in a pseudo key with object ID
> {{OBJECT_ID_RECLAIM_BLOCKS}} (0) and the commit puts it in {{deletedTable}}.
> The marker means "reclaim unconditionally", on the assumption that such a
> block was never visible to a snapshot.
> That assumption does not hold for an hsync'ed key. An hsync publishes the
> block list to {{keyTable}}, so a snapshot taken after the hsync lists those
> blocks. If a later hsync or the final commit leaves one of them out, it is
> released as an uncommitted block when the key is closed.
> {{ReclaimableKeyFilter.getPreviousSnapshotKeyInfo}}
> ({{ReclaimableKeyFilter.java:166}}) treats an object ID mismatch with the
> previous snapshot's key as "not the same key" and returns reclaimable. The
> pseudo key always mismatches, so {{KeyDeletingService}} sends the block to
> SCM for deletion while a snapshot still references it.
> h3. Trigger
> # A writer hsyncs key {{k}} with blocks {{[A, B]}}.
> # Snapshot {{S1}} is created. {{S1}} lists {{k}} with {{[A, B]}}.
> # The writer allocates block {{C}} and commits {{k}} with {{[A, C]}}.
> # The commit finds {{B}} uncommitted and writes a {{deletedTable}} row for it
> with object ID 0.
> # {{KeyDeletingService}} reclaims the row. {{B}} is deleted from datanodes
> and reads of {{k}} through {{S1}} fail.
> Variant with two snapshots: between steps 2 and 3 the writer hsyncs {{[A,
> C]}} and snapshot {{S2}} is created. The previous snapshot {{S2}} then lists
> {{[A, C]}} and shares no block with the pseudo key, while {{S1}} still
> references {{B}}.
> The same helper is used by {{OMKeyCommitRequestWithFSO}} ({{:314}}) and
> {{S3MultipartUploadCommitPartRequest}} ({{:305}}).
> h3. Impact
> Data loss for snapshot reads. The active key is not affected.
> Reproduced on master with a {{TestKeyDeletingService}} case that issues
> {{openKey}}, {{allocateBlock}}, {{hsyncKey}}, {{createSnapshot}} and
> {{commitKey}} against OM directly. Whether an unmodified Ozone client can
> drop a block it has already hsync'ed has not been established, so this is
> currently an OM side hazard: OM accepts the commit and deletes snapshot data.
> h3. Proposed fix
> The pseudo key must keep both the object ID and the hsync metadata of the key.
> * Object ID: stop overwriting it with 0, so the filter finds the key's
> version in the previous snapshot.
> * Hsync metadata: the pseudo key is already built before {{HSYNC_CLIENT_ID}}
> is removed ({{OMKeyCommitRequest.java:380}} and {{:394}},
> {{OMKeyCommitRequestWithFSO.java:314}} and {{:338}}). That ordering must be
> kept and documented at the helper.
> With both, {{SnapshotUtils.isBlockLocationInfoSame}}
> ({{SnapshotUtils.java:303}}) retains the row while the previous snapshot
> holds an hsync'ed version of the key with the same object ID. This also
> covers the two snapshot variant, where the previous snapshot does not list
> the block itself. The row becomes reclaimable once no previous snapshot holds
> an hsync'ed version of the key.
> The {{deletedTable}} row name is derived from the transaction index, not the
> object ID, so rows do not collide.
> Known side effects:
> * While the previous snapshot holds an hsync'ed version of the key, the whole
> pseudo row is retained, including blocks no snapshot references.
> * A retained pseudo row adds the snapshot key's size to that snapshot's
> exclusive size even though the active key still shares blocks with it.
> The then unused {{OBJECT_ID_RECLAIM_BLOCKS}} constant and its exemption in
> {{WithObjectID.validate}} can be removed in a follow up.
> h3. Test
> * New
> {{TestKeyDeletingService.testUncommittedBlockOfHsyncedKeyRetainedBySnapshot}}
> runs the two snapshot variant. The {{deletedTable}} row for {{B}} must stay
> while {{S2}} is the previous snapshot and after {{S2}} is deleted, and must
> be purged only after {{S1}} is deleted.
> *
> {{TestOMKeyCommitRequest.testValidateAndUpdateCacheOnOverwriteWithUncommittedBlocks}}
> asserts that the pseudo key keeps a non zero object ID. It runs for OBS and
> FSO.
> Both fail on master without the fix. The new {{TestKeyDeletingService}} case
> also fails with the object ID kept but the hsync metadata removed from the
> pseudo key.
> Line numbers are from master at e62603cf3ca.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
