[
https://issues.apache.org/jira/browse/HDDS-17604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17604:
------------------------------
Affects Version/s: 2.2.1
2.0.0
1.4.0
2.1.2
2.1.0
1.4.1
2.1.1
2.2.0
> A duplicate snapshot purge removes a new ACTIVE snapshot that was created
> with the purged snapshot's name
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17604
> URL: https://issues.apache.org/jira/browse/HDDS-17604
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.4.0, 2.0.0, 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-1-skip-purge-of-snapshot-not-deleted.patch,
> TestBugMC1DuplicatePurgeRecreatedSnapshot.java
>
>
> h3. Mechanism
> A {{SnapshotPurgeRequest}} carries only snapshot table keys
> ({{/volume/bucket/name}}). {{OMSnapshotPurgeRequest.validateAndUpdateCache}}
> purges whatever row the key resolves to when the entry is applied. It does
> not check that the row is still the snapshot {{SnapshotDeletingService}}
> selected, or that it is still {{SNAPSHOT_DELETED}}. The only guard, added by
> HDDS-10408, skips a key whose row is already gone.
> {{OMSnapshotPurgeResponse.addToDBBatch}} then reads the row by key from the
> DB and deletes it together with its checkpoint directory, also for keys that
> the request skipped.
> So if a second purge for a key is applied after a client created a new
> snapshot with the same name, the new ACTIVE snapshot is removed from the
> snapshot chain, the snapshot table and the disk on every OM.
> h3. Trigger
> # Create snapshot {{s1}} in a bucket, then delete it.
> # {{SnapshotDeletingService}} on the leader submits a purge for {{s1}}. The
> entry commits, then leadership moves to another OM before that OM has applied
> it.
> # The new leader reports ready as soon as Ratis dispatches its startup entry
> to the state machine ({{LeaderStateImpl.checkReady}} from
> {{RaftServerImpl.applyLogToStateMachine}}), while OM still has earlier
> entries queued on its single apply executor, so the purge can still be queued
> (the leader readiness gap is HDDS-17579). {{SnapshotDeletingService}} checks
> only {{OzoneManager.isLeaderReady}} and {{shouldIgnoreSnapshot}} (status
> DELETED and changes flushed) and keeps no record of submitted purges, so it
> can build a second purge for {{s1}}.
> # A client creates a snapshot named {{s1}} again, and the create is ordered
> after the first purge and before the second. It succeeds with a new snapshot
> ID, because the first purge's cache tombstone makes {{isExist}} return false.
> # The second purge resolves the key to the new ACTIVE snapshot and purges it.
> Steps 2 to 4 need a leader change while the first purge is unapplied on the
> new leader, a {{SnapshotDeletingService}} run (default interval 30s) inside
> that apply backlog, and a create with the same name submitted in the same
> window. The window is narrow. The readiness gap of step 3 is shown on a 3 OM
> mini cluster in HDDS-17579; the duplicate purge on top of it was not run on a
> cluster. HDDS-16795 is the same pattern in {{KeyDeletingService}}. This issue
> is not a duplicate of either: it is a different consequence, and the purge
> needs its own check even after HDDS-17579 is fixed.
> On 1.4.0 and 1.4.1 the duplicate does not need a leader change:
> {{SnapshotDeletingService}} selected snapshots by iterating the on disk
> snapshot table, so a run between the first purge's apply and its flush could
> build it on a single leader.
> h3. Impact
> The client's create returned success, but afterwards the snapshot is not
> found. Its row is deleted from the snapshot table, its chain entry is
> removed, so a chain rebuilt from the table on restart does not have it
> either, and its checkpoint directory is deleted. The deep clean flags of the
> following snapshots are reset as for a normal purge. Snapshot create moves
> the bucket's pending deletes ({{deletedTable}}, {{deletedDirTable}},
> {{snapshotRenamedTable}} entries) from the active DB into the new snapshot,
> and the purge does not move them on first, so they are lost and their blocks
> are never reclaimed (argued, not run). The loss is replicated and permanent.
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugMC1DuplicatePurgeRecreatedSnapshot.java]
> ({{hadoop-ozone/ozone-manager}}) uses the request level fixture of the
> snapshot request tests (real metadata manager, snapshot chain manager,
> snapshot manager and checkpoint directories, mocked {{OzoneManager}}). It
> runs create, delete, purge, create with the same name and the same purge
> again through {{preExecute}}, {{validateAndUpdateCache}} and the response DB
> update. After the second purge the new snapshot's row is gone from the cache
> and the DB, a chain rebuilt from the table does not have it, its checkpoint
> directory is deleted, and {{SnapshotUtils.getSnapshotInfo}} reports it as not
> found. A passing test means the defect is present. The test supplies the
> second purge; how it arises on a cluster (Trigger, steps 2 and 3) is argued
> from the code, not run.
> h3. Suggested fix
> The patch makes the apply side skip a key whose row is not
> {{SNAPSHOT_DELETED}} and passes only the purged keys to the response. It
> fixes the reported case without a protocol change. It does not cover a
> snapshot recreated with the same name and deleted again before the duplicate
> purge is applied: that row is DELETED and is still purged before
> {{SnapshotDeletingService}} has moved its pending deletes, which are then
> lost as above. Carrying the snapshot IDs that {{SnapshotDeletingService}}
> selected in {{SnapshotPurgeRequest}} and comparing them at apply would cover
> that too, but needs a new request field behind a layout feature.
> h3. Patch
> [^MC-1-skip-purge-of-snapshot-not-deleted.patch], against
> 284fad307bfd129fca3f4966d854ac45ea3b0d11. It also applies to master at
> 6135e81a09ac6d618dc0ad4d220dd8cf254586b6.
> {{OMSnapshotPurgeRequest.validateAndUpdateCache}} skips, with a warning, a
> key whose current row is not {{SNAPSHOT_DELETED}}, and gives
> {{OMSnapshotPurgeResponse}} only the keys it purged, so the response no
> longer deletes the row or checkpoint of a skipped key. A skipped row is not
> rewritten, unless the same request updated it as the next snapshot of a
> purged one. {{NumSnapshotDeleted}} is decremented by the number of purged
> keys, which now are all DELETED.
> On a mixed version ring during a rolling upgrade, in exactly the bug case an
> upgraded OM keeps the recreated snapshot and an old one purges it; every
> other purge behaves the same on both versions.
> Covered by two new tests in the existing
> {{TestOMSnapshotPurgeRequestAndResponse}}, next to
> {{testDuplicateSnapshotPurge}} from HDDS-11997:
> {{testDuplicateSnapshotPurgeKeepsRecreatedSnapshot}} (the recreated snapshot
> is unchanged in the cache, the DB, the chain and on disk after the duplicate
> purge) and {{testSkippedSnapshotNextToPurgedSnapshotIsUpdated}}, run with the
> skipped key first and with the purged key first (a skipped snapshot that
> follows a purged one keeps its row and checkpoint, and gets its deep clean
> flags and previous snapshot IDs updated in the cache and the DB). Without the
> change all three cases fail because the recreated snapshot is purged. Four
> existing tests in the suite purged ACTIVE snapshots directly; they now mark
> them deleted first.
> Found by TLA+ model checking and code review of the OM snapshot purge, set
> property and double buffer apply paths under HDDS-15926, on commit
> 284fad307bfd129fca3f4966d854ac45ea3b0d11. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
