[
https://issues.apache.org/jira/browse/HDDS-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16441:
------------------------------
Status: Patch Available (was: In Progress)
> Cancelled snapshot diff generation publishes an unreadable DONE report on
> resubmit
> ----------------------------------------------------------------------------------
>
> Key: HDDS-16441
> URL: https://issues.apache.org/jira/browse/HDDS-16441
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Critical
> Labels: pull-request-available
> Attachments: HDDS-16441.001.patch
>
>
> h3. Finding
> A cancelled generation's task publishes its result as the new generation's
> DONE, so the resubmit path (submitSnapshotDiff, cancelSnapshotDiff,
> submitSnapshotDiff) leaves a report the reader cannot read (jobId-prefixed
> key mismatch fails the integrity check) and NPEs on read; the job is stuck
> DONE, is not client-resubmittable, and the client's legitimately recomputed
> diff is discarded, with no recovery until the report-retention TTL GC.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: Critical
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-1
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: om-snapshot-diff-jobs
> Guidance:
> campaigns/ozone-9fbf9ee/targets/009-om-snapshot-diff-jobs/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=high --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "om-snapshot-diff-jobs|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16430 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> 1. Client submits a diff for (fromSnapshot, toSnapshot); generation gen1 runs
> (jobKey status IN_PROGRESS), gen1's executor task is live.
> 2. Client cancels: cancelSnapshotDiff flips the jobKey IN_PROGRESS to
> CANCELLED but does not interrupt the still-running gen1 task.
> 3. Client resubmits (CANCELLED is resubmittable): a new generation gen2 with
> a fresh jobId is created under the same jobKey and the jobKey status is
> flipped back to IN_PROGRESS.
> 4. The orphaned gen1 task reaches its publish step. Its guard checks only
> that the jobKey status is IN_PROGRESS (now true again for gen2), so it calls
> updateJobStatusToDone and publishes gen1's result as the current generation's
> DONE.
> 5. The stored generation is now gen2's jobId but the report was published for
> gen1. The report is unreadable (the report keys embed the jobId, so the
> integrity check fails) and the job is stuck DONE (not resubmittable), so the
> client's legitimately recomputed gen2 diff is discarded until the
> report-retention GC.
> h3. Root cause
> hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java:
> the running task's publish (updateJobStatusToDone) was guarded only by
> status == IN_PROGRESS, never by the task's own generation. Cancellation is
> cooperative and a resubmit reuses the same jobKey with a new jobId, so a
> superseded generation's task could publish onto the new generation.
> h3. Suggested fix
> Thread the running task's jobId into updateJobStatusToDone and publish only
> when the stored SnapshotDiffJob.getJobId() still equals the task's jobId;
> otherwise log and skip. This makes publication generation-aware so a
> cancelled generation's orphaned task cannot publish onto a newer one. The
> method is also null-guarded. A unit test asserts a superseded-generation
> publish is rejected while the live generation publishes normally. See
> [^HDDS-16441.001.patch].
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge. A deterministic end-to-end reproduction of the
> race would require placing the orphaned task at its publish step across the
> resubmit (an executor rendezvous seam not present in shipping code); the test
> therefore pins the generation-match invariant on the publish path directly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
