[ 
https://issues.apache.org/jira/browse/HDDS-17740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDDS-17740:
------------------------------
    Attachment: CR-6-keep-resubmit-queued-report-for-one-cleanup-run.patch

> Ordered snapshot diff resubmit skips the cleanup grace period, so a 
> concurrent read of the DONE report fails
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17740
>                 URL: https://issues.apache.org/jira/browse/HDDS-17740
>             Project: Apache Ozone
>          Issue Type: Bug
>    Affects Versions: 2.3.0
>            Reporter: Siyao Meng
>            Priority: Minor
>         Attachments: 
> CR-6-keep-resubmit-queued-report-for-one-cleanup-run.patch, 
> TestBugCR6ResubmitSkipsCleanupGrace.java
>
>
> h3. Mechanism
> {{SnapshotDiffCleanupService.run}} first deletes the reports of jobs in the 
> purged job table and only then moves expired or failed jobs there. The 
> comment in {{run}} explains why: a job moved in one run keeps its report 
> until the next run, so that a request that already saw the job as DONE can 
> still read the report. With {{ozone.om.snapshot.diff.optimized.enabled}} = 
> true (default false), an ordered request for a pair on an FSO bucket whose 
> job is DONE with an unordered report replaces that job, and 
> {{SnapshotDiffManager.createSnapDiffJob}} writes the old job ID straight into 
> the purged job table. The very next cleanup run then deletes the old report. 
> {{getSnapshotDiffReport}} takes no lock, so a call that read the DONE job 
> before the resubmit can open the report after it was deleted.
> h3. Trigger
> # An unordered snapshot diff of a pair finishes DONE.
> # A client calls {{getSnapshotDiffReport}} and reads the DONE job.
> # Before that call reads the report, another request submits the pair with an 
> ordered report, and a cleanup run happens (default every 60 minutes).
> The resubmit and the cleanup run must both fall inside one read call. The 
> next page call reads the job table again and gets the new job's IN_PROGRESS 
> status, so a client paging through a report is not affected beyond the one 
> call.
> h3. Impact
> That read call fails instead of returning the page: with an empty report the 
> integrity check in {{checkReportsIntegrity}} fails, and at the moment the 
> call ends in the NullPointerException of HDDS-16442 (with the open fix for 
> HDDS-16442 it would be an IOException "Report integrity check failed", read 
> from that change, not run). A retry returns the new job's status.
> h3. Reproduction
> PASS, deterministic, unmodified source. 
> [^TestBugCR6ResubmitSkipsCleanupGrace.java] ({{hadoop-ozone/ozone-manager}}) 
> runs the real {{SnapshotDiffManager}} and {{SnapshotDiffCleanupService}} on a 
> real RocksDB with the rest of the OM mocked. A stand in for the report 
> generation task writes five report rows and publishes DONE through the real 
> {{updateJobStatusToDone}}. The only timing aid holds the reader in 
> {{getSnapshotRootPath}}, the call right after it read the DONE job; meanwhile 
> the test submits an ordered request and runs the cleanup once. Output on 
> d3b75c75bf9ad475551202477c3a21a3f4b968b7: 0 of 5 report rows left after one 
> cleanup run, and the reader's call fails with a NullPointerException from 
> {{updateJobStatus}}. A passing test means the defect is present.
> h3. Patch
> [^CR-6-keep-resubmit-queued-report-for-one-cleanup-run.patch], against 
> d3b75c75bf9ad475551202477c3a21a3f4b968b7. It also applies to master at 
> bbc8505e00d.
> {{SnapshotDiffCleanupService}} remembers which purged entries it saw in the 
> previous run. {{removeOlderJobReport}} deletes only the reports of entries 
> that were already there in the previous run and keeps an entry it sees for 
> the first time for one more run. Jobs the cleanup moves itself are deleted on 
> the next run as before, so the existing grace period is unchanged and a 
> resubmit now gets the same one. The change is local to the cleanup service, 
> with no change to the stored format. After an OM restart the first run only 
> records the existing entries, so their reports are deleted one interval later 
> than today.
> Covered by {{testReportQueuedBetweenRunsIsKeptForOneRun}} in the existing 
> {{TestSnapshotDiffCleanupService}}: after one cleanup run, a report and its 
> purged entry are written; they must survive the next run and be gone after 
> the one after it. Without the change it fails because the run after the entry 
> was written deletes the report. With it {{TestSnapshotDiffManager}}, 
> {{TestSnapshotDiffCleanupService}} and {{TestSnapshotDiffManagerMXBean}} (88 
> tests) pass, checkstyle is clean, and the reproduction fails because all five 
> rows remain after its cleanup run.
> Found by code review of the OM optimized snapshot diff paths, as part of the 
> TLA+ verification effort under HDDS-15926, on commit 
> d3b75c75bf9ad475551202477c3a21a3f4b968b7. Checked against HDDS issues and 
> apache/ozone pull requests for duplicates before filing. The attached patch 
> is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to