Siyao Meng created HDDS-17733:
---------------------------------
Summary: Snapshot diff page token is not checked against the
current report
Key: HDDS-17733
URL: https://issues.apache.org/jira/browse/HDDS-17733
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-8-reject-page-token-not-in-current-report.patch
h3. Mechanism
A snapshot diff page token is the suffix of the key of the next report row.
Each page request looks up the job of the snapshot pair again, and
{{SnapshotDiffManager.createPageResponse}} scans from {{<jobId>-<token>}}, the
row the token names, without checking that this row belongs to the current
report.
A report is stored in one of two key formats: dependency ordered rows are
{{<jobId>-0<20 digits>}}, unordered rows are {{<jobId>-<1..4><20 digits>}}.
When the job of a snapshot pair has been regenerated as an unordered report
between two page requests of a dependency ordered report, the token of the
earlier run sorts before every row of the new run, and the resumed read
continues from the first row of the new run.
h3. Trigger
Requires {{ozone.om.snapshot.diff.optimized.enabled}} = true (default false)
and an FSO bucket, where dependency ordered reports are produced (added by
HDDS-15391, not in any release).
# A client reads page 1 of a dependency ordered report and keeps the token.
# The job expires through {{SnapshotDiffCleanupService}}
({{ozone.om.snapshot.diff.job.report.persistent.time}}, 7 days by default,
counted from job creation).
# An unordered report is requested for the same snapshot pair and completes.
# The client requests the next page with its token.
h3. Impact
The resumed read returns the whole new report, so the rows of the pages already
read appear twice; for a 5 entry report read with page size 2 the client
assembles 7 rows. No error is returned.
h3. Reproduction
This issue has no standalone reproduction; the regression test in the patch is
the reproduction.
{{TestSnapshotDiffManager#testCreatePageResponseRejectsTokenOfRegeneratedReport}}
applies a token of a dependency ordered run to an unordered report of the same
snapshot pair and expects it to be rejected. On
d3b75c75bf9ad475551202477c3a21a3f4b968b7 it fails with "Expected
java.io.IOException to be thrown, but nothing was thrown", because the scan
restarts at the first row.
h3. Patch
[^MC-8-reject-page-token-not-in-current-report.patch], against
d3b75c75bf9ad475551202477c3a21a3f4b968b7. It also applies to master at
bbc8505e00d.
{{createPageResponse}} now looks up the row the token names before scanning
and, for a report with entries, rejects a token that names no row of the
current report with "Token ... does not belong to the current report of this
snapshot diff. The report may have been regenerated, retry from the first page
without a token." A token of the current report pages exactly as before; the
cost is one point lookup per page request that carries a token. The token
format is unchanged.
The new test also checks that a token of the current report still returns the
rows from the one it names. Without the change it fails as above; with it
{{TestSnapshotDiffManager}} (85 tests) and the mini cluster suite
{{TestOmSnapshotFsoWithoutNativeLib}} pass and checkstyle is clean.
Found by TLA+ model checking and code review of the OM optimized snapshot diff
paths under HDDS-15926, on commit d3b75c75bf9ad475551202477c3a21a3f4b968b7.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
