Siyao Meng created HDDS-17734:
---------------------------------

             Summary: Optimized snapshot diff of an OBS bucket is labelled 
"dependency ordered" although its entries are not
                 Key: HDDS-17734
                 URL: https://issues.apache.org/jira/browse/HDDS-17734
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: CR-1-obs-report-not-dependency-ordered.patch, 
TestBugCR1ObsReportLabeledOrdered.java

h3. Mechanism
For OBS (and LEGACY) buckets, that is every bucket that is not FSO, 
{{MergeJoinSnapDiffWriter.writeReport}} writes each entry as the merge join 
produces it. The report key orders the entries by diff type (DELETE, MODIFY, 
RENAME, CREATE), then in merge join (object ID) order, and there is no 
dependency ordering step (the {{writeReport}} Javadoc, the {{--ordered}} help 
and the {{ObjectStore.submitSnapshotDiff}} Javadoc all say ordering applies to 
FSO buckets). The OBS branch still sets the ordered outcome to true 
unconditionally, so {{SnapshotDiffManager}} finishes every optimized OBS job, 
ordered request or not, with the reason "Report is dependency ordered.".

The label was set on purpose: the diff type order puts deletes before renames 
and creates, and renames before creates, which meets the path conflict rules 
the original design relied on for OBS buckets, and 
{{TestOptimizedSnapshotDiff#testOptimizedObsDiff}} expected "ordered" even for 
an unordered request. It does not order a rename into a key that another rename 
frees, and the OBS path has no equivalent of the FSO graph rule for that case 
("RENAME(source) before RENAME(target) that reuses the same path" in 
{{SnapDiffDependencyGraph}}).

h3. Trigger
Optimized snapshot diff enabled ({{ozone.om.snapshot.diff.optimized.enabled}}, 
default false) and an OBS bucket. Create key {{a}}, then key {{b}}, take 
snapshot 1, rename {{b}} to {{c}}, then {{a}} to {{b}}, take snapshot 2, and 
request a snapshot diff.

h3. Impact
The report is {{RENAME a -> b, RENAME b -> c}} with the reason "Report is 
dependency ordered.". Applying it in that order to the keys of snapshot 1 fails 
at the first entry with KEY_ALREADY_EXISTS, because {{b}} still exists. A 
client that trusts the label to replay the report fails or has to reorder it 
itself. No data in the bucket is changed.

h3. Reproduction
[^TestBugCR1ObsReportLabeledOrdered.java] is a self contained mini cluster 
class (target path and Maven command in the class comment) that enables the 
optimized diff with a forced full diff and native diff disabled and uses only 
public client calls. On unmodified source the ordered request ends DONE "Report 
is dependency ordered." with {{RENAME a -> b, RENAME b -> c}}, and renaming the 
keys of a second OBS bucket that holds the snapshot 1 keys in report order 
fails at {{RENAME a -> b}} with KEY_ALREADY_EXISTS. A passing test means the 
defect is present.

h3. Patch
[^CR-1-obs-report-not-dependency-ordered.patch], against 
d3b75c75bf9ad475551202477c3a21a3f4b968b7. It also applies to master at 
bbc8505e00d.

The OBS branch now reports the outcome as not ordered, so optimized OBS jobs 
end with "Report is dependency unordered.", the reason FSO already uses when it 
cannot order a report. The entry order and the request handling are unchanged. 
The new label is conservative on purpose: a report without renames, whose type 
order already replays, is now labelled unordered too. An exact label (ordered 
unless a rename targets another rename's source) would need a rename path index 
that grows with the report, and a swap of two keys cannot be ordered anyway. 
Ordering OBS reports for real (a path dependency graph for renames, deletes and 
creates) would be a feature and is not attempted.

Covered by {{testObsReportIsNotDependencyOrdered}} in the existing 
{{TestMergeJoinSnapDiffWriter}}, which writes the rename chain above and 
expects the rows in object ID order and an unordered result. 
{{TestOptimizedSnapshotDiff#testOptimizedObsDiff}} asserted the ordered reason 
for an OBS diff and now expects "Report is dependency unordered."; that class 
needs the native library and was compiled but not run. Without the change the 
new test fails with "expected: <false> but was: <true>"; with it 
{{TestSnapDiffDependencyGraph}}, {{TestMergeJoinSnapDiffWriter}} and 
{{TestSnapshotDiffManager}} pass (122 tests) and checkstyle is clean. With the 
patch the reproduction fails as intended, with the reason "Report is dependency 
unordered.".

Found by code review of the OM optimized snapshot diff paths, as part of the 
TLA+ verification effort under HDDS-15926, on commit 
d3b75c75bf9ad475551202477c3a21a3f4b968b7. Checked against HDDS issues and 
apache/ozone pull requests for duplicates before filing. The attached patch is 
a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to