Siyao Meng created HDDS-17745:
---------------------------------

             Summary: Optimized FSO snapshot diff drops every file change below 
the bucket root when no directory row changed
                 Key: HDDS-17745
                 URL: https://issues.apache.org/jira/browse/HDDS-17745
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: MC-5-run-directory-stage-when-only-files-changed.patch, 
TestBugMC5EmptyDirectoryDelta.java

h3. Mechanism
{{SnapshotDiffManager.scanTablesForDiff}} returns early when a table has no 
delta SST files, before it picks a reader. For an FSO bucket the directory 
stage is also what builds the path edges that {{SnapDiffPathResolver}} uses to 
resolve file entries. Creating or deleting a file under an existing directory 
writes nothing to {{directoryTable}}, so when only files changed the directory 
delta is empty, the directory stage is skipped, every file entry whose parent 
is not the bucket resolves to null, and {{MergeJoinSnapDiffWriter}} drops it 
with a DEBUG log. The job ends DONE. 
{{DagDiffSequentialReader.scanDirectoryTables}} already handles an empty delta 
by scanning both directory tables; the early return keeps it from running.

h3. Trigger
FSO bucket, {{ozone.om.snapshot.diff.optimized.enabled=true}}, native rocks 
tools library loaded (the default, {{ozone.om.snapshot.load.native.lib=true}}):
# Create {{/a/old}}, take snapshot snap1.
# Create {{/a/new}}, delete {{/a/old}}, take snapshot snap2.
# Request the snapshot diff snap1 to snap2, ordered or unordered.

The diff path does not matter once the native library is loaded: the DAG 
reader, a forced full diff and the automatic full diff fallback all build the 
directory delta only from SST files the two snapshots do not share, and all 
pass through the same early return. A forced full diff with the native library 
was run and gives the same empty report; the automatic fallback uses the same 
code (by reading). Without the native library the optimized path runs only with 
a forced full diff, and the non native delta computation then adds every SST 
file of the from snapshot to the delta, so the directory delta is never empty 
and the bug does not show.

h3. Impact
Both reports are empty instead of {{[CREATE a/new, DELETE a/old]}}. Any diff in 
which files were only created, overwritten or deleted, with no directory row 
written (no directory created, renamed or deleted, and no file renamed, since a 
file rename updates the modification time of both parent directories), loses 
every change below the bucket root. The report is persisted and returned for 
every later read of the pair.

h3. Reproduction
[^TestBugMC5EmptyDirectoryDelta.java] ({{hadoop-ozone/integration-test}}, needs 
{{-Drocks_tools_native}}) runs the steps above through the FileSystem and 
snapshot APIs on a mini cluster with the native library, on the default DAG 
path (the forced full diff variant was checked separately). On the unfixed code 
it fails with {{Expecting actual: [] to contain exactly in any order: [+ 
./a/new, - ./a/old]}}; with the patch it passes.

h3. Patch
[^MC-5-run-directory-stage-when-only-files-changed.patch], against 
d3b75c75bf9ad475551202477c3a21a3f4b968b7. It also applies to master at 
bbc8505e00d.

The early return now skips the directory stage only when the earlier file stage 
put nothing into the new and old lists ({{SnapDiffJobStore.hasListEntries}}, 
from the counters the store already keeps). The directory stage then runs with 
an empty delta and builds the edges. A diff with no changes at all keeps the 
fast path. A diff in which only files changed now scans both directory tables 
of the bucket, which a diff with one directory change already does.

Covered by {{testOptimizedDiffWithUnchangedDirectoryTable}} in the existing 
{{TestOptimizedSnapshotDiff}} (native), for both report orderings. Without the 
change both reports are empty. With it {{TestOptimizedSnapshotDiff}}, the 
snapshot diff unit suites and {{TestSnapshotDiffManager}} (191 tests) pass, and 
checkstyle is clean.

Found by TLA+ model checking and code review of the OM optimized snapshot diff 
paths under HDDS-15926, on commit d3b75c75bf9ad475551202477c3a21a3f4b968b7. 
Checked against HDDS issues and apache/ozone pull requests for duplicates 
before filing. The attached patch is a proposal for review. Generated with 
Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to