[
https://issues.apache.org/jira/browse/HDDS-17744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17744:
------------------------------
Attachment: TestBugMC1RenameIntoDeletedDir.java
> Optimized snapshot diff omits a file or directory moved into a directory that
> is deleted before its children are purged
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17744
> URL: https://issues.apache.org/jira/browse/HDDS-17744
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-1-report-move-into-deleted-dir-as-delete.patch,
> TestBugMC1RenameIntoDeletedDir.java
>
>
> h3. Mechanism
> A file or directory moved under a directory that is then deleted keeps its
> row in {{fileTable}} or {{directoryTable}} until the directory deep clean
> purges the children, so a snapshot taken in between holds it under a parent
> that no longer exists in {{directoryTable}}. The merge join sees the same
> objectId on both sides and classifies it as RENAME. In
> {{MergeJoinSnapDiffWriter.resolveReportEntries}} the target path cannot be
> resolved, so the entry becomes null, and {{writeReportEntriesForRows}} drops
> it with a DEBUG log. Nothing else in the report removes it. The legacy diff
> ({{SnapshotDiffManager.generateDiffReport}}) handles the same shape in its
> branch for an old path that resolves and a new path that does not, and
> reports a DELETE of the old path.
> h3. Trigger
> FSO bucket, {{ozone.om.snapshot.diff.optimized.enabled=true}} (default false;
> the optimized diff was added by HDDS-15391 and is not in any release yet):
> # Create file {{/a}} and directory {{/b}}, take snapshot snap1.
> # {{mv /a /b/a}}, then {{rm -r /b}}.
> # Take snapshot snap2 before the directory deleting service has purged the
> children of {{b}} (it runs on an interval).
> # Request the snapshot diff snap1 to snap2, ordered or unordered.
> h3. Impact
> The DONE report is {{[DELETE b]}} instead of {{[DELETE a, DELETE b]}}.
> Replaying it onto snap1 keeps {{/a}}, which does not exist in snap2. The
> report is persisted and returned for every later read of the pair. The legacy
> diff of an identical pair on the same OM reports both deletes. If the moved
> object is a directory, its whole subtree stays on replay: moving a directory
> into a trash directory and then deleting the trash directory recursively is a
> common pattern.
> h3. Reproduction
> [^TestBugMC1RenameIntoDeletedDir.java] ({{hadoop-ozone/integration-test}})
> runs the steps above through the FileSystem and snapshot APIs on a mini
> cluster, with forced full diff and native diff disabled so the native library
> is not needed. The base class keeps the KeyManager background services
> stopped, which holds the window open. It first computes the legacy report of
> an identical pair (both deletes), then asserts the optimized unordered and
> ordered reports. On the unfixed code it fails with {{Expecting actual: [-
> ./b] ... could not find the following elements: [- ./a]}}; with the patch it
> passes.
> h3. Patch
> [^MC-1-report-move-into-deleted-dir-as-delete.patch], against
> d3b75c75bf9ad475551202477c3a21a3f4b968b7. It also applies to master at
> bbc8505e00d.
> In the RENAME branch of {{MergeJoinSnapDiffWriter.resolveReportEntries}},
> when the source path resolves and the target path does not, the writer now
> emits DELETE of the source path, the same rule as the legacy diff. Entries
> where neither path resolves are still dropped. The dependency ordered report
> needs no extra handling, since the graph takes the entry type from the report
> entry.
> Covered by {{testRenameIntoDeletedDirectoryReportsDeleteOfSource}} in the
> existing {{TestMergeJoinSnapDiffWriter}}, for both report orderings. Without
> the change it fails because only {{DELETE b}} is reported. With it the
> snapshot diff unit suites and {{TestSnapshotDiffManager}} (193 tests) and
> {{TestOptimizedSnapshotDiff}} with the native library pass, and checkstyle is
> clean.
> One previously correct report changes: if the source's parent is deleted as
> well ({{mv /c/a /b/a}}, {{rm -r /b}}, {{rm -r /c}}), the report gains a
> redundant but harmless {{DELETE c/a}} next to {{DELETE c}}, ordered before
> it; the legacy diff emits it too.
> Not changed: if the moved file was also modified, a MODIFY of the old path is
> still emitted next to the DELETE (the legacy diff emits only the DELETE).
> Found by TLA+ model checking and code review of the OM optimized snapshot
> diff paths under HDDS-15926, on commit
> d3b75c75bf9ad475551202477c3a21a3f4b968b7. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
