[
https://issues.apache.org/jira/browse/HDDS-17045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17045:
------------------------------
Attachment: CR-3-resolve-old-parent-of-legacy-directory-update.patch
> Recon adds a Legacy directory's totals to its ancestors again on every in
> place update of the directory key, such as an ACL change
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17045
> URL: https://issues.apache.org/jira/browse/HDDS-17045
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> CR-3-resolve-old-parent-of-legacy-directory-update.patch,
> TestBugCR3LegacyDirUpdateDoubleCount.java
>
>
> h3. Mechanism
> For an UPDATE event on a directory key in a LEGACY bucket with
> {{ozone.om.enable.filesystem.paths=true}},
> {{NSSummaryTaskWithLegacy.processWithFileSystemLayout}} builds the old
> directory with {{oldKeyInfo.getParentObjectID()}}. keyTable entries do not
> carry a parent object ID (it is 0), which is why the same method looks the
> parent up by name with {{setKeyParentID}} for the new value and for files.
> {{NSSummaryTaskDbEventHandler.handleDeleteDirEvent}} then finds no summary
> for parent 0, logs "The namespace table is not correctly populated." and
> returns without subtracting anything, while {{handlePutDirEvent}} adds the
> directory's existing totals to its real parent and all ancestors.
> FSO and OBS buckets, Legacy buckets without filesystem paths, and Legacy file
> keys resolve the old parent correctly.
> h3. Trigger
> LEGACY bucket, {{ozone.om.enable.filesystem.paths=true}} (not the default).
> Create directory {{dir1}} and a file {{dir1/file1}}, then change the ACL of
> the key {{dir1/}} with {{ObjectStore.addAcl}}. The other requests that
> rewrite a directory key in place ({{setAcl}}, {{removeAcl}}, {{setTimes}})
> take the same path (argued from the code, not run).
> h3. Impact
> * Each update of a non empty directory key adds the directory's file count
> and size again to its parent and every ancestor up to the bucket, and so to
> the volume and root totals, which the endpoints compute by adding up the
> bucket summaries. In the reproduction one {{addAcl}} turned a bucket with 1
> file and 100 bytes into 2 files and 200 bytes.
> * The extra amount is never subtracted: later deletes subtract only the real
> content. It is served by the namespace summary and disk usage endpoints until
> a full NSSummary rebuild. OM data is not affected.
> * Since when: the parent id has been read this way since HDDS-7121, but a
> missed subtraction only became a double count when HDDS-13432 made directory
> updates carry totals up the tree, so 2.1.0 and later are affected.
> h3. Reproduction
> Deterministic, unmodified source.
> [^TestBugCR3LegacyDirUpdateDoubleCount.java]
> ({{hadoop-ozone/integration-test-recon}}) starts a mini cluster with Recon
> and filesystem paths enabled and uses only public client calls:
> {{createDirectory("dir1")}}, a 100 byte {{dir1/file1}}, a Recon sync,
> {{addAcl}} on {{dir1/}}, a 10 byte {{marker}} key, and another sync. The
> marker shows when Recon has processed the sync with the ACL change. On the
> unfixed code it fails with {{Bucket after ACL change on dir1/ and the marker
> key: numOfFiles=3 sizeOfFiles=210 (expected 2 and 110)}}. With the patch it
> passes.
> h3. Patch
> [^CR-3-resolve-old-parent-of-legacy-directory-update.patch], against
> 0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at
> 2264d4d48ac.
> The old directory gets the parent already resolved for the new value. For an
> UPDATE both values are stored under the same key and have the same key name,
> so the parent is the same, and no second lookup is needed. Covered by the new
> {{testProcessDirKeyUpdateKeepsAncestorTotals}} in the existing
> {{TestNSSummaryTaskWithLegacy}} (an UPDATE on {{dir1/dir2/}} must leave
> {{dir1}} and the bucket unchanged), and by {{testProcessUpdateFileSize}},
> which asserted the double count (2 files in a bucket that holds 1) and now
> expects 1. Without the change both fail. With it the existing Recon
> NSSummary, endpoint and event handler suites pass and checkstyle is clean.
> Found by code review of the Recon namespace summary and task sync paths, as
> part of the TLA+ verification effort under HDDS-15926, on commit
> 0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
