[
https://issues.apache.org/jira/browse/HDDS-16770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16770:
------------------------------
Attachment: CR-2-copy-bucket-info-into-fso-rename-response.patch
> FSO rename response serializes the live cached OmBucketInfo, corrupting
> bucket usage after crash and replay
> -----------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16770
> URL: https://issues.apache.org/jira/browse/HDDS-16770
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: CR-2-copy-bucket-info-into-fso-rename-response.patch,
> TestCR2RenameBucketAlias.java
>
>
> h3. Mechanism
> {{OMKeyRenameRequestWithFSO.setModificationTime}} puts a new {{OmBucketInfo}}
> into the bucket table cache, and {{renameKey}} hands the same instance to
> {{OMKeyRenameResponseWithFSO}} ({{:353}}), which serializes it on the double
> buffer flush thread. Later requests mutate that cached instance in place
> ({{OMKeyRequest.getBucketInfo}} returns the cached reference) before the
> rename's batch is written.
> The bucket row is therefore committed in a batch whose {{TransactionInfo}} is
> still below the later request's index, but already contains that request's
> {{usedNamespace}} / {{usedBytes}} delta.
> h3. Trigger
> A rename whose source or destination parent is the bucket root (a rename
> between directories carries no bucket in the response), followed by a request
> that changes bucket usage, such as a key commit, applied while the flush
> thread is still working through the batch that holds the rename, then an OM
> crash before the later request's batch is committed.
> h3. Impact
> On restart Ratis replays the later request on top of a bucket row that
> already includes it. With a tight namespace quota an acknowledged commit is
> rejected with {{QUOTA_EXCEEDED}} on replay and the file is missing on that
> OM. Otherwise bucket usage is double counted until a quota repair is run.
> h3. Reproduction
> PASS at component level: [^TestCR2RenameBucketAlias.java]
> ({{hadoop-ozone/ozone-manager}}) uses real RocksDB, table caches, the double
> buffer flush thread and the request classes, with the flush timing
> controlled. The crash and the replay are driven by the test, not by a real
> Ratis restart.
> h3. Suggested fix
> Give the response its own snapshot: pass {{omBucketInfo.copyObject()}} (when
> non null) to the response, matching the other FSO requests. HDDS-16125 fixed
> the same aliasing in {{OMQuotaRepairRequest}}.
> h3. Patch
> [^CR-2-copy-bucket-info-into-fso-rename-response.patch], against
> 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at
> c99d291e863.
> One line: {{renameKey}} passes {{omBucketInfo.copyObject()}} (or null when
> the rename does not touch the bucket) to the response. The copy is taken
> under the bucket lock, so no later request can have touched the object yet.
> This is what the other FSO key requests already do.
> Covered by {{testFlushedBucketExcludesLaterUsageChange}} in the existing
> {{TestOMKeyRenameRequestWithFSO}}: it applies a root level rename, changes
> {{usedNamespace}} on the cached bucket as a later request would, flushes the
> rename response and reads the bucket row from RocksDB. Without the change it
> fails with "expected: 0L but was: 1L". The rename and double buffer unit
> suites pass (44 tests), checkstyle is clean. A bucket that already double
> counted needs the existing quota repair.
> Found by TLA+ model checking and code review of the OM FSO namespace mutation
> paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
