Siyao Meng created HDDS-16770:
---------------------------------
Summary: FSO rename response serializes the live cached
OmBucketInfo, corrupting bucket usage after crash and replay
Key: HDDS-16770
URL: https://issues.apache.org/jira/browse/HDDS-16770
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-2-copy-bucket-info-into-fso-rename-response.patch,
TestCR2RenameBucketAlias.java
h3. Mechanism
{{OMKeyRenameRequestWithFSO.setModificationTime}} puts a new {{OmBucketInfo}}
into the bucket table cache, and {{renameKey}} hands the same instance to
{{OMKeyRenameResponseWithFSO}} ({{:353}}), which serializes it on the double
buffer flush thread. Later requests mutate that cached instance in place
({{OMKeyRequest.getBucketInfo}} returns the cached reference) before the
rename's batch is written.
The bucket row is therefore committed in a batch whose {{TransactionInfo}} is
still below the later request's index, but already contains that request's
{{usedNamespace}} / {{usedBytes}} delta.
h3. Trigger
A rename whose source or destination parent is the bucket root (a rename
between directories carries no bucket in the response), followed by a request
that changes bucket usage, such as a key commit, applied while the flush thread
is still working through the batch that holds the rename, then an OM crash
before the later request's batch is committed.
h3. Impact
On restart Ratis replays the later request on top of a bucket row that already
includes it. With a tight namespace quota an acknowledged commit is rejected
with {{QUOTA_EXCEEDED}} on replay and the file is missing on that OM. Otherwise
bucket usage is double counted until a quota repair is run.
h3. Reproduction
PASS at component level: [^TestCR2RenameBucketAlias.java]
({{hadoop-ozone/ozone-manager}}) uses real RocksDB, table caches, the double
buffer flush thread and the request classes, with the flush timing controlled.
The crash and the replay are driven by the test, not by a real Ratis restart.
h3. Suggested fix
Give the response its own snapshot: pass {{omBucketInfo.copyObject()}} (when
non null) to the response, matching the other FSO requests. HDDS-16125 fixed
the same aliasing in {{OMQuotaRepairRequest}}.
h3. Patch
[^CR-2-copy-bucket-info-into-fso-rename-response.patch], against
20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at
c99d291e863.
One line: {{renameKey}} passes {{omBucketInfo.copyObject()}} (or null when the
rename does not touch the bucket) to the response. The copy is taken under the
bucket lock, so no later request can have touched the object yet. This is what
the other FSO key requests already do.
Covered by {{testFlushedBucketExcludesLaterUsageChange}} in the existing
{{TestOMKeyRenameRequestWithFSO}}: it applies a root level rename, changes
{{usedNamespace}} on the cached bucket as a later request would, flushes the
rename response and reads the bucket row from RocksDB. Without the change it
fails with "expected: 0L but was: 1L". The rename and double buffer unit suites
pass (44 tests), checkstyle is clean. A bucket that already double counted
needs the existing quota repair.
Found by TLA+ model checking and code review of the OM FSO namespace mutation
paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
