Siyao Meng created HDDS-16769:
---------------------------------
Summary: OMKeyRenameResponseWithFSO reads the bucket ID at flush
time and terminates the OM if the bucket was deleted
Key: HDDS-16769
URL: https://issues.apache.org/jira/browse/HDDS-16769
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-5-carry-bucket-id-in-fso-rename-response.patch,
TestBugMC5RenameFlushBucketId.java
h3. Mechanism
{{OMKeyRenameResponseWithFSO.addToDBBatch}} calls {{getVolumeId}} and
{{getBucketId}} on the double buffer flush thread instead of carrying the IDs
resolved at apply time. Both are cache aware, so they see requests applied
after the rename. The lookup was added so that rename updates the parent
directory's modification time (HDDS-7382).
The client reply for a write does not wait for the flush, and
{{OMBucketDeleteRequest}} does not wait for unflushed responses of the bucket.
h3. Trigger
An FSO rename is applied and acknowledged. Before the flush thread reaches its
response, everything in the bucket is deleted and the bucket is deleted, and
optionally recreated under the same name.
h3. Impact
* Bucket deleted: {{getBucketId}} throws {{BUCKET_NOT_FOUND}} and
{{OzoneManagerDoubleBuffer.addToBatch}} terminates the OM with exit status 1. A
restart replays the Ratis log and recovers.
* Bucket recreated: the parent directory rows are written under the new bucket
ID while still carrying parent IDs of the old bucket's tree, and stay in
{{directoryTable}}. No client visible effect of these stray rows was observed.
h3. Reproduction
PASS, 2 of 2 runs, with client requests against one real OM while the flush is
held by {{OzoneManagerDoubleBuffer#pause}}:
[^TestBugMC5RenameFlushBucketId.java] ({{hadoop-ozone/ozone-manager}}). Two
concurrent designs without the hook did not hit the window.
h3. Suggested fix
Resolve {{volumeId}} and {{bucketId}} in
{{OMKeyRenameRequestWithFSO.validateAndUpdateCache}}, which already has both
under the bucket lock, and pass them to the response, as HDDS-6934 did for
{{OMKeysDeleteResponseWithFSO}} after the same race crashed an OM.
{{OMKeyRenameResponseWithFSO}} is the only response that still looks up these
IDs at flush time.
h3. Patch
[^MC-5-carry-bucket-id-in-fso-rename-response.patch], against
20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at
c99d291e863.
{{OMKeyRenameResponseWithFSO}} takes {{volumeId}} and {{bucketId}} as
constructor arguments and {{addToDBBatch}} uses them; the flush time lookups
are removed. {{OMKeyRenameRequestWithFSO.renameKey}} already has both under the
bucket lock and passes them on. The parent rows are then always written under
the bucket the rename was applied to; a later bucket delete removes them in log
order as usual.
Covered by {{testAddToDBBatchAfterBucketDeleted}} in the existing
{{TestOMKeyRenameResponseWithFSO}}, which fails without the change with
{{BUCKET_NOT_FOUND}} thrown from {{addToDBBatch}} (the exception
{{OzoneManagerDoubleBuffer.addToBatch}} turns into a terminate). The rename,
double buffer and {{TestOMSnapshotCreateRequest}} unit suites pass (62 tests),
checkstyle is clean.
Found by TLA+ model checking and code review of the OM FSO namespace mutation
paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
