[
https://issues.apache.org/jira/browse/HDDS-16769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16769:
------------------------------
Attachment: MC-5-carry-bucket-id-in-fso-rename-response.patch
> OMKeyRenameResponseWithFSO reads the bucket ID at flush time and terminates
> the OM if the bucket was deleted
> ------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16769
> URL: https://issues.apache.org/jira/browse/HDDS-16769
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-5-carry-bucket-id-in-fso-rename-response.patch,
> TestBugMC5RenameFlushBucketId.java
>
>
> h3. Mechanism
> {{OMKeyRenameResponseWithFSO.addToDBBatch}} calls {{getVolumeId}} and
> {{getBucketId}} on the double buffer flush thread instead of carrying the IDs
> resolved at apply time. Both are cache aware, so they see requests applied
> after the rename. The lookup was added so that rename updates the parent
> directory's modification time (HDDS-7382).
> The client reply for a write does not wait for the flush, and
> {{OMBucketDeleteRequest}} does not wait for unflushed responses of the bucket.
> h3. Trigger
> An FSO rename is applied and acknowledged. Before the flush thread reaches
> its response, everything in the bucket is deleted and the bucket is deleted,
> and optionally recreated under the same name.
> h3. Impact
> * Bucket deleted: {{getBucketId}} throws {{BUCKET_NOT_FOUND}} and
> {{OzoneManagerDoubleBuffer.addToBatch}} terminates the OM with exit status 1.
> A restart replays the Ratis log and recovers.
> * Bucket recreated: the parent directory rows are written under the new
> bucket ID while still carrying parent IDs of the old bucket's tree, and stay
> in {{directoryTable}}. No client visible effect of these stray rows was
> observed.
> h3. Reproduction
> PASS, 2 of 2 runs, with client requests against one real OM while the flush
> is held by {{OzoneManagerDoubleBuffer#pause}}:
> [^TestBugMC5RenameFlushBucketId.java] ({{hadoop-ozone/ozone-manager}}). Two
> concurrent designs without the hook did not hit the window.
> h3. Suggested fix
> Resolve {{volumeId}} and {{bucketId}} in
> {{OMKeyRenameRequestWithFSO.validateAndUpdateCache}}, which already has both
> under the bucket lock, and pass them to the response, as HDDS-6934 did for
> {{OMKeysDeleteResponseWithFSO}} after the same race crashed an OM.
> {{OMKeyRenameResponseWithFSO}} is the only response that still looks up these
> IDs at flush time.
> h3. Patch
> [^MC-5-carry-bucket-id-in-fso-rename-response.patch], against
> 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at
> c99d291e863.
> {{OMKeyRenameResponseWithFSO}} takes {{volumeId}} and {{bucketId}} as
> constructor arguments and {{addToDBBatch}} uses them; the flush time lookups
> are removed. {{OMKeyRenameRequestWithFSO.renameKey}} already has both under
> the bucket lock and passes them on. The parent rows are then always written
> under the bucket the rename was applied to; a later bucket delete removes
> them in log order as usual.
> Covered by {{testAddToDBBatchAfterBucketDeleted}} in the existing
> {{TestOMKeyRenameResponseWithFSO}}, which fails without the change with
> {{BUCKET_NOT_FOUND}} thrown from {{addToDBBatch}} (the exception
> {{OzoneManagerDoubleBuffer.addToBatch}} turns into a terminate). The rename,
> double buffer and {{TestOMSnapshotCreateRequest}} unit suites pass (62
> tests), checkstyle is clean.
> Found by TLA+ model checking and code review of the OM FSO namespace mutation
> paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
