[
https://issues.apache.org/jira/browse/HDDS-16788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16788:
------------------------------
Attachment: TestBugCR4CompleteMissingOpenKey.java
> CompleteMultipartUpload terminates the OM when the open key of an
> OBJECT_STORE or LEGACY upload is missing
> ----------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16788
> URL: https://issues.apache.org/jira/browse/HDDS-16788
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> CR-4-fail-multipart-complete-when-open-key-is-missing.patch,
> TestBugCR4CompleteMissingOpenKey.java
>
>
> h3. Mechanism
> {{S3MultipartUploadCompleteRequest.getOmKeyInfo}} reads the open key of the
> upload and uses the result without a null check in both of its branches:
> {{dbOpenKeyInfo.toBuilder()}} when the key does not exist yet,
> {{dbOpenKeyInfo.getFileEncryptionInfo()}} when an existing key is
> overwritten. If the upload still has its {{multipartInfoTable}} entry but no
> open key, the {{NullPointerException}} passes the {{IOException |
> InvalidPathException}} handler in {{validateAndUpdateCache}}, reaches
> {{OzoneManagerStateMachine.runCommand}}, and the OM calls
> {{ExitUtils.terminate}}.
> Other requests already accept this state: {{S3MultipartUploadAbortRequest}}
> (since HDDS-14601) and {{S3ExpiredMultipartUploadsAbortRequest}} both
> continue when the open key is missing. FSO buckets are not affected, because
> the FSO complete request adds a missing open key back before it is read
> (checked by unit test).
> h3. Trigger
> # An upload on an OBJECT_STORE or LEGACY bucket has its
> {{multipartInfoTable}} entry but no open key. {{OpenKeyCleanupService}} in
> 1.3.x deleted the open key of uploads older than
> {{ozone.om.open.key.expire.threshold}} (default 7d) and left the
> {{multipartInfoTable}} entry. HDDS-9098 stopped that in 1.4.0, but entries
> created before survive the upgrade. The HDDS-9095 comments of 2026-02-09
> report such entries from a running cluster, which led to HDDS-14601.
> # A client sends CompleteMultipartUpload for that upload.
> I found no way to create the state with the current code through the client
> API or the background services. Since 1.4.0 {{MultipartUploadCleanupService}}
> aborts such uploads once they are older than
> {{ozone.om.open.mpu.expire.threshold}} (default 30d), so with default
> settings the entries disappear about a month after the upgrade. The report
> above shows that they can outlive that in practice (cause not investigated).
> h3. Impact
> The OM terminates while applying the request. The request is a committed
> Ratis log entry, so a restarted OM applies it again and terminates again (run
> on a single node OM). In an HA ring every OM applies the same entry (argued
> from the state machine code, not run). Nothing skips a failing log entry, so
> the OMs stay down until an operator intervenes (argued, not run).
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugCR4CompleteMissingOpenKey.java] ({{hadoop-ozone/ozone-manager}})
> drives one real OM with single node Ratis through {{OzoneManagerProtocol}}.
> The only step that is not a client call is the {{DeleteOpenKeys}} request
> that removes the open key, which is the request {{OpenKeyCleanupService}}
> submitted for such keys before HDDS-9098. The test shows the terminate for
> OBJECT_STORE and for LEGACY with a {{NullPointerException}} in
> {{S3MultipartUploadCompleteRequest.getOmKeyInfo}}, the second terminate on
> log replay after a restart, and the client abort succeeding on the same
> state. {{System.exit}} is disabled in the test JVM, so the terminate is
> observed as the recorded {{ExitUtils.terminate(1, ...)}} call. A passing test
> means the defect is present.
> h3. Patch
> [^CR-4-fail-multipart-complete-when-open-key-is-missing.patch], against
> d067ba38e7aea917db7efd290b4af53987ec985a. It also applies to master at
> a6332fb5987.
> {{getOmKeyInfo}} reads the open key once, before the two branches, and fails
> with {{NO_SUCH_MULTIPART_UPLOAD_ERROR}} when it is missing. The existing
> handler returns that as an error response, the OM keeps running, and the
> {{multipartInfoTable}} entry stays so the upload can still be aborted. The
> key is not rebuilt from the parts as it is for FSO, because for these layouts
> the ACLs, metadata, tags and encryption info of the upload are stored only in
> the open key. A patched OM that replays an already logged complete for such
> an upload returns the error instead of terminating (argued, not run).
> Covered by {{testValidateAndUpdateCacheWithOrphanMultipartInfo}} in the
> existing {{TestS3MultipartUploadCompleteRequest}}, without and with an
> already committed key, inherited by the FSO subclass. Without the change both
> OBJECT_STORE cases fail with the {{NullPointerException}}. The multipart
> request and response unit suites and the open key and multipart cleanup
> service suites (149 tests) pass, checkstyle is clean. No mini cluster suite
> was run. With the patch the three complete tests of the reproduction fail,
> because the client receives the error and the OM does not terminate.
> Found by TLA+ model checking and code review of the OM expired multipart
> upload cleanup paths under HDDS-15926, on commit
> d067ba38e7aea917db7efd290b4af53987ec985a. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
