Siyao Meng created HDDS-17388:
---------------------------------
Summary: Recon unhealthy container export returns 429 Too Many
Requests for a duplicate export, which retrying never clears
Key: HDDS-17388
URL: https://issues.apache.org/jira/browse/HDDS-17388
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-7-return-409-for-duplicate-export.patch,
TestBugCR7DuplicateExportStatus.java
h3. Mechanism
{{ExportJobManager.submitJob}} throws {{IllegalStateException}} in two cases:
an export for the requested state is already QUEUED, RUNNING or COMPLETED, or
the export queue is full. {{ContainerEndpoint.startExport}} maps every
{{IllegalStateException}} to {{429 Too Many Requests}} with {{"error": "Too
Many Requests"}}. The published {{ozone-default.xml}} ties this 429 to the
queue limit: {{ozone.recon.export.max.jobs.total}} says "Submissions beyond
this limit are rejected with HTTP 429." The in-tree {{ReconApi.md}} lists only
400 and 429 for {{POST /api/v1/containers/unhealthy/export}} and describes 429
as "the export queue is full; retry later". A queue full condition clears by
waiting, a duplicate does not: it lasts until the existing export is deleted,
the job fails or Recon restarts (jobs are kept in memory only).
{{downloadExport}} on the same resource already uses {{409 Conflict}} for its
"not completed yet" conflict. Its 429 for the per job download limit
({{ozone.recon.export.max.downloads}}) is a separate, documented case and is
out of scope here.
h3. Trigger
# {{POST /api/v1/containers/unhealthy/export?state=MISSING}}, wait until the
job is COMPLETED.
# Repeat the same request. Every repetition gets 429.
h3. Impact
A client that follows the documented meaning of 429 retries a request that can
never succeed. The Recon UI is not affected in practice: it blocks a duplicate
itself when its job list is current, shows the server's message for any error
status and does not retry (only its toast duration depends on 429). The futile
retry case is argued from the documented contract, not run with a real client.
h3. Reproduction
[^TestBugCR7DuplicateExportStatus.java] ({{hadoop-ozone/recon}}, package
{{org.apache.hadoop.ozone.recon.api}}) calls the real
{{ContainerEndpoint.startExport}} over a real {{ExportJobManager}}, with only
the DB collaborator mocked. It exports MISSING, waits for COMPLETED and repeats
the request three times; each gets status 429 with error "Too Many Requests"
and the message "An export for state MISSING already exists. Please delete the
existing export from the Completed Exports table before starting a new one."
The test passes on unmodified source, deterministic; a passing test means the
defect is present.
h3. Patch
[^CR-7-return-409-for-duplicate-export.patch], against
fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. It also applies to master at
9c3e5477ffb.
The duplicate check now throws {{ExportJobManager.DuplicateExportException}}, a
package private subclass of {{IllegalStateException}} so existing callers are
unaffected, and {{startExport}} answers it with {{409 Conflict}} and {{"error":
"Conflict"}}; the message is unchanged. Queue full keeps 429 with the same
body. The Recon UI needs no change: a 409 shows the same message.
Covered by
{{startExportReturnsConflictForDuplicateStateAndTooManyRequestsWhenQueueFull}}
in the existing {{TestExportJobManager}}, which checks 409 for a duplicate and
429 for a full queue through the endpoint. Without the change it fails with
{{expected: 409 but was: 429}}. With it {{TestExportJobManager}},
{{TestExportJob}} and {{TestContainerEndpoint}} (54 tests) pass and checkstyle
is clean. With the patch the reproduction fails at its first retry, which gets
409. The in-tree {{ReconApi.md}} and {{recon-api.yaml}} still list only 400 and
429 for this call and are not changed, since that docs tree is being retired.
The {{ozone-default.xml}} text stays correct, because 429 now means only that
the queue limit was hit.
Found by code review of the Recon unhealthy container export job manager, as
part of the TLA+ verification effort under HDDS-15926, on commit
fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
