Siyao Meng created HDDS-17392:
---------------------------------
Summary: Recon export job stays RUNNING forever after an Error
such as OutOfMemoryError, and new exports for its state are rejected
Key: HDDS-17392
URL: https://issues.apache.org/jira/browse/HDDS-17392
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-3-mark-export-job-failed-on-error.patch
h3. Mechanism
{{ExportJobManager.executeExport}} handles only {{IOException}} and
{{RuntimeException}} (plus {{InterruptedException}} for cancellation). An
{{Error}} thrown while the job runs, for example {{OutOfMemoryError}}, skips
the code that marks the job FAILED; the {{finally}} only clears
{{runningTasks}}. The task is a {{FutureTask}} whose result nobody reads, so
the Error is not logged either. The job stays RUNNING in the job tracker, and
the duplicate check in {{ExportJobManager.submitJob}} refuses every new export
for that state.
h3. Trigger
# An export is running.
# An {{Error}} is thrown inside the worker, for example the heap runs out while
rows are written. Recon keeps running afterwards (Ozone sets no exit on
OutOfMemoryError by default).
h3. Impact
* The job shows RUNNING with frozen progress indefinitely. Every new export
request for that state is refused: REST API callers get HTTP 429 "An export for
state ... already exists", and the UI shows the same warning without sending
the request.
* The UI cannot clear it: the Active Exports table has no Delete or Cancel, and
the message points to the Completed Exports table, where the job never appears.
Only {{DELETE /api/v1/containers/unhealthy/export/<jobId>}} or a Recon restart
clears it.
* Exports for other states still work; the worker thread survives.
h3. Reproduction
PASS on the unmodified source, deterministic.
[^TestBugMC3ErrorStrandsExportJob.java] ({{hadoop-ozone/recon}}) uses only
public {{ExportJobManager}} methods; the mocked DB layer throws
{{OutOfMemoryError}} from the first count query, standing in for heap
exhaustion. After a second job for another state has completed on the single
worker (so the first task has ended), the first job is still RUNNING with no
error message, a new export for its state is rejected with "already exists",
and only {{cancelJob}} lets a new one through. Nothing about the Error appears
in the log. A real heap exhaustion was not used in this test.
h3. Patch
[^MC-3-mark-export-job-failed-on-error.patch], against
fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. It also applies to master at
9c3e5477ffb.
The patch adds {{Error}} to the existing {{catch (IOException |
RuntimeException e)}}, so an Error gets the same handling as an exception: the
job is marked FAILED with the message, its partial files are removed, and the
Error is logged once with its stack trace. It is not rethrown; a rethrow would
only be stored in the task's {{Future}}, which nothing reads, so the log line
is where it is reported.
Covered by the new
{{TestExportJobManager#errorInWorkerMarksJobFailedAndAllowsRetry}}, which fails
without the change ("Timed out waiting for job ... to reach FAILED; last
observed=RUNNING"). With the change {{TestExportJobManager}}, {{TestExportJob}}
and {{TestContainerEndpoint}} pass and checkstyle is clean. With the patch the
reproduction fails because the job is FAILED with the Error's message.
Not covered: an {{Error}} in the few assignments before the {{try}} would still
leave the job QUEUED. An {{Error}} while the job directory is created (before
the job leaves the queue) now marks the job FAILED but leaves it in the queue,
as an {{IOException}} there does; HDDS-17391 fixes that.
Related: HDDS-17046 (the same pattern, an {{Error}} leaving a RUNNING state
behind, in the NSSummary rebuild).
Found by TLA+ model checking and code review of the Recon unhealthy container
export job manager under HDDS-15926, on commit
fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
