[
https://issues.apache.org/jira/browse/HDDS-17392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17392:
------------------------------
Affects Version/s: 2.2.1
2.2.0
> Recon export job stays RUNNING forever after an Error such as
> OutOfMemoryError, and new exports for its state are rejected
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17392
> URL: https://issues.apache.org/jira/browse/HDDS-17392
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.2.0, 2.2.1
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: MC-3-mark-export-job-failed-on-error.patch
>
>
> h3. Mechanism
> {{ExportJobManager.executeExport}} handles only {{IOException}} and
> {{RuntimeException}} (plus {{InterruptedException}} for cancellation). An
> {{Error}} thrown while the job runs, for example {{OutOfMemoryError}}, skips
> the code that marks the job FAILED; the {{finally}} only clears
> {{runningTasks}}. The task is a {{FutureTask}} whose result nobody reads, so
> the Error is not logged either. The job stays RUNNING in the job tracker, and
> the duplicate check in {{ExportJobManager.submitJob}} refuses every new
> export for that state.
> h3. Trigger
> # An export is running.
> # An {{Error}} is thrown inside the worker, for example the heap runs out
> while rows are written. Recon keeps running afterwards (Ozone sets no exit on
> OutOfMemoryError by default).
> h3. Impact
> * The job shows RUNNING with frozen progress indefinitely. Every new export
> request for that state is refused: REST API callers get HTTP 429 "An export
> for state ... already exists", and the UI shows the same warning without
> sending the request.
> * The UI cannot clear it: the Active Exports table has no Delete or Cancel,
> and the message points to the Completed Exports table, where the job never
> appears. Only {{DELETE /api/v1/containers/unhealthy/export/<jobId>}} or a
> Recon restart clears it.
> * Exports for other states still work; the worker thread survives.
> h3. Reproduction
> PASS on the unmodified source, deterministic.
> [^TestBugMC3ErrorStrandsExportJob.java] ({{hadoop-ozone/recon}}) uses only
> public {{ExportJobManager}} methods; the mocked DB layer throws
> {{OutOfMemoryError}} from the first count query, standing in for heap
> exhaustion. After a second job for another state has completed on the single
> worker (so the first task has ended), the first job is still RUNNING with no
> error message, a new export for its state is rejected with "already exists",
> and only {{cancelJob}} lets a new one through. Nothing about the Error
> appears in the log. A real heap exhaustion was not used in this test.
> h3. Patch
> [^MC-3-mark-export-job-failed-on-error.patch], against
> fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. It also applies to master at
> 9c3e5477ffb.
> The patch adds {{Error}} to the existing {{catch (IOException |
> RuntimeException e)}}, so an Error gets the same handling as an exception:
> the job is marked FAILED with the message, its partial files are removed, and
> the Error is logged once with its stack trace. It is not rethrown; a rethrow
> would only be stored in the task's {{Future}}, which nothing reads, so the
> log line is where it is reported.
> Covered by the new
> {{TestExportJobManager#errorInWorkerMarksJobFailedAndAllowsRetry}}, which
> fails without the change ("Timed out waiting for job ... to reach FAILED;
> last observed=RUNNING"). With the change {{TestExportJobManager}},
> {{TestExportJob}} and {{TestContainerEndpoint}} pass and checkstyle is clean.
> With the patch the reproduction fails because the job is FAILED with the
> Error's message.
> Not covered: an {{Error}} in the few assignments before the {{try}} would
> still leave the job QUEUED. An {{Error}} while the job directory is created
> (before the job leaves the queue) now marks the job FAILED but leaves it in
> the queue, as an {{IOException}} there does; HDDS-17391 fixes that.
> Related: HDDS-17046 (the same pattern, an {{Error}} leaving a RUNNING state
> behind, in the NSSummary rebuild).
> Found by TLA+ model checking and code review of the Recon unhealthy container
> export job manager under HDDS-15926, on commit
> fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
