shuan1026 opened a new pull request, #11989:
URL: https://github.com/apache/ozone/pull/11989

   ## What changes were proposed in this pull request?
   
   `ExportJobManager.executeExport` creates the job directory with 
`Files.createDirectories` and only then removes the job from `jobQueue`. When 
directory creation throws an `IOException` (wrong permissions, a read-only 
remount, or a regular file in place of the export directory, which is also what 
happens when Recon could not create the directory at startup), the catch marks 
the job `FAILED` and deletes its files, but the job stays in `jobQueue`; the 
`finally` only clears `runningTasks`.
   
   `submitJob` rejects a submission once `jobQueue.size()` reaches 
`ozone.recon.export.max.jobs.total` (default 4), and its duplicate check 
ignores `FAILED` jobs, so every such failure keeps one slot. After four 
failures every new export gets HTTP 429 `Export queue is full (max 4 jobs)` 
although nothing is queued or running, and retrying never helps once the fault 
is fixed. Because `jobQueue` is insertion ordered, the stale entries also 
inflate `getQueuePosition` of later queued jobs. The only ways out are deleting 
each `FAILED` row in the UI or restarting Recon.
   
   This patch moves the removal from `jobQueue` to the start of 
`executeExport`, ahead of `Files.createDirectories`, so the job leaves the 
queue before anything in the worker can fail. The success path is unchanged: 
the job still becomes `RUNNING` after its directory exists, and `cancelJob`'s 
"no-op for COMPLETED/FAILED jobs already off the queue" now holds for every 
`FAILED` job. Counting only `QUEUED`/`RUNNING` jobs in `submitJob` was not 
chosen because it changes what the limit counts and leaves the queue positions 
inflated.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-17391
   
   ## How was this patch tested?
   
   - New `TestExportJobManager#jobDirectoryCreationFailureFreesQueueSlot`: with 
a regular file in place of the export directory and 
`ozone.recon.export.max.jobs.total=1`, the first export ends `FAILED` with the 
export path in its error message and must have no queue position; after the 
directory is restored, a second export is accepted and completes. It fails with 
`expected: 0 but was: 1` before the change (the second submit would otherwise 
be rejected with `Export queue is full (max 1 jobs)`) and passes after it.
   - CI: https://github.com/shuan1026/ozone/actions/runs/42613119322


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to