[
https://issues.apache.org/jira/browse/HDDS-17393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17393:
------------------------------
Attachment: TestBugMC5ExportLimitExcludesRunningJob.java
> Description of ozone.recon.export.max.jobs.total says it counts the executing
> export job, but only waiting jobs are counted
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17393
> URL: https://issues.apache.org/jira/browse/HDDS-17393
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.2.0, 2.2.1
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: MC-5-describe-export-job-limit-as-waiting-jobs.patch,
> TestBugMC5ExportLimitExcludesRunningJob.java
>
>
> h3. Mechanism
> The description of {{ozone.recon.export.max.jobs.total}} in
> {{ozone-default.xml}} and the Javadoc of
> {{ReconServerConfigKeys.OZONE_RECON_EXPORT_MAX_JOBS_TOTAL}} say it is the
> maximum number of export jobs "(waiting + executing)".
> {{ExportJobManager.submitJob}} compares the limit with {{jobQueue.size()}},
> and {{ExportJobManager.executeExport}} removes a job from {{jobQueue}} when
> it starts, so the running job is not counted. The limit applies to waiting
> jobs only; one running job can exist on top of it. Until HDDS-17391 is fixed,
> a job whose directory creation failed also stays in {{jobQueue}} and counts
> against the limit, although it is neither waiting nor executing.
> {{TestExportJobManager#submitQueueFullThrows}} asserts the code's behaviour:
> with the limit at 1 it accepts one running and one waiting job and rejects
> the third.
> h3. Trigger
> # One export is running.
> # Further exports for other states are submitted.
> h3. Impact
> With the default of 4, five jobs are accepted (one RUNNING, four QUEUED) and
> only the sixth gets HTTP 429. Operators reading the description expect four.
> This is reachable through the REST API; the UI offers five states and allows
> one live job per state, so a UI user does not reach the limit at the default.
> There is no other effect: a single worker runs the jobs, so the extra job
> only waits.
> h3. Reproduction
> PASS on the unmodified source, deterministic.
> [^TestBugMC5ExportLimitExcludesRunningJob.java] ({{hadoop-ozone/recon}})
> holds one job RUNNING on a blocking mocked cursor, submits four more, and
> checks that 5 jobs are QUEUED or RUNNING and that the sixth is rejected with
> "Export queue is full (max 4 jobs)".
> h3. Suggested fix
> Correct the description rather than the code. Counting the executing job
> would lower the effective limit for every deployment (from 5 active jobs to 4
> at the default), change the behaviour of the 2.2 releases and contradict the
> existing unit test, for no operational gain. If the description is considered
> the contract instead, the alternative is to count QUEUED and RUNNING jobs in
> {{submitJob}} and adjust {{submitQueueFullThrows}}.
> h3. Patch
> [^MC-5-describe-export-job-limit-as-waiting-jobs.patch], against
> fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. It also applies to master at
> 9c3e5477ffb.
> Documentation only: the first sentence of the description becomes "Maximum
> number of export jobs waiting in the queue at once, not counting the job that
> is executing.", and the Javadoc line matches. No code or test change;
> {{submitQueueFullThrows}} already covers the behaviour. The reproduction
> passes unchanged with the patch.
> Found by TLA+ model checking and code review of the Recon unhealthy container
> export job manager under HDDS-15926, on commit
> fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
