akshaychitneni opened a new pull request, #2279: URL: https://github.com/apache/datafusion-ballista/pull/2279
Which issue does this PR close? Follow-up to #2029 / #2212, addressing the timeout interaction surfaced by #2240 (https://github.com/apache/datafusion-ballista/pull/2240#discussion_r3732320008) Rationale for this change The empty-cluster grace timer defaulted to 30s, but the executor heartbeat interval defaults to 60s. Since #2240 routes task-launch-failure removals through the same ExecutorLost path (aggressive removal that also drops the heartbeat), a healthy executor removed by a transient blip re-registers only on its next heartbeat (~≤60s) — which the 30s timer could beat, spuriously failing the job. What changes are included in this PR? no_executors_grace_period_seconds now defaults to executor_timeout_seconds (guaranteed longer than the heartbeat interval; the scheduler can't read the executor's heartbeat setting directly). CLI arg is Option, resolving to executor_timeout_seconds when unset; SchedulerConfig::Default matches. Removal stays immediate (fast rescheduling); only the fail decision waits, and the timer's re-check still cancels the failure if the executor re-registers. Explicit values (incl. 0 = fail-fast) honored. Adds config tests (default resolution + grace >= executor_timeout invariant). Are there user-facing changes? Yes — the default no-executors grace period changes from 30s to executor_timeout_seconds (180s by default); --no-executors-grace-period-seconds is now optional and defaults to that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
