Jiwon Park created SPARK-58446:
----------------------------------
Summary: Dynamic allocation can remain at zero executors after
TaskStart events arrive after StageCompleted
Key: SPARK-58446
URL: https://issues.apache.org/jira/browse/SPARK-58446
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 5.0.0
Reporter: Jiwon Park
When dynamic allocation is enabled with spark.dynamicAllocation.minExecutors=0,
an application can stop requesting executors after TaskStart events are
processed after StageCompleted for the same stage attempt.
The completed stage can contribute a negative pending task count. After all
executors are removed by idle timeout, later jobs can remain pending because
the target executor count stays at 0 and no request is sent to the cluster
manager.
Reproduction event sequence:
StageSubmitted -> TaskStart -> StageCompleted -> late TaskStart -> TaskEnd ->
new StageSubmitted
Expected behavior:
Completed stage attempts do not contribute to pending task counts, and
submitting a new stage increases the executor target.
Actual behavior:
The pending task count can remain negative, causing the executor target to stay
at 0 and preventing new tasks from starting.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
