sunchao commented on PR #58380: URL: https://github.com/apache/spark/pull/58380#issuecomment-5469977214
**[P2] Preserve the extra executor with the StatefulSet allocator** At reviewed commit `9b194f91389bed3cdd5ace9509bb2d67ef5c4085`, the new condition in [ExecutorAllocationManager.scala](https://github.com/apache/spark/blob/9b194f91389bed3cdd5ace9509bb2d67ef5c4085/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L429-L435) can introduce executor allocation/deletion churn when dynamic allocation and speculation are enabled with `spark.kubernetes.allocation.pods.allocator=statefulset`. For example, with two 2-core executors, allocation ratio 1, and a maximum of at least 3 executors, one executor can be full while the other runs a slow task whose speculative copy cannot use the spare slot on the original host. Three running tasks plus one pending speculative copy gives `ceil(4 / 2) = 2`, so this branch raises the requested count to 3. Once the third executor registers, the equality check no longer holds and the next allocation update reduces the request to 2. Starting the copy does not prevent that reduction: four running tasks still give a calculated requirement of 2. [StatefulSetPodsAllocator](https://github.com/apache/spark/blob/9b194f91389bed3cdd5ace9509bb2d67ef5c4085/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/StatefulSetPodsAllocator.scala#L86-L112) directly applies that reduced request with `scale(expected, false)`. [StatefulSet scale-down](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#deployment-and-scaling-guarantees) can therefore delete the newly added highest-ordinal pod, potentially discarding the speculative attempt. Allocation/deletion can repeat while the original remains slow and no other eligible slot becomes available. The allocator's immediate downscale behavior predates this PR; the newly introduced part is the `2 -> 3 -> 2` request at unchanged task demand, where the base stays at 2. This concern is specific to the StatefulSet allocator, not the default direct allocator. Could we handle or guard this combination so the additional executor survives long enough to do useful speculative work? This is based on source inspection, not a local Kubernetes runtime reproduction. The earlier fixture and lint issues are fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
