Github user rdblue commented on the issue: https://github.com/apache/spark/pull/21977 Okay, workers are tracked in `PythonWorkerFactory`. Its `create` method returns an idle worker if one is available. When a task finishes, it calls `SparkEnv.releasePythonWorker` that calls `releaseWorker` on the factory to return the worker socket to the list of idle workers. So workers should grow to the number of concurrent tasks. I'll update this to always divide by the number of cores.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org