[
https://issues.apache.org/jira/browse/SPARK-50277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-50277.
-----------------------------------
Resolution: Invalid
According to the above discussion, I'm closing this as an invalid issue . You
can reopen this if this is a valid one.
> [k8s] Apply for executor pods in parallel
> -----------------------------------------
>
> Key: SPARK-50277
> URL: https://issues.apache.org/jira/browse/SPARK-50277
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 3.5.1
> Reporter: Bowen
> Priority: Major
>
> The performance of spark on k8s is worse than that of yarn. It is found that
> the application of executor pod is executed sequentially. The k8s interface
> for applying pod is
> kubernetesClient.pods().inNamespace(namespace).resource(podWithAttachedContainer).create(),
> which is asynchronous. However, each execution still takes an average of
> 62.57ms. Applying 280 pods takes 17520ms, which means that the speed of
> applying pod is about 15-16 pods/second. If a job requires more executors,
> this speed will become a bottleneck. I would like to ask whether this logic
> can be changed to concurrently apply for executor pods, and whether there
> will be any negative impact.
> The logic of applying for executor is in method：
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator#requestNewExecutors
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
