[
https://issues.apache.org/jira/browse/SPARK-31173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062785#comment-17062785
]
Jiaxin Shan commented on SPARK-31173:
-------------------------------------
I am trying to get more details.
There's two level performance issues.
# As every pod need to be mutated by webhook, it drags down to overall
throughput.
# Nodeselector, tolerations or node affinities have impact on kubernetes
scheduler performance.
Could I understand if you benchmark difference reflects both of above two
points?
BTW, Tolerations should be supported in PodTemplate in 3.0.0 release.
> Spark Kubernetes add tolerations and nodeName support
> -----------------------------------------------------
>
> Key: SPARK-31173
> URL: https://issues.apache.org/jira/browse/SPARK-31173
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes
> Affects Versions: 3.1.0, 2.4.6
> Environment: Alibaba Cloud ACK with spark
> operator(v1beta2-1.1.0-2.4.5) and spark(2.4.5)
> Reporter: zhongwei liu
> Priority: Trivial
> Labels: features
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> When you run spark on serverless kubernetes cluster(virtual-kubelet). you
> need to specific the nodeSelectors,tolerations even nodeName when you want to
> gain better scheduling performance. Currently spark doesn't support
> tolerations. If you want to use this feature, You must use admission
> controller webhook to decorate the pod. But the performance is extremely bad.
> Here is the benchmark.
> With webhook
> Batch Size: 500 Pod creation: about 7 Pods/s All Pods running: 5min
> Without webhook
> Batch Size: 500 Pod creation: more than 500 Pods/s All Pods running: 45s
> Adding tolerations and nodeName in spark will bring great help when you want
> to run a large scale job on serverless kubernetes cluster.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
