[
https://issues.apache.org/jira/browse/SPARK-36964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17426523#comment-17426523
]
Apache Spark commented on SPARK-36964:
--------------------------------------
User 'gaoyajun02' has created a pull request for this issue:
https://github.com/apache/spark/pull/34231
> Reuse CachedDNSToSwitchMapping for yarn container requests
> -----------------------------------------------------------
>
> Key: SPARK-36964
> URL: https://issues.apache.org/jira/browse/SPARK-36964
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, YARN
> Affects Versions: 3.0.3, 3.1.2
> Reporter: gaoyajun02
> Priority: Major
>
> Similar to SPARK-13704, In some cases, YarnAllocator add or remove container
> requests can be expensive, it may call the topology script for rack awareness.
> When submit a very large job in a very large Yarn cluster, the topology
> script may take signifiant time to run. And this blocks receiving
> YarnSchedulerBackend's RequestExecutors rpc calls, This request comes from
> spark dynamic executor allocation thread, which may blocks the
> ExecutorAllocationListener, and then result in executorManagement queue
> backlog.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
