[ https://issues.apache.org/jira/browse/SPARK-39023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun closed SPARK-39023. --------------------------------- > Add Executor Pod inter-pod anti-affinity > ---------------------------------------- > > Key: SPARK-39023 > URL: https://issues.apache.org/jira/browse/SPARK-39023 > Project: Spark > Issue Type: New Feature > Components: Kubernetes > Affects Versions: 3.2.1 > Reporter: binjie yang > Priority: Major > > h3. Why should we need this? > When Spark On Kubernetes is running, Executor Pod clusters occur in certain > conditions (uneven resource allocation in Kubernetes, high load On some > nodes, low load On some nodes), causing Shuffle data skew. This causes Spark > application to fail or performance bottlenecks, such as Shuffle Fetch timeout > and connection refuse after connection number. > h3. Why should use this? > The functionality mentioned in this PR was tested on a cluster. > Using three Kubernetes Node(node-1, node-2, node-3). > In the case of sufficient or insufficient cluster resources, it has the same > effect Whether the feature is enabled or not. Kubernetes assigns pods to > nodes with low load based on global resources. When only one node has a small > load, Kubernetes will schedule all executor pods to this node. -- This message was sent by Atlassian Jira (v8.20.7#820007) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org