[ https://issues.apache.org/jira/browse/SPARK-47702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Jie resolved SPARK-47702. ------------------------------ Fix Version/s: 3.5.4 4.0.0 Resolution: Fixed Issue resolved by pull request 47779 [https://github.com/apache/spark/pull/47779] > Shuffle service endpoint is not removed from the locations list when RDD > block is removed form a node. > ------------------------------------------------------------------------------------------------------ > > Key: SPARK-47702 > URL: https://issues.apache.org/jira/browse/SPARK-47702 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 3.5.1 > Reporter: mahesh kumar behera > Priority: Major > Labels: pull-request-available > Fix For: 3.5.4, 4.0.0 > > > If SHUFFLE_SERVICE_FETCH_RDD_ENABLED is set to true, driver stores both > executor end point and the external shuffle end points for a RDD block. When > the RDD is migrated, the location info is updated to add the end point > corresponds to new location and the old end point is removed. But currently, > only the executor end point is removed. The shuffle service end point is not > removed. This cause failure during RDD read if the shuffle service end point > is chosen due to task locality. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org