[ https://issues.apache.org/jira/browse/SPARK-38401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
XiDuo You updated SPARK-38401: ------------------------------ Description: It has several issues with method `ShuffledRowRDD#getPreferredLocations`. * it does not respect the config `spark.shuffle.reduceLocality.enabled`, so we can not disable it. * it does not respect `REDUCER_PREF_LOCS_FRACTION`, so it has no effect if DAG schedule task to an executor who has less data. In worse, driver will take more memory to store the useless locations. was: It has several issues in the method `getPreferredLocations` of `ShuffledRowRDD`. * it does not respect the config `spark.shuffle.reduceLocality.enabled`, so we can not disable it. * it does not respect `REDUCER_PREF_LOCS_FRACTION`, so it has no effect if DAG schedule task to an executor who has less data. In worse, driver will take more memory to store the useless locations. > Unify get preferred locations for shuffle in AQE > ------------------------------------------------ > > Key: SPARK-38401 > URL: https://issues.apache.org/jira/browse/SPARK-38401 > Project: Spark > Issue Type: Sub-task > Components: SQL > Affects Versions: 3.3.0 > Reporter: XiDuo You > Priority: Major > > It has several issues with method `ShuffledRowRDD#getPreferredLocations`. > * it does not respect the config `spark.shuffle.reduceLocality.enabled`, so > we can not disable it. > * it does not respect `REDUCER_PREF_LOCS_FRACTION`, so it has no effect if > DAG schedule task to an executor who has less data. In worse, driver will > take more memory to store the useless locations. > -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org