[ https://issues.apache.org/jira/browse/SPARK-35910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun resolved SPARK-35910. ----------------------------------- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33109 [https://github.com/apache/spark/pull/33109] > Update remoteBlockBytes based on merged block info > -------------------------------------------------- > > Key: SPARK-35910 > URL: https://issues.apache.org/jira/browse/SPARK-35910 > Project: Spark > Issue Type: Improvement > Components: Shuffle > Affects Versions: 3.2.0 > Reporter: Kent Yao > Assignee: Kent Yao > Priority: Major > Fix For: 3.2.0 > > > Currently, we calculate the `remoteBlockBytes` based on the original block > info list. If the original reducer size is big but the actual reducer size is > small due to automatically partition coalescing of AQE, the reducer will take > more time to calculate `remoteBlockBytes`. We can reduce this cost via remote > requests which contain merged block info lists. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org