Min Shen created SPARK-32922:
--------------------------------
Summary: Add support for ShuffleBlockFetcherIterator to read from
merged shuffle partitions and to fallback to original shuffle blocks if
encountering failures
Key: SPARK-32922
URL: https://issues.apache.org/jira/browse/SPARK-32922
Project: Spark
Issue Type: Sub-task
Components: Shuffle, Spark Core
Affects Versions: 3.1.0
Reporter: Min Shen
With the extended MapOutputTracker, the reducers can now get the task input
data from the merged shuffle partitions for more efficient shuffle data fetch.
The reducers should also be able to fallback to fetching the original unmarked
blocks if it encounters failures when fetching the merged shuffle partitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
