Min Shen created SPARK-32920:
--------------------------------
Summary: Add support in Spark driver to coordinate the
finalization of the push/merge phase in push-based shuffle for a given shuffle
and the initiation of the reduce stage
Key: SPARK-32920
URL: https://issues.apache.org/jira/browse/SPARK-32920
Project: Spark
Issue Type: Sub-task
Components: Shuffle, Spark Core
Affects Versions: 3.1.0
Reporter: Min Shen
With push-based shuffle, we are currently decoupling map task executions from
the shuffle block push process. Thus, when all map tasks finish, we might want
to wait for some small extra time to allow more shuffle blocks to get pushed
and merged. This requires some extra coordination in the Spark driver when it
transitions from a shuffle map stage to the corresponding reduce stage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
