zhoubin created SPARK-45463:
-------------------------------
Summary: Allow ShuffleDriverComponent to decide weather shuffle
data is reliably stored when different stage has different policy
Key: SPARK-45463
URL: https://issues.apache.org/jira/browse/SPARK-45463
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.0.0, 3.5.1
Reporter: zhoubin
After SPARK-42689, ShuffleDriverComponent.supportsReliableStorage is determined
globally.
Downstream projects may have different shuffle policy to adapt to cluster loads
for different stages, for example or Apache Uniffle with Gluten or Apache
Celeborn. In this situation, ShuffleDriverComponent should use the
mapTrackerMaster to decide weather support reliable storage by the specified
executorId
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
