Peiyu Zhuang created SPARK-26380:
------------------------------------
Summary: ExternalShuffleBlockResolver should not block shuffle
managers by name.
Key: SPARK-26380
URL: https://issues.apache.org/jira/browse/SPARK-26380
Project: Spark
Issue Type: Bug
Components: Shuffle
Affects Versions: 2.3.2
Reporter: Peiyu Zhuang
We configure a shuffle manager that is compatible with the original Spark
shuffle manager (but with other network/storage implementation) in our
environment. It doesn't require any modification to Spark source code and It
works fine until we enable dynamic allocation. The problem is that in
{{ExternalShuffleBlockResolver}} class it checks the name of the shuffle
manager class and raise an {{UnsupportedOperationException}} if shuffle manager
is not {{SortShuffleManager}} or {{UnsafeShuffleManager}}.
Since shuffle manager is totally configurable, I think user should be able to
decide which shuffle manager to use when dynamic allocation is enabled.
Maybe we could change {{UnsupportedOperationException}} to a warning log
telling user that the shuffle manager they are using may not be compatible with
dynamic allocation and gives them detail reasons why. On the other hand, user
may still choose to use their own shuffle manager while knowing the risk behind
it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
