[
https://issues.apache.org/jira/browse/SPARK-33037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BoYang updated SPARK-33037:
---------------------------
Description:
Spark has a hardcode list to contain known shuffle managers, which has two
values now. It does not contain user's custom shuffle manager which is set
through Spark config "spark.shuffle.manager".
We hit issue when set "spark.shuffle.manager" with our own shuffle manager
plugin (Uber Remote Shuffle Service implementation,
[https://github.com/uber/RemoteShuffleService]). Other users will hit same
issue when they implement their own shuffle manager.
Need to add "spark.shuffle.manager" config value to the known managers list as
well.
The know managers list is in code:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
{quote}private final List<String> knownManagers = Arrays.asList(
"org.apache.spark.shuffle.sort.SortShuffleManager",
"org.apache.spark.shuffle.unsafe.UnsafeShuffleManager");
{quote}
was:
Spark has a hardcode list to contain known shuffle managers, which has two
values now. It does not contain user's custom shuffle manager which is set
through Spark config "spark.shuffle.manager".
We hit issue when set "spark.shuffle.manager" with our own shuffle manager
plugin (Uber Remote Shuffle Service implementation,
https://github.com/uber/RemoteShuffleService).
Need to add "spark.shuffle.manager" config value to the known managers list as
well.
The know managers list is in code:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
{quote}private final List<String> knownManagers = Arrays.asList(
"org.apache.spark.shuffle.sort.SortShuffleManager",
"org.apache.spark.shuffle.unsafe.UnsafeShuffleManager");
{quote}
> Add "spark.shuffle.manager" value to knownManagers
> --------------------------------------------------
>
> Key: SPARK-33037
> URL: https://issues.apache.org/jira/browse/SPARK-33037
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle
> Affects Versions: 2.4.7, 3.0.1
> Reporter: BoYang
> Priority: Major
>
> Spark has a hardcode list to contain known shuffle managers, which has two
> values now. It does not contain user's custom shuffle manager which is set
> through Spark config "spark.shuffle.manager".
>
> We hit issue when set "spark.shuffle.manager" with our own shuffle manager
> plugin (Uber Remote Shuffle Service implementation,
> [https://github.com/uber/RemoteShuffleService]). Other users will hit same
> issue when they implement their own shuffle manager.
>
> Need to add "spark.shuffle.manager" config value to the known managers list
> as well.
>
> The know managers list is in code:
> common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
> {quote}private final List<String> knownManagers = Arrays.asList(
> "org.apache.spark.shuffle.sort.SortShuffleManager",
> "org.apache.spark.shuffle.unsafe.UnsafeShuffleManager");
> {quote}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
