Alessandro Bellina created SPARK-45792:
------------------------------------------
Summary: SPIP: ShuffleManager short name registration via
SparkPlugin
Key: SPARK-45792
URL: https://issues.apache.org/jira/browse/SPARK-45792
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.0.0
Reporter: Alessandro Bellina
Fix For: 4.0.0
We would like to make it possible for a SparkPlugin to expose custom
ShuffleManager implementations using short names, in addition to default
configurations for each short name in order to improve ease of use.
Today, users leveraging a spark plugin need to set a class name under
spark.plugins and if the plugin also provides a ShuffleManager, the user needs
to set spark.shuffle.manager to a fully qualified class name. Additionally,
users need to make sure they have set Spark configurations that are required or
recommended for the shuffle manager instance. This can be cumbersome for the
user, adding barriers to the pluggable interface that Spark provides.
Spark provides a short name for SortShuffleManager (“sort”) today. The idea in
this SPIP is to make this set of short names extensible by a SparkPlugin.
SPIP:
https://docs.google.com/document/d/1flijDjMMAAGh2C2k-vg1u651RItaRquLGB_sVudxf6I/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
