[ https://issues.apache.org/jira/browse/SPARK-45792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782931#comment-17782931 ]
Alessandro Bellina commented on SPARK-45792: -------------------------------------------- Added a link to an issue that needs to be solved in order for this SPIP to move forward. > SPIP: ShuffleManager short name registration via SparkPlugin > ------------------------------------------------------------ > > Key: SPARK-45792 > URL: https://issues.apache.org/jira/browse/SPARK-45792 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 4.0.0 > Reporter: Alessandro Bellina > Priority: Minor > Fix For: 4.0.0 > > > We would like to make it possible for a SparkPlugin to expose custom > ShuffleManager implementations using short names, in addition to default > configurations for each short name in order to improve ease of use. > Today, users leveraging a spark plugin need to set a class name under > spark.plugins and if the plugin also provides a ShuffleManager, the user > needs to set spark.shuffle.manager to a fully qualified class name. > Additionally, users need to make sure they have set Spark configurations that > are required or recommended for the shuffle manager instance. This can be > cumbersome for the user, adding barriers to the pluggable interface that > Spark provides. > Spark provides a short name for SortShuffleManager (“sort”) today. The idea > in this SPIP is to make this set of short names extensible by a SparkPlugin. > SPIP: > https://docs.google.com/document/d/1flijDjMMAAGh2C2k-vg1u651RItaRquLGB_sVudxf6I/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org