Why do we need this? The reason data source APIs need it is because it will be
used by very unsophisticated end users and used all the time (for each
connection / query). Shuffle is something you set up once, presumably by fairly
sophisticated admins / engineers.
On Sat, Nov 04, 2023 at 2:42
Hello devs,
I would like to start discussion on the SPIP "ShuffleManager short name
registration via SparkPlugin"
The idea behind this change is to allow a driver plugin (spark.plugins) to
export ShuffleManagers via short names, along with sensible default
configurations. Users can then use this
