Hello devs,
We are going to be tabling the SPIP proposal given that we don't see
responses in the discussion thread. We still believe that making custom
ShuffleManagers easier to configure is worthwhile, given interactions with
our users, but we can revisit this later. If anyone in the list has
Thanks for the comments Reynold. This is an ease of use change, and it is
not absolutely required (as other ease of use changes are not required
either). That said, do we not want to invest in making Spark easier to
configure for the average user, or even the user that is trying out Spark?
Here
Why do we need this? The reason data source APIs need it is because it will be
used by very unsophisticated end users and used all the time (for each
connection / query). Shuffle is something you set up once, presumably by fairly
sophisticated admins / engineers.
On Sat, Nov 04, 2023 at 2:42
Hello devs,
I would like to start discussion on the SPIP "ShuffleManager short name
registration via SparkPlugin"
The idea behind this change is to allow a driver plugin (spark.plugins) to
export ShuffleManagers via short names, along with sensible default
configurations. Users can then use this
