Why do we need this? The reason data source APIs need it is because it will be
used by very unsophisticated end users and used all the time (for each
connection / query). Shuffle is something you set up once, presumably by fairly
sophisticated admins / engineers.
On Sat, Nov 04, 2023 at 2:42 PM, Alessandro Bellina < abell...@gmail.com >
wrote:
>
> Hello devs,
>
>
> I would like to start discussion on the SPIP "ShuffleManager short name
> registration via SparkPlugin"
>
>
> The idea behind this change is to allow a driver plugin (spark.plugins) to
> export ShuffleManagers via short names, along with sensible default
> configurations. Users can then use this short name to enable this
> ShuffleManager + configs using spark.shuffle.manager.
>
>
> SPIP: https:/ / docs. google. com/ document/ d/
> 1flijDjMMAAGh2C2k-vg1u651RItaRquLGB_sVudxf6I/
> edit#heading=h. vqpecs4nrsto (
> https://docs.google.com/document/d/1flijDjMMAAGh2C2k-vg1u651RItaRquLGB_sVudxf6I/edit#heading=h.vqpecs4nrsto
> )
> JIRA: https:/ / issues. apache. org/ jira/ browse/ SPARK-45792 (
> https://issues.apache.org/jira/browse/SPARK-45792 )
>
>
> I look forward to hearing your feedback.
>
>
> Thanks
>
>
> Alessandro
>
smime.p7s
Description: S/MIME Cryptographic Signature
