As part of the discussion on this topic, I would love to highlight the work that the community is currently doing to support SparkML, which is traditionally very RDD-heavy, natively in Spark Connect. Bobby's awesome work shows that, over time, we can extend the features of Spark Connect and support workloads that we previously thought could not be supported easily.
https://github.com/apache/spark/pull/48791 Martin On Wed, Nov 27, 2024 at 11:42 AM Yang,Jie(INF) <[email protected]> wrote: > +1 > -------- 原始邮件 -------- > 发件人：Hyukjin Kwon<[email protected]> > 时间：2024-11-27 08:04:06 > 主题：[外部邮件] Re： Spark Connect the default API in Spark 4.0 > 收件人：Bjørn Jørgensen<[email protected]>; > 抄送人：Herman van Hovell<[email protected]>;Spark dev list< > [email protected]>; > +1 > > On Mon, 25 Nov 2024 at 23:33, Bjørn Jørgensen <[email protected]> > wrote: > >> +1 >> >> man. 25. nov. 2024 kl. 14:48 skrev Herman van Hovell >> <[email protected]>: >> >>> Hi All, >>> >>> I would like to start a discussion on "Spark Connect the default API in >>> Spark 4.0". >>> >>> The rationale for this change is that Spark Connect brings a lot of >>> improvements with respect to simplicity, stability, isolation, >>> upgradability, and extensibility (all detailed in the SPIP). In a nutshell: >>> we want to introduce a flag, spark.api.mode, that allows a user to >>> choose between classic or connect mode, the default being connect. A >>> user can easily fallback to Classic by setting spark.api.mode to classic >>> . >>> >>> SPIP: >>> https://docs.google.com/document/d/1C0kuQEliG78HujVwdnSk0wjNwHEDdwo2o8aVq7kbhTo/edit?tab=t.0#heading=h.r2c3xrbiklu3 >>> <https://mailshield.baidu.com/check?q=5uIK5BsJhkKEitTyTno8Yb7Zq%2boLHvRsgSoBr5oTNJEHXWS9Np0U8pCuv2DeJDfCQJiI52FAoCrxDEqnj1jOqX9A3jtJcetvkKkKE696xfrLfKuuRuyCC9YrwN5IW4OUtkhdHz7C%2bER2GN9EPqnlIlX2osm36Zbn> >>> JIRA: https://issues.apache.org/jira/browse/SPARK-50411 >>> <https://mailshield.baidu.com/check?q=vc5arXeK3OKfjk5Oxe1F%2fMNjR%2fSx5pTdbaOArWe9m2MpZDOF702CYYagPMQmbDqV7xnWwxsUdOc%3d> >>> >>> I am looking forward to your feedback! >>> >>> Cheers, >>> Herman >>> >> >> >> -- >> Bjørn Jørgensen >> Vestre Aspehaug 4, 6010 Ålesund >> Norge >> >> +47 480 94 297 >> >
