Can I simply understand Spark Connect this way: The client process is now the Spark driver? ________________________________ From: Brian Huynh <brianhuy...@gmail.com> Sent: Thursday, August 10, 2023 10:15 PM To: Kezhi Xiong <kxi...@splunk.com.invalid> Cc: user@spark.apache.org <user@spark.apache.org> Subject: Re: Spark Connect, Master, and Workers
Hi Kezhi, Yes, you no longer need to start a master to make the client work. Please see the quickstart. https://spark.apache.org/docs/latest/api/python/getting_started/quickstart_connect.html<https://urldefense.com/v3/__https://spark.apache.org/docs/latest/api/python/getting_started/quickstart_connect.html__;!!DvJyiiI1!NhpY0PdG0PhEMLoUiL5aI9FgG8SuvOwlCk2B5WTmgGj-CoKTnwCCPQ1-UFNi-jAMPuyYwcqAsCxcwJ7dR-Y$> You can think of Spark Connect as an API on top of Master so workers can be added to the cluster same as before. https://blog.insightdatascience.com/simply-install-spark-cluster-mode-341843a52b88<https://urldefense.com/v3/__https://blog.insightdatascience.com/simply-install-spark-cluster-mode-341843a52b88__;!!DvJyiiI1!NhpY0PdG0PhEMLoUiL5aI9FgG8SuvOwlCk2B5WTmgGj-CoKTnwCCPQ1-UFNi-jAMPuyYwcqAsCxcf5w8Y18$> Regards, Brian On Wed, Aug 9, 2023 at 3:20 PM Kezhi Xiong <kxi...@splunk.com.invalid> wrote: Hi, I'm recently learning Spark Connect but have some questions regarding the connect server's relation with master or workers: so when I'm using the connect server, I don't have to start a master alone side to make clients work. Is the connect server simply using "local[*]" as master? Then, if I want to add workers for my connect server, is it supported and what should I do? Kezhi -- From Brian H.
