Can I simply understand Spark Connect this way: The client process is now the
Spark driver?
From: Brian Huynh
Sent: Thursday, August 10, 2023 10:15 PM
To: Kezhi Xiong
Cc: user@spark.apache.org
Subject: Re: Spark Connect, Master, and Workers
Hi Kezhi,
Yes, you no longer need to start a master to make the client work. Please see
the quickstart.
https://spark.apache.org/docs/latest/api/python/getting_started/quickstart_connect.html<https://urldefense.com/v3/__https://spark.apache.org/docs/latest/api/python/getting_started/quickstart_connect.html__;!!DvJyiiI1!NhpY0PdG0PhEMLoUiL5aI9FgG8SuvOwlCk2B5WTmgGj-CoKTnwCCPQ1-UFNi-jAMPuyYwcqAsCxcwJ7dR-Y$>
You can think of Spark Connect as an API on top of Master so workers can be
added to the cluster same as before.
https://blog.insightdatascience.com/simply-install-spark-cluster-mode-341843a52b88<https://urldefense.com/v3/__https://blog.insightdatascience.com/simply-install-spark-cluster-mode-341843a52b88__;!!DvJyiiI1!NhpY0PdG0PhEMLoUiL5aI9FgG8SuvOwlCk2B5WTmgGj-CoKTnwCCPQ1-UFNi-jAMPuyYwcqAsCxcf5w8Y18$>
Regards,
Brian
On Wed, Aug 9, 2023 at 3:20 PM Kezhi Xiong wrote:
Hi,
I'm recently learning Spark Connect but have some questions regarding the
connect server's relation with master or workers: so when I'm using the connect
server, I don't have to start a master alone side to make clients work. Is the
connect server simply using "local[*]" as master? Then, if I want to add
workers for my connect server, is it supported and what should I do?
Kezhi
--
From Brian H.