By default for now, yes. One Spark Connect server handles multiple Spark
Sessions. To multiplex or run multiple Drivers, you need some work such as
gateway.
On Thu, 14 Dec 2023 at 12:03, Kezhi Xiong wrote:
> Hi,
>
> My understanding is there is only one driver/spark context for all user
>
Hi,
My understanding is there is only one driver/spark context for all user
sessions. When you run the bin/start-connect-server script, you are
submitting one long standing spark job / application. Every time a new user
request comes in, a new user session is created under that. Please correct
me
If multiple applications are running, we would need multiple spark connect
servers? If so, is the user responsible for creating these servers or they
are just created on the fly when the user requests a new spark session?
On Thu, Dec 14, 2023 at 10:28 AM Nikhil Goyal wrote:
> Hi folks,
> I am
Hi folks,
I am trying to understand one question. Does Spark Connect create a new
driver in the backend for every user or there are a fixed number of drivers
running to which requests are sent to?
Thanks
Nikhil
