By default for now, yes. One Spark Connect server handles multiple Spark Sessions. To multiplex or run multiple Drivers, you need some work such as gateway.
On Thu, 14 Dec 2023 at 12:03, Kezhi Xiong <kxi...@splunk.com.invalid> wrote: > Hi, > > My understanding is there is only one driver/spark context for all user > sessions. When you run the bin/start-connect-server script, you are > submitting one long standing spark job / application. Every time a new user > request comes in, a new user session is created under that. Please correct > me if I am wrong. > > Kezhi > > On Thu, Dec 14, 2023 at 10:35 AM Nikhil Goyal <nownik...@gmail.com> wrote: > >> [ External sender. Exercise caution. ] >> >> If multiple applications are running, we would need multiple spark >> connect servers? If so, is the user responsible for creating these servers >> or they are just created on the fly when the user requests a new spark >> session? >> >> On Thu, Dec 14, 2023 at 10:28 AM Nikhil Goyal <nownik...@gmail.com> >> wrote: >> >>> Hi folks, >>> I am trying to understand one question. Does Spark Connect create a new >>> driver in the backend for every user or there are a fixed number of drivers >>> running to which requests are sent to? >>> >>> Thanks >>> Nikhil >>> >>
