Spark Connect, Master, and Workers

[Kezhi Xiong]

Hi,

I'm recently learning Spark Connect but have some questions regarding the
connect server's relation with master or workers: so when I'm using the
connect server, I don't have to start a master alone side to make clients
work. Is the connect server simply using "local[*]" as master? Then, if I
want to add workers for my connect server, is it supported and what should
I do?

Kezhi

PySpark 3.5.0 on PyPI

[Kezhi Xiong]

Hi,

Are there any plans to upload PySpark 3.5.0 to PyPI (
https://pypi.org/project/pyspark/)? It's still 3.4.1.

Thanks,
Kezhi

Re: PySpark 3.5.0 on PyPI

[Kezhi Xiong]

Oh, I saw it now. Thanks!

On Wed, Sep 20, 2023 at 1:04 PM Sean Owen  wrote:

> [ External sender. Exercise caution. ]
>
> I think the announcement mentioned there were some issues with pypi and
> the upload size this time. I am sure it's intended to be there when
> possible.
>
> On Wed, Sep 20, 2023, 3:00 PM Kezhi Xiong 
> wrote:
>
>> Hi,
>>
>> Are there any plans to upload PySpark 3.5.0 to PyPI (
>> https://pypi.org/project/pyspark/)? It's still 3.4.1.
>>
>> Thanks,
>> Kezhi
>>
>>
>>

Spark Connect Multi-tenant Support

[Kezhi Xiong]

Hi,

>From Spark Connect's official site's image, it mentions the "Multi-tenant
Application Gateway" on driver. Are there any more documents about it? Can
I know how users can utilize such a feature?

Thanks,
Kezhi

Re: Architecture of Spark Connect

[Kezhi Xiong]

Hi,

My understanding is there is only one driver/spark context for all user
sessions. When you run the bin/start-connect-server script, you are
submitting one long standing spark job / application. Every time a new user
request comes in, a new user session is created under that. Please correct
me if I am wrong.

Kezhi

On Thu, Dec 14, 2023 at 10:35 AM Nikhil Goyal  wrote:

> [ External sender. Exercise caution. ]
>
> If multiple applications are running, we would need multiple spark connect
> servers? If so, is the user responsible for creating these servers or they
> are just created on the fly when the user requests a new spark session?
>
> On Thu, Dec 14, 2023 at 10:28 AM Nikhil Goyal  wrote:
>
>> Hi folks,
>> I am trying to understand one question. Does Spark Connect create a new
>> driver in the backend for every user or there are a fixed number of drivers
>> running to which requests are sent to?
>>
>> Thanks
>> Nikhil
>>
>