Spark Connect, Master, and Workers
Hi, I'm recently learning Spark Connect but have some questions regarding the connect server's relation with master or workers: so when I'm using the connect server, I don't have to start a master alone side to make clients work. Is the connect server simply using "local[*]" as master? Then, if I want to add workers for my connect server, is it supported and what should I do? Kezhi
PySpark 3.5.0 on PyPI
Hi, Are there any plans to upload PySpark 3.5.0 to PyPI ( https://pypi.org/project/pyspark/)? It's still 3.4.1. Thanks, Kezhi
Re: PySpark 3.5.0 on PyPI
Oh, I saw it now. Thanks! On Wed, Sep 20, 2023 at 1:04 PM Sean Owen wrote: > [ External sender. Exercise caution. ] > > I think the announcement mentioned there were some issues with pypi and > the upload size this time. I am sure it's intended to be there when > possible. > > On Wed, Sep 20, 2023, 3:00 PM Kezhi Xiong > wrote: > >> Hi, >> >> Are there any plans to upload PySpark 3.5.0 to PyPI ( >> https://pypi.org/project/pyspark/)? It's still 3.4.1. >> >> Thanks, >> Kezhi >> >> >>
Spark Connect Multi-tenant Support
Hi, >From Spark Connect's official site's image, it mentions the "Multi-tenant Application Gateway" on driver. Are there any more documents about it? Can I know how users can utilize such a feature? Thanks, Kezhi
Re: Architecture of Spark Connect
Hi, My understanding is there is only one driver/spark context for all user sessions. When you run the bin/start-connect-server script, you are submitting one long standing spark job / application. Every time a new user request comes in, a new user session is created under that. Please correct me if I am wrong. Kezhi On Thu, Dec 14, 2023 at 10:35 AM Nikhil Goyal wrote: > [ External sender. Exercise caution. ] > > If multiple applications are running, we would need multiple spark connect > servers? If so, is the user responsible for creating these servers or they > are just created on the fly when the user requests a new spark session? > > On Thu, Dec 14, 2023 at 10:28 AM Nikhil Goyal wrote: > >> Hi folks, >> I am trying to understand one question. Does Spark Connect create a new >> driver in the backend for every user or there are a fixed number of drivers >> running to which requests are sent to? >> >> Thanks >> Nikhil >> >