+1 on the motivation. We have seen users trying PoCs to run Spark Connect servers with a number of manual workarounds. A general gateway solution looks valuable.
On Sun, Aug 2, 2026 at 11:58 AM L. C. Hsieh <[email protected]> wrote: > I agree with the framing in the last comment, and it matches how the > reference implementation is deliberately structured: the components that > use-case pressure tends to hit sit behind interface seams (backend > discovery, the affinity store, authentication, tenant resolution), with > today's > built-ins being the common-denominator implementations. The aim is exactly > as stated: keep the initial scope small, and let concrete needs land > as new implementations behind existing seams rather than speculating fixed > features now. > > Both earlier asks map onto the same layered structure rather than > requiring redesign. Version/capability-aware routing is a selection-layer > extension: the metadata already exists in these deployments (e.g. the > spark-version labels operators put on driver pods), so the discovery layer > can be extended to surface it alongside addresses, and a metadata-aware > selection strategy becomes a contained change, rolling upgrades of a > mixed-version pool would be the first use case to benefit. > > The per-user, opt-in driver shape lands in the discovery/pool layer (pools > keyed on identity and created on demand rather than fixed in config) > which is a larger step and also involves the provisioning side, but it > extends the same layer. > > Neither is in the proposed initial scope, and I don't think they should > be, but the design should not preclude them. The proposed architecture > already gives them the room they need: both land as extensions within > existing layers, capability-aware selection inside the routing layer, > identity-keyed pools inside the discovery layer. > > > On Sun, Aug 2, 2026 at 6:19 AM Mark Hamstra <[email protected]> wrote: > >> There definitely needs to be planning and implementation that allow >> for future extensions, since the functionality proposed thus far is >> necessary but not sufficient for many use cases. On the other hand, >> trying at this point to implement fixed, concrete functionality that >> covers all conceivable use cases is probably not a good idea, so >> flexible extensibility should be the aim. >> >> On Sat, Aug 1, 2026 at 12:17 PM Holden Karau <[email protected]> >> wrote: >> > >> > What do we think about adding version discovery / capability discovery >> so we can do more intelligent routing? Probably useful in layer 2 >> > >> > >> > Twitter: https://twitter.com/holdenkarau >> > Fight Health Insurance: https://www.fighthealthinsurance.com/ >> > Books (Learning Spark, High Performance Spark, etc.): >> https://amzn.to/2MaRAG9 >> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> > Pronouns: she/her >> > >> > On Sat, Aug 1, 2026 at 4:54 AM Adam Binford <[email protected]> wrote: >> >> >> >> I see the README in the project actually lists my proxy as another >> existing implementation, thanks for the call out. It's also Rust based for >> performance, but proxies raw HTTP/2 traffic instead of parsing and >> forwarding gRPC. As one of the people who has built a custom gateway/proxy >> like this, there's a few use cases it doesn't meet, mostly for cases of >> fine-grained data access controls per user, not necessarily per "tenant". I >> know on-prem HDFS/Kerberos based users are probably a dying breed, so I'm >> not sure how many other people actually need that functionality, but in >> this environment a pre-created Spark Driver that's shared across multiple >> users simply doesn't work. I'm a little curious how this works for other >> existing use cases but for us: >> >> - Authorization to data is tied to the user running the Spark job, so >> each user needs its own job/driver >> >> - Spark drivers can't be preallocated for every user, the users need >> to be able to opt in to creating them >> >> - Even if users could share drivers, we would constantly deal with >> dumb users killing other users sessions with driver OOM or other poor >> practices, each user having its own driver at least limits the blast radius. >> >> >> >> That's why one of the main purposes of my implementation is a REST API >> for creating new Spark jobs, which then maps everything together with >> static tokens at the HTTP/2 level, so no gRPC traffic needs to be parsed. >> This might just be a custom enough use case for my environment, but I'm >> curious if anyone else would need similar functionality to actually make >> this useful. >> >> >> >> To be clear though, my project is very much a pet project that is not >> heavily battle tested. I was also thinking if similar functionality was >> ever upstreamed it would likely be in Scala, though Rust is the perfect >> language for such a thing, and one of the main reasons you call out is >> hyper's support for trailing headers. >> >> >> >> I do think it's useful to have some kind of official implementation of >> this, as there is a lot of extra work required to actually make Spark >> Connect useful in real environments. >> >> >> >> Adam >> >> >> >> On Sat, Aug 1, 2026 at 6:32 AM Mark Hamstra <[email protected]> >> wrote: >> >>> >> >>> I can't say that the proposal to use Rust or any other non-JVM >> >>> language for something that feels like a new core component of Spark >> >>> sits easy with me, That is not to say that I am firmly opposed or >> >>> can't be persuaded, but just that I am now uneasy. The arguments in >> >>> the SPIP for using Rust don't strike me as particularly strong, being >> >>> closer to "I/we like Rust" rather than a more persuasive >> >>> compare-and-contrast of the relative merits for this component written >> >>> and maintained in Rust, Scala or Java. The idea that the basic design >> >>> of this gateway should be independent enough that it could conceivably >> >>> be written in any of several languages without the end users caring >> >>> much or at all is, I think, a good one, but that on its own doesn't >> >>> persuade me that it shouldn't be written in the same version of Scala >> >>> as the rest of core Spark, or in Scala 3, which has considerable >> >>> advantages over Scala 2 for those components that are independent >> >>> enough from the Scala 2 Spark core. >> >>> >> >>> Also, this new gateway does not strike me as analogous to >> >>> spark-connect-swift or spark-connect-go or spark-connect-rust or some >> >>> of the nascent talk about something like spark-connect-scala3 since >> >>> those are all efforts to provide functionality for particular >> >>> language-using subsets of the community, not for more or less >> >>> everyone. >> >>> >> >>> No conclusions from me yet, just wanting to see more discussion >> before deciding. >> >>> >> >>> On Fri, Jul 31, 2026 at 11:54 PM vaquar khan <[email protected]> >> wrote: >> >>> > >> >>> > +1 >> >>> > >> >>> > A community-maintained implementation already exists for this >> functionality. I have added a few comments to the SIP document outlining >> how this prior art relates to the current proposal. >> >>> > >> >>> > Regards, >> >>> > Viquar Khan >> >>> > >> >>> > On Fri, 31 Jul 2026 at 23:37, Dongjoon Hyun <[email protected]> >> wrote: >> >>> >> >> >>> >> For the record, Apache Spark community currently maintains seven >> GitHub repositories like the following: >> >>> >> >> >>> >> - https://github.com/apache/spark >> (Last Commit: Today) >> >>> >> - https://github.com/apache/spark-connect-swift (Last >> Commit: Today) >> >>> >> - https://github.com/apache/spark-kubernetes-operator (Last >> Commit: 3 days ago) >> >>> >> - https://github.com/apache/spark-website >> (Last Commit 5 days ago) >> >>> >> - https://github.com/apache/spark-docker >> (Last Commit: 1 week ago) >> >>> >> - https://github.com/apache/spark-connect-rust (Last >> Commit: 2 months ago) >> >>> >> - https://github.com/apache/spark-connect-go (Last >> Commit: 11 months ago) >> >>> >> >> >>> >> Since there is no bug-free software, I hope the new proposed >> repository builds a new sub-community by being one of active Apache Spark >> repositories instead of one-time snapshot drop-off. >> >>> >> >> >>> >> - https://github.com/apache/spark-connect-gateway >> >>> >> >> >>> >> Sincerely, >> >>> >> Dongjoon. >> >>> >> >> >>> >> On 2026/08/01 04:30:05 John Zhuge wrote: >> >>> >> > +1 Thanks. >> >>> >> > >> >>> >> > On Fri, Jul 31, 2026 at 8:41 PM Dongjoon Hyun < >> [email protected]> wrote: >> >>> >> > >> >>> >> > > Thank you, Liang-Chi. >> >>> >> > > >> >>> >> > > Since it's written in Rust like "apache/spark-connect-rust", >> >>> >> > > "apache/spark-connect-gateway" proposal sounds good to me. >> >>> >> > > >> >>> >> > > Where is the reference implementation? I didn't find any >> GitHub link in >> >>> >> > > SPARK-58455 and SPIP. >> >>> >> > > >> >>> >> > > Dongjoon. >> >>> >> > > >> >>> >> > > On 2026/07/31 22:49:44 Chao Sun wrote: >> >>> >> > > > +1 >> >>> >> > > > >> >>> >> > > > On Fri, Jul 31, 2026 at 1:26 PM DB Tsai <[email protected]> >> wrote: >> >>> >> > > > >> >>> >> > > > > +1 >> >>> >> > > > > >> >>> >> > > > > An open-source reference gateway that makes it easier to >> run Spark >> >>> >> > > Connect >> >>> >> > > > > at scale with session affinity, authentication, high >> availability, >> >>> >> > > > > multi-tenancy, rate limiting, and audit logging built in >> is great for >> >>> >> > > > > production adoption of Spark Connect. >> >>> >> > > > > >> >>> >> > > > > >> >>> >> > > > > DB Tsai | https://www.dbtsai.com/ | PGP 0x9FB9FAA3 >> >>> >> > > > > >> >>> >> > > > > On Thursday, July 30th, 2026 at 3:43 PM, L. C. Hsieh < >> [email protected] >> >>> >> > > > >> >>> >> > > > > wrote: >> >>> >> > > > > >> >>> >> > > > > Hi all, >> >>> >> > > > > >> >>> >> > > > > I'd like to start a discussion on a SPIP proposing an >> official gateway >> >>> >> > > > > component for Spark Connect deployments. >> >>> >> > > > > >> >>> >> > > > > Problem: A Spark Connect client has to know the address of >> one specific >> >>> >> > > > > server and stay with it for the lifetime of its session, >> because >> >>> >> > > session >> >>> >> > > > > state lives in that server's driver. As soon as a >> deployment runs more >> >>> >> > > than >> >>> >> > > > > one Spark Connect server — per-team servers, capacity, >> redundancy — >> >>> >> > > there >> >>> >> > > > > is no built-in way to put a single address in front of the >> fleet. A >> >>> >> > > generic >> >>> >> > > > > load balancer can't do this either: the routing key >> (session_id) lives >> >>> >> > > > > inside the request protobuf, not in headers. So today >> every team >> >>> >> > > writes its >> >>> >> > > > > own proxy glue, and everyone solves the same problem from >> scratch. >> >>> >> > > > > >> >>> >> > > > > Proposal: The Spark Connect Gateway is a standalone, >> wire-compatible >> >>> >> > > gRPC >> >>> >> > > > > proxy that sits in front of a pool of Spark Connect >> servers. >> >>> >> > > > > >> >>> >> > > > > Clients are untouched: the connection string points at the >> gateway and >> >>> >> > > > > everything else is unchanged. The proposal adds nothing to >> and changes >> >>> >> > > > > nothing in the Spark Connect protocol. >> >>> >> > > > > >> >>> >> > > > > SPIP doc: >> >>> >> > > > > >> >>> >> > > >> https://docs.google.com/document/d/16RNnRJjSDBSlcgeBjVz_Mv-SkqvH33F9tIsR1Ii_P3c/edit?tab=t.0#heading=h.221xxblzqzu >> >>> >> > > > > JIRA: https://issues.apache.org/jira/browse/SPARK-58455 >> >>> >> > > > > >> >>> >> > > > > Feedback is very welcome. >> >>> >> > > > > >> >>> >> > > > > Thanks, >> >>> >> > > > > Liang-Chi >> >>> >> > > > > >> >>> >> > > > > >> >>> >> > > > > >> >>> >> > > > >> >>> >> > > >> >>> >> > > >> --------------------------------------------------------------------- >> >>> >> > > To unsubscribe e-mail: [email protected] >> >>> >> > > >> >>> >> > > >> >>> >> > >> >>> >> > -- >> >>> >> > John Zhuge >> >>> >> > >> >>> >> >> >>> >> >> --------------------------------------------------------------------- >> >>> >> To unsubscribe e-mail: [email protected] >> >>> >> >> >>> >> >>> --------------------------------------------------------------------- >> >>> To unsubscribe e-mail: [email protected] >> >>> >> >> >> >> >> >> -- >> >> Adam Binford >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: [email protected] >> >> -- *Zhou JIANG*
