Hi all, Thanks for the feedback in this thread and in the SPIP document. I would plan to start voting soon.
On Mon, Aug 3, 2026 at 6:11 PM Zhou Jiang <[email protected]> wrote: > +1 on the motivation. We have seen users trying PoCs to run Spark Connect > servers with a number of manual workarounds. A general gateway solution > looks valuable. > > On Sun, Aug 2, 2026 at 11:58 AM L. C. Hsieh <[email protected]> wrote: > >> I agree with the framing in the last comment, and it matches how the >> reference implementation is deliberately structured: the components that >> use-case pressure tends to hit sit behind interface seams (backend >> discovery, the affinity store, authentication, tenant resolution), with >> today's >> built-ins being the common-denominator implementations. The aim is >> exactly as stated: keep the initial scope small, and let concrete needs land >> as new implementations behind existing seams rather than speculating >> fixed features now. >> >> Both earlier asks map onto the same layered structure rather than >> requiring redesign. Version/capability-aware routing is a selection-layer >> extension: the metadata already exists in these deployments (e.g. the >> spark-version labels operators put on driver pods), so the discovery layer >> can be extended to surface it alongside addresses, and a metadata-aware >> selection strategy becomes a contained change, rolling upgrades of a >> mixed-version pool would be the first use case to benefit. >> >> The per-user, opt-in driver shape lands in the discovery/pool layer >> (pools keyed on identity and created on demand rather than fixed in config) >> which is a larger step and also involves the provisioning side, but it >> extends the same layer. >> >> Neither is in the proposed initial scope, and I don't think they should >> be, but the design should not preclude them. The proposed architecture >> already gives them the room they need: both land as extensions within >> existing layers, capability-aware selection inside the routing layer, >> identity-keyed pools inside the discovery layer. >> >> >> On Sun, Aug 2, 2026 at 6:19 AM Mark Hamstra <[email protected]> >> wrote: >> >>> There definitely needs to be planning and implementation that allow >>> for future extensions, since the functionality proposed thus far is >>> necessary but not sufficient for many use cases. On the other hand, >>> trying at this point to implement fixed, concrete functionality that >>> covers all conceivable use cases is probably not a good idea, so >>> flexible extensibility should be the aim. >>> >>> On Sat, Aug 1, 2026 at 12:17 PM Holden Karau <[email protected]> >>> wrote: >>> > >>> > What do we think about adding version discovery / capability discovery >>> so we can do more intelligent routing? Probably useful in layer 2 >>> > >>> > >>> > Twitter: https://twitter.com/holdenkarau >>> > Fight Health Insurance: https://www.fighthealthinsurance.com/ >>> > Books (Learning Spark, High Performance Spark, etc.): >>> https://amzn.to/2MaRAG9 >>> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> > Pronouns: she/her >>> > >>> > On Sat, Aug 1, 2026 at 4:54 AM Adam Binford <[email protected]> wrote: >>> >> >>> >> I see the README in the project actually lists my proxy as another >>> existing implementation, thanks for the call out. It's also Rust based for >>> performance, but proxies raw HTTP/2 traffic instead of parsing and >>> forwarding gRPC. As one of the people who has built a custom gateway/proxy >>> like this, there's a few use cases it doesn't meet, mostly for cases of >>> fine-grained data access controls per user, not necessarily per "tenant". I >>> know on-prem HDFS/Kerberos based users are probably a dying breed, so I'm >>> not sure how many other people actually need that functionality, but in >>> this environment a pre-created Spark Driver that's shared across multiple >>> users simply doesn't work. I'm a little curious how this works for other >>> existing use cases but for us: >>> >> - Authorization to data is tied to the user running the Spark job, so >>> each user needs its own job/driver >>> >> - Spark drivers can't be preallocated for every user, the users need >>> to be able to opt in to creating them >>> >> - Even if users could share drivers, we would constantly deal with >>> dumb users killing other users sessions with driver OOM or other poor >>> practices, each user having its own driver at least limits the blast radius. >>> >> >>> >> That's why one of the main purposes of my implementation is a REST >>> API for creating new Spark jobs, which then maps everything together with >>> static tokens at the HTTP/2 level, so no gRPC traffic needs to be parsed. >>> This might just be a custom enough use case for my environment, but I'm >>> curious if anyone else would need similar functionality to actually make >>> this useful. >>> >> >>> >> To be clear though, my project is very much a pet project that is not >>> heavily battle tested. I was also thinking if similar functionality was >>> ever upstreamed it would likely be in Scala, though Rust is the perfect >>> language for such a thing, and one of the main reasons you call out is >>> hyper's support for trailing headers. >>> >> >>> >> I do think it's useful to have some kind of official implementation >>> of this, as there is a lot of extra work required to actually make Spark >>> Connect useful in real environments. >>> >> >>> >> Adam >>> >> >>> >> On Sat, Aug 1, 2026 at 6:32 AM Mark Hamstra <[email protected]> >>> wrote: >>> >>> >>> >>> I can't say that the proposal to use Rust or any other non-JVM >>> >>> language for something that feels like a new core component of Spark >>> >>> sits easy with me, That is not to say that I am firmly opposed or >>> >>> can't be persuaded, but just that I am now uneasy. The arguments in >>> >>> the SPIP for using Rust don't strike me as particularly strong, being >>> >>> closer to "I/we like Rust" rather than a more persuasive >>> >>> compare-and-contrast of the relative merits for this component >>> written >>> >>> and maintained in Rust, Scala or Java. The idea that the basic design >>> >>> of this gateway should be independent enough that it could >>> conceivably >>> >>> be written in any of several languages without the end users caring >>> >>> much or at all is, I think, a good one, but that on its own doesn't >>> >>> persuade me that it shouldn't be written in the same version of Scala >>> >>> as the rest of core Spark, or in Scala 3, which has considerable >>> >>> advantages over Scala 2 for those components that are independent >>> >>> enough from the Scala 2 Spark core. >>> >>> >>> >>> Also, this new gateway does not strike me as analogous to >>> >>> spark-connect-swift or spark-connect-go or spark-connect-rust or some >>> >>> of the nascent talk about something like spark-connect-scala3 since >>> >>> those are all efforts to provide functionality for particular >>> >>> language-using subsets of the community, not for more or less >>> >>> everyone. >>> >>> >>> >>> No conclusions from me yet, just wanting to see more discussion >>> before deciding. >>> >>> >>> >>> On Fri, Jul 31, 2026 at 11:54 PM vaquar khan <[email protected]> >>> wrote: >>> >>> > >>> >>> > +1 >>> >>> > >>> >>> > A community-maintained implementation already exists for this >>> functionality. I have added a few comments to the SIP document outlining >>> how this prior art relates to the current proposal. >>> >>> > >>> >>> > Regards, >>> >>> > Viquar Khan >>> >>> > >>> >>> > On Fri, 31 Jul 2026 at 23:37, Dongjoon Hyun <[email protected]> >>> wrote: >>> >>> >> >>> >>> >> For the record, Apache Spark community currently maintains seven >>> GitHub repositories like the following: >>> >>> >> >>> >>> >> - https://github.com/apache/spark >>> (Last Commit: Today) >>> >>> >> - https://github.com/apache/spark-connect-swift >>> (Last Commit: Today) >>> >>> >> - https://github.com/apache/spark-kubernetes-operator (Last >>> Commit: 3 days ago) >>> >>> >> - https://github.com/apache/spark-website >>> (Last Commit 5 days ago) >>> >>> >> - https://github.com/apache/spark-docker >>> (Last Commit: 1 week ago) >>> >>> >> - https://github.com/apache/spark-connect-rust >>> (Last Commit: 2 months ago) >>> >>> >> - https://github.com/apache/spark-connect-go >>> (Last Commit: 11 months ago) >>> >>> >> >>> >>> >> Since there is no bug-free software, I hope the new proposed >>> repository builds a new sub-community by being one of active Apache Spark >>> repositories instead of one-time snapshot drop-off. >>> >>> >> >>> >>> >> - https://github.com/apache/spark-connect-gateway >>> >>> >> >>> >>> >> Sincerely, >>> >>> >> Dongjoon. >>> >>> >> >>> >>> >> On 2026/08/01 04:30:05 John Zhuge wrote: >>> >>> >> > +1 Thanks. >>> >>> >> > >>> >>> >> > On Fri, Jul 31, 2026 at 8:41 PM Dongjoon Hyun < >>> [email protected]> wrote: >>> >>> >> > >>> >>> >> > > Thank you, Liang-Chi. >>> >>> >> > > >>> >>> >> > > Since it's written in Rust like "apache/spark-connect-rust", >>> >>> >> > > "apache/spark-connect-gateway" proposal sounds good to me. >>> >>> >> > > >>> >>> >> > > Where is the reference implementation? I didn't find any >>> GitHub link in >>> >>> >> > > SPARK-58455 and SPIP. >>> >>> >> > > >>> >>> >> > > Dongjoon. >>> >>> >> > > >>> >>> >> > > On 2026/07/31 22:49:44 Chao Sun wrote: >>> >>> >> > > > +1 >>> >>> >> > > > >>> >>> >> > > > On Fri, Jul 31, 2026 at 1:26 PM DB Tsai <[email protected]> >>> wrote: >>> >>> >> > > > >>> >>> >> > > > > +1 >>> >>> >> > > > > >>> >>> >> > > > > An open-source reference gateway that makes it easier to >>> run Spark >>> >>> >> > > Connect >>> >>> >> > > > > at scale with session affinity, authentication, high >>> availability, >>> >>> >> > > > > multi-tenancy, rate limiting, and audit logging built in >>> is great for >>> >>> >> > > > > production adoption of Spark Connect. >>> >>> >> > > > > >>> >>> >> > > > > >>> >>> >> > > > > DB Tsai | https://www.dbtsai.com/ | PGP 0x9FB9FAA3 >>> >>> >> > > > > >>> >>> >> > > > > On Thursday, July 30th, 2026 at 3:43 PM, L. C. Hsieh < >>> [email protected] >>> >>> >> > > > >>> >>> >> > > > > wrote: >>> >>> >> > > > > >>> >>> >> > > > > Hi all, >>> >>> >> > > > > >>> >>> >> > > > > I'd like to start a discussion on a SPIP proposing an >>> official gateway >>> >>> >> > > > > component for Spark Connect deployments. >>> >>> >> > > > > >>> >>> >> > > > > Problem: A Spark Connect client has to know the address >>> of one specific >>> >>> >> > > > > server and stay with it for the lifetime of its session, >>> because >>> >>> >> > > session >>> >>> >> > > > > state lives in that server's driver. As soon as a >>> deployment runs more >>> >>> >> > > than >>> >>> >> > > > > one Spark Connect server — per-team servers, capacity, >>> redundancy — >>> >>> >> > > there >>> >>> >> > > > > is no built-in way to put a single address in front of >>> the fleet. A >>> >>> >> > > generic >>> >>> >> > > > > load balancer can't do this either: the routing key >>> (session_id) lives >>> >>> >> > > > > inside the request protobuf, not in headers. So today >>> every team >>> >>> >> > > writes its >>> >>> >> > > > > own proxy glue, and everyone solves the same problem from >>> scratch. >>> >>> >> > > > > >>> >>> >> > > > > Proposal: The Spark Connect Gateway is a standalone, >>> wire-compatible >>> >>> >> > > gRPC >>> >>> >> > > > > proxy that sits in front of a pool of Spark Connect >>> servers. >>> >>> >> > > > > >>> >>> >> > > > > Clients are untouched: the connection string points at >>> the gateway and >>> >>> >> > > > > everything else is unchanged. The proposal adds nothing >>> to and changes >>> >>> >> > > > > nothing in the Spark Connect protocol. >>> >>> >> > > > > >>> >>> >> > > > > SPIP doc: >>> >>> >> > > > > >>> >>> >> > > >>> https://docs.google.com/document/d/16RNnRJjSDBSlcgeBjVz_Mv-SkqvH33F9tIsR1Ii_P3c/edit?tab=t.0#heading=h.221xxblzqzu >>> >>> >> > > > > JIRA: https://issues.apache.org/jira/browse/SPARK-58455 >>> >>> >> > > > > >>> >>> >> > > > > Feedback is very welcome. >>> >>> >> > > > > >>> >>> >> > > > > Thanks, >>> >>> >> > > > > Liang-Chi >>> >>> >> > > > > >>> >>> >> > > > > >>> >>> >> > > > > >>> >>> >> > > > >>> >>> >> > > >>> >>> >> > > >>> --------------------------------------------------------------------- >>> >>> >> > > To unsubscribe e-mail: [email protected] >>> >>> >> > > >>> >>> >> > > >>> >>> >> > >>> >>> >> > -- >>> >>> >> > John Zhuge >>> >>> >> > >>> >>> >> >>> >>> >> >>> --------------------------------------------------------------------- >>> >>> >> To unsubscribe e-mail: [email protected] >>> >>> >> >>> >>> >>> >>> --------------------------------------------------------------------- >>> >>> To unsubscribe e-mail: [email protected] >>> >>> >>> >> >>> >> >>> >> -- >>> >> Adam Binford >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: [email protected] >>> >>> > > -- > *Zhou JIANG* > >
