Love the idea and can't wait to try it out. My biggest question/concern is if the "pyspark" package name should be reused or it it should be scoped under something like "pyspark_rust". I get the idea of a drop in replacement, but having had to deal with enough conflicts for people between pyspark and pyspark-client, this seems like it would just cause even more confusion. Users may not know which library they are using, and hit issues with the rust client and accidentally report them as native spark client bugs. Plus if you try to support multiple versions of a Spark server, including versions older than 4.2, you would need to support both the native client and rust client simultaneously.
Adam On Sat, Aug 29, 2026, 9:53 PM Hyukjin Kwon <[email protected]> wrote: > Yeah I was thinking about pyspark-client-rust. It is just same API > background but quite fast for small quries. Let me kill the release vote on > my own first and clarify it and start next RC > > On Sun, Aug 30, 2026 at 10:50 AM Holden Karau <[email protected]> > wrote: > >> So I guess my question is are we planning to distribute it as a Python >> package? If so let's dig in some more first and try and clarify in docs the >> difference between the two different official connect clients first, if not >> then I think it doesn't matter (until we do). >> >> On Sat, Aug 29, 2026 at 6:38 PM Hyukjin Kwon <[email protected]> >> wrote: >> >>> Hi all, >>> >>> Spinning this out of the RC1 vote thread. Alongside the Rust Spark >>> Connect client, >>> we now have pyspark-client-rust: a Python wrapper that installs the same >>> pyspark >>> surface as pyspark-client (existing PySpark Connect code runs >>> unchanged), but with >>> the native Rust client (tonic) as the engine instead of grpcio/py4j. It >>> is checked >>> against PySpark 4.2.0 (official test suite + call-by-call API parity), >>> and the Rust >>> crates work standalone too. >>> >>> I personally thought that it does not need a SPIP - it is a separate >>> feature/package, not a core >>> change - but it is user-facing, so I wanted a dedicated thread for any >>> thoughts. >>> Not blocking the RC1 vote. >>> >>> Architecture: https://apache.github.io/spark-connect-rust/architecture/ >>> Repo: https://github.com/apache/spark-connect-rust >>> >>> So the main question is, how do you guys like it? >>> >>> Thanks, >>> Hyukjin Kwon >>> >> >> >> -- >> Twitter: https://twitter.com/holdenkarau >> Fight Health Insurance: https://www.fighthealthinsurance.com/ >> <https://www.fighthealthinsurance.com/?q=hk_email> >> Books (Learning Spark, High Performance Spark, etc.): >> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> Pronouns: she/her >> >
