Yeah I was thinking about pyspark-client-rust. It is just same API background but quite fast for small quries. Let me kill the release vote on my own first and clarify it and start next RC
On Sun, Aug 30, 2026 at 10:50 AM Holden Karau <[email protected]> wrote: > So I guess my question is are we planning to distribute it as a Python > package? If so let's dig in some more first and try and clarify in docs the > difference between the two different official connect clients first, if not > then I think it doesn't matter (until we do). > > On Sat, Aug 29, 2026 at 6:38 PM Hyukjin Kwon <[email protected]> wrote: > >> Hi all, >> >> Spinning this out of the RC1 vote thread. Alongside the Rust Spark >> Connect client, >> we now have pyspark-client-rust: a Python wrapper that installs the same >> pyspark >> surface as pyspark-client (existing PySpark Connect code runs unchanged), >> but with >> the native Rust client (tonic) as the engine instead of grpcio/py4j. It >> is checked >> against PySpark 4.2.0 (official test suite + call-by-call API parity), >> and the Rust >> crates work standalone too. >> >> I personally thought that it does not need a SPIP - it is a separate >> feature/package, not a core >> change - but it is user-facing, so I wanted a dedicated thread for any >> thoughts. >> Not blocking the RC1 vote. >> >> Architecture: https://apache.github.io/spark-connect-rust/architecture/ >> Repo: https://github.com/apache/spark-connect-rust >> >> So the main question is, how do you guys like it? >> >> Thanks, >> Hyukjin Kwon >> > > > -- > Twitter: https://twitter.com/holdenkarau > Fight Health Insurance: https://www.fighthealthinsurance.com/ > <https://www.fighthealthinsurance.com/?q=hk_email> > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > Pronouns: she/her >
