HyukjinKwon opened a new pull request, #51: URL: https://github.com/apache/spark-connect-rust/pull/51
## Summary Layers a thin **drop-in Python (`pyspark`) wrapper** over the Rust Spark Connect client so existing Spark Connect code runs unchanged. Published to PyPI as `pyspark-client-rust`; the import package stays `pyspark`. > **Stacked on #50** (the Rust core). This PR is based on the `split-rust-core` branch, so its diff shows only the Python additions. It will be retargeted to `master` once #50 merges. ## What's in this PR - **`crates/pyspark-rs`** — PyO3 bindings that build the `_pyspark` extension module, wrapping the pure-Rust `spark-connect` API (`publish = false`; shipped as a wheel). - **`python/pyspark/**`** — the drop-in `pyspark` package (thin wrapper + vendored `pyspark.pandas` / `cloudpickle` / `pyspark.testing` for compatibility). - **`pyproject.toml`** — maturin build; distribution name `pyspark-client-rust`. - **`.cargo/config.toml`** — macOS link flags for the CPython extension. - **CI** — `build_python_connect.yml` runs the reference-vs-ours parity gate and an end-to-end benchmark against a real Spark 4.2.0 Connect server; `release.yml` extended to build abi3 wheels and publish to PyPI (committer-fork model). - Docs, the parity ledger, and the official-test harness scripts. ## Verification - `cargo build -p pyspark-rs` compiles the extension; core `cargo test` still passes (21 suites). - The wrapper imports under Python 3.11 with 444 functions exposed; it is a complete drop-in for the reference `pyspark` Spark Connect client. JIRA: https://issues.apache.org/jira/browse/SPARK-58960 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
