HyukjinKwon opened a new pull request, #49: URL: https://github.com/apache/spark-connect-rust/pull/49
## Summary Restructures this repository so the **Rust core is the Spark Connect client** and a thin **Python wrapper** (published to PyPI as `pyspark-client-rust`; import package stays `pyspark`) is layered over it. The previous single async crate `spark-connect-rs` is replaced by a four-crate workspace plus a Python skin, adopting the design of the `pyspark-client-fast` project as the standard. ## What changed - **Rust core client** — `spark-connect-proto` (generated `spark.connect.*`) → `spark-connect-core` (transport: channel, retries, reattach, artifacts, errors) → `spark-connect` (pure-Rust, **synchronous** DataFrame API mirroring PySpark) → `pyspark-rs` (PyO3 → the `_pyspark` extension module, `publish = false`). - **Python wrapper** — `python/pyspark/**` drop-in package built with maturin; distribution name `pyspark-client-rust`. - **Version** — the crates and the wheel are versioned **4.2.0** to track the Apache Spark release users run against. - **Project identity** — authorship/URLs point at the ASF project (`[email protected]`, `github.com/apache/spark-connect-rust`, `spark.apache.org`); JIRA (`SPARK`) is the issue tracker. - **`.asf.yaml`** — GitHub Issues disabled (`issues: false`); issues live in ASF JIRA. - **Tests** — golden-proto tests (`crates/**`, `tests/golden/`) and the official Apache connect test-suite parity harness (`scripts/`, `tests/official/`). - **CI** — `cargo build/test`, `rustfmt`/`clippy`, `ruff` on `scripts/`, maturin wheel, a reference-vs-ours parity gate against **Spark 4.2.0**, and a job that builds/tests the optional `datafusion,polars` conversion features. - **`release.yml`** — publishes the Rust crates to crates.io and the wheel to PyPI, following the Apache Spark model where a committer cuts releases from **their own fork** (`workflow_dispatch` with a dry-run default, or a `v<version>` tag; secrets `CARGO_REGISTRY_TOKEN` / `PYPI_API_TOKEN` live in the fork). - **Protos** remain the vendored 4.1.0-SNAPSHOT set the client is written against (includes the `Zip` relation removed in 4.2.0); wire-compatible with a 4.2.0 server. - **Rust-native conversions** restored as off-by-default features: `DataFrame::to_datafusion()` and `DataFrame::to_polars()` (the latter bridged via Arrow IPC so polars' vendored arrow need not match arrow-rs), plus a real `collect_record_batches()` helper. - **Examples** rewritten to the synchronous API; **README** rewritten to the new standard, with the old per-API "Features" coverage checklist removed (full parity is the goal). ## Verification - Core builds; all Rust golden/unit tests pass (21 suites, 0 failures). - The `_pyspark` extension loads and the Python wrapper imports (444 functions) under Python 3.11. - Examples compile; `to_datafusion()` compiles behind its feature. The `polars` feature and the full `datafusion,polars` build are exercised in CI. Opened as a **draft** for review. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
