HyukjinKwon commented on PR #50: URL: https://github.com/apache/spark-connect-rust/pull/50#issuecomment-5390613520
## Summary of API changes vs the previous `spark-connect-rs` client **Headline: the client API is now synchronous.** The previous client was async — `SparkSessionBuilder::remote(url).build().await`, `df.collect().await`, ~35 `async fn`s on `DataFrame`. This PR makes the whole surface **synchronous** — `SparkSession::builder().remote(url).get_or_create()`, `df.collect()` — blocking on an internal Tokio runtime at the call boundary. **Why:** consistency. The client mirrors `pyspark.sql.connect.*` one-to-one, and the drop-in Python (`pyspark`) wrapper layered on top (follow-up PR) is synchronous. A sync core keeps the Python skin thin and the Rust surface shaped exactly like PySpark's. ### Other changes - **Workspace restructure:** the single `spark-connect-rs` crate becomes three — `spark-connect-proto` (generated protos), `spark-connect-core` (transport), `spark-connect` (DataFrame API). - **Protocol / deps bump:** Spark 3.5 protos → Spark 4.x (validated against 4.2.0); tonic 0.11 → 0.14, prost 0.12 → 0.14, arrow 55 → 58. - **`collect()` now returns `Vec<Row>`** (was `RecordBatch`). Arrow batches are available via the new `collect_record_batches() -> Vec<RecordBatch>`. - **Wider function coverage:** ~440 SQL functions (was ~99), asserted byte-for-byte against captured reference protos (golden-proto tests). - **Kept:** optional `datafusion` / `polars` conversions (`to_datafusion()`, `to_polars()`), catalog, window, streaming, `na()` / `stat()`, DataFrameReader/Writer (V1). - **Added:** golden-proto test suite, no-stub audit, crates.io release workflow, and `.asf.yaml` (GitHub Issues routed to ASF JIRA). ### Known gaps in the Rust client itself (vs the previous client) - **`DataFrame.writeTo` / `DataFrameWriterV2` is not yet ported** — the previous client had it; this PR does not. This is the one notable regression in the Rust surface; the rest of the previous API is present. - `DataFrame::to_arrow()` is currently a placeholder — use `collect_record_batches()` for real Arrow output. - Closure-based ops (`foreach`, `mapInPandas`, `pandas_udf`) remain server-executed via the Python UDF path (added in the wrapper PR); they are not part of the pure-Rust surface — same as the previous client. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
