dbtsai opened a new pull request, #103: URL: https://github.com/apache/spark-connect-rust/pull/103
### What changes were proposed in this pull request? Introduce `spark-python-arrow`, a standalone Rust library shared by Python-facing Spark clients and workers, and adopt it in `pyspark-rs`. - `bridge`: Arrow/PyArrow C Data import and export for schemas, types, arrays, and record batches. Compatible decoded Arrow data buffers are shared, preserving metadata, slices, batch boundaries, buffer lifetimes, and zero-column row counts. - `scalar`: borrowed scalar materialization and Python extraction, with exact-built-in UDF rules separated from coercive client-input rules. - `udf`: reusable `SQL_ARROW_BATCHED_UDF` input/result conversion policy. Callers supply Python classes and fallback callbacks; the crate does not import a particular PySpark distribution or own worker execution. - Route the Python client's `toArrow()` through the shared bridge, replacing local IPC reserialization and the Python bytes copy. Reuse shared scalar helpers in row materialization and session input conversion. - Add standalone converter tests, public-client handoff tests, CI wiring, and a documented integration/ownership contract in `crates/spark-python-arrow/README.md`. The crate depends only on Arrow and PyO3. It does not select embedding versus extension-module linking or a minimum Python ABI, and the Rust client core remains Python-free. Downstream worker migration and dependency intake are separate from this PR. ### Why are the changes needed? Clients and Python workers need one versioned implementation of Arrow handoff and conversion semantics instead of maintaining copied converter modules. This extraction provides that dependency boundary and removes an avoidable serialization/copy step from `toArrow()`. The shared UDF policy retains fallback behavior, safe-cast retry, mutable-result handling, and conversion-error ordering. Its native subset covers signed integers, floats, booleans, strings, binary, and tagged Variant structs, plus dates on input. Nested, decimal, timestamp, and date-result columns continue through caller-supplied Python fallback. ### Does this PR introduce _any_ user-facing change? Yes: a reusable Rust conversion API is introduced. Existing Python API names and signatures are unchanged; `toArrow()` now shares compatible Arrow data buffers with PyArrow rather than reserializing them locally. Zero-copy applies only to the Rust/PyArrow data-buffer handoff. It does not cover network IPC decoding, schema/Python object allocation, scalar/row materialization, or pandas conversion. `collect()` still materializes owned rows, `createDataFrame(pandas)` still materializes Python rows, and `toPandas()` may copy. Table export uses an IPC compatibility fallback for embedded NUL field names or timezones, without another RPC. Low-level C exports reject these strings. Bare extension types/arrays are rejected instead of silently losing their identity; schema/record-batch paths preserve extension field metadata. ABI-safe string extraction can allocate under `abi3-py39`; no benchmark or end-to-end speedup is claimed for this extraction. ### How was this patch tested? Local validation used cached dependencies with Cargo offline: | Validation | Result | | --- | --- | | Standalone converter tests, PyO3 0.28.3 | 32 passed (9 unit, 23 integration) | | Python 3.12 / PyArrow 26: offline client + handoff tests | 290 passed (273 existing, 17 handoff) | | Python 3.9 / PyArrow 10: handoff tests | 17 passed | | `apache-spark-connect` release unit tests | 364 passed | | All normal integration tests, release `--no-run` | Compiled | | All WASM-enabled integration tests, release `--no-run` | Compiled | | Release Python wheel | Built and installed locally | | Standalone converter Clippy, all targets, `abi3-py39`, `-D warnings` | Passed | | `cargo fmt --all --check`, `git diff --check`, no-stub audit | Passed | | Standalone `cargo package` with package verification | Passed | | Official Spark 4.2.0 collection parity through Rust transport | 15 passed, 1 skipped | The new tests verify buffer identity after Rust IPC decoding, lifetime/release behavior, metadata and slices, empty shapes, compatibility fallback, scalar behavior, and UDF fallback/error ordering. The coverage workflow now runs the standalone converter and Python handoff suites. **Draft caveats / outstanding verification:** - Full workspace Clippy is **not green**. The 140 binding errors were also reproduced on clean upstream `c41ff82deb8140b41b129d4d1b3a3640ac06dc52`; that baseline also has approximate-constant errors in core tests. The new converter passes strict standalone Clippy. - Public-name introspection across the 12 required classes found no names removed relative to the same baseline. Against official PySpark, `Column.to_plan` is intentionally absent and `SparkSession.Hook` is a pre-existing gap. This is a names-only check, not proof of signature, behavior, or package parity. - The full official Connect parity suite remains pending CI; the targeted collection result above is not a full-parity claim. - The declared PyO3 range permits 0.28 and 0.29, but **only 0.28.3 was tested**. PyO3 0.29 remains unverified because missing dependencies were not downloaded. - Downstream Bazel integration and worker differential fuzzers remain unverified. No new benchmark results are included. This is intentionally a draft while those validation gaps are tracked. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (exact application version unavailable). This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
