DB Tsai created SPARK-60675:
-------------------------------
Summary: Support PyArrow Table input through the C Stream
interface in Spark Connect Rust
Key: SPARK-60675
URL: https://issues.apache.org/jira/browse/SPARK-60675
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
The native Rust-backed SparkSession.createDataFrame accepts Python lists and
pandas DataFrames, but a pyarrow.Table currently reaches the list cast and
fails.
Support pyarrow.Table input through arrow-rs's owned C Stream importer. Reuse
the existing PySpark Arrow schema and timestamp helpers for Connect-compatible
normalization, import a RecordBatchReader, and serialize the Rust batches
directly into the local-relation IPC payload. Avoid materializing column values
as Python rows. Casts, buffer alignment and IPC serialization may still copy.
Preserve explicit schemas, column-name lists, timestamp configuration, nested
values, and empty-input behavior. Provide a reusable Rust
create_dataframe_from_arrow entry point with batch-schema validation.
This builds on SPARK-60586 /
https://github.com/apache/spark-connect-rust/pull/108, which implements the
reverse Rust-to-PyArrow boundary. No workflow changes are needed.
Validation includes core Rust unit tests, full integration-test compilation
with and without wasm-udf, the drop-in Python suite, and live-server Arrow
input cases on PyArrow 10 and 26. Existing workspace lint and API gaps are
tracked separately.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
