[
https://issues.apache.org/jira/browse/SPARK-60675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-60675:
----------------------------
Fix Version/s: connect-rust-4.3.0
> Support PyArrow Table input through the C Stream interface in Spark Connect
> Rust
> --------------------------------------------------------------------------------
>
> Key: SPARK-60675
> URL: https://issues.apache.org/jira/browse/SPARK-60675
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> The native Rust-backed SparkSession.createDataFrame accepts Python lists and
> pandas DataFrames, but a pyarrow.Table currently reaches the list cast and
> fails.
> Support pyarrow.Table input through arrow-rs's owned C Stream importer. Reuse
> the existing PySpark Arrow schema and timestamp helpers for
> Connect-compatible normalization, import a RecordBatchReader, and serialize
> the Rust batches directly into the local-relation IPC payload. Avoid
> materializing column values as Python rows. Casts, buffer alignment and IPC
> serialization may still copy.
> Preserve explicit schemas, column-name lists, timestamp configuration, nested
> values, and empty-input behavior. Provide a reusable Rust
> create_dataframe_from_arrow entry point with batch-schema validation.
> This builds on SPARK-60586 /
> https://github.com/apache/spark-connect-rust/pull/108, which implements the
> reverse Rust-to-PyArrow boundary. No workflow changes are needed.
> Validation includes core Rust unit tests, full integration-test compilation
> with and without wasm-udf, the drop-in Python suite, and live-server Arrow
> input cases on PyArrow 10 and 26. Existing workspace lint and API gaps are
> tracked separately.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
