DB Tsai created SPARK-60676:
-------------------------------
Summary: Convert pandas inputs through columnar Arrow in Spark
Connect Rust
Key: SPARK-60676
URL: https://issues.apache.org/jira/browse/SPARK-60676
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Assignee: DB Tsai
Fix For: connect-rust-4.3.0
The Rust Python client currently converts pandas DataFrame input through Python
row objects. In the frozen client-only benchmark, a prepared 1,000,000 x 3
int64 DataFrame takes about 1166.75 ms versus 6.71 ms for the official Python
client, and sampled peak RSS is about 852 versus 212 MiB. Investigate and
introduce a columnar Arrow fast path, preserving explicit/inferred schemas,
integer precision, nulls, temporal semantics, and fallback compatibility. Reuse
the Arrow Table input work in PR #118 if appropriate and disclose any
dependency. Validate with unchanged public-API benchmark cases, value/schema
checks and regression guards. Baseline Rust: d884e347452c; harness:
dbtsai/pyspark-connect-benchmark at 1bbdec9f3665. This ticket is for the pandas
input path, not Arrow Table support already tracked by SPARK-60675.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
