[
https://issues.apache.org/jira/browse/SPARK-60676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60676:
-----------------------------------
Labels: pull-request-available (was: )
> Convert pandas inputs through columnar Arrow in Spark Connect Rust
> ------------------------------------------------------------------
>
> Key: SPARK-60676
> URL: https://issues.apache.org/jira/browse/SPARK-60676
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The Rust Python client currently converts pandas DataFrame input through
> Python row objects. In the frozen client-only benchmark, a prepared 1,000,000
> x 3 int64 DataFrame takes about 1166.75 ms versus 6.71 ms for the official
> Python client, and sampled peak RSS is about 852 versus 212 MiB. Investigate
> and introduce a columnar Arrow fast path, preserving explicit/inferred
> schemas, integer precision, nulls, temporal semantics, and fallback
> compatibility. Reuse the Arrow Table input work in PR #118 if appropriate and
> disclose any dependency. Validate with unchanged public-API benchmark cases,
> value/schema checks and regression guards. Baseline Rust: d884e347452c;
> harness: dbtsai/pyspark-connect-benchmark at 1bbdec9f3665. This ticket is for
> the pandas input path, not Arrow Table support already tracked by SPARK-60675.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
