[
https://issues.apache.org/jira/browse/SPARK-60699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60699:
-----------------------------------
Labels: pull-request-available (was: )
> Build Arrow columns directly for exact integer tuple input in Spark Connect
> Rust
> --------------------------------------------------------------------------------
>
> Key: SPARK-60699
> URL: https://issues.apache.org/jira/browse/SPARK-60699
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The frozen combined client benchmark input-scale-tuples-1000000x3
> remains140.09ms vsPython107.57ms after borrowed-value encoding/shared field
> names/exact builtin dispatch. Current Python binding materializes a
> Vec<Value> per tuple then a million Row objects before columnar Arrow
> construction. Evaluate a conservative direct Arrow-column path limited to
> exact built-in tuple rows and exact int/None cells with an explicit
> compatible LongType struct schema, avoiding intermediate per-row allocations.
> Preserve schema/name/nullability/error behavior, integer overflow, empty
> inputs, mixed/subclass fallback and offline transport semantics. Require
> frozen paired baseline/candidate benchmarks with identical existing input
> optimizations on both sides and independent all-cell IPC/schema validators.
> Publish only with meaningful measured improvement and no material guard
> regression. This is distinct from PR111 exact-number detection, PR113 name
> sharing and PR114 borrowing existing Value rows; it removes intermediate row
> materialization only for the supported narrow input. No JVM.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
