DB Tsai created SPARK-60666:
-------------------------------
Summary: Dispatch Arrow decoding once per typed column in Spark
Connect Rust
Key: SPARK-60666
URL: https://issues.apache.org/jira/browse/SPARK-60666
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: 5.0.0
Reporter: DB Tsai
Arrow row materialization in spark-connect-rust repeatedly downcasts column
arrays and slices list/map children. Resolve a borrowed typed decoder once per
Arrow column and nested child, then reuse it during row materialization.
Preserve row-major evaluation, lazy errors, absolute child offsets, and owned
output values. No behavior or API change is intended. Add regression coverage
for sliced and nested arrays, nulls, exact scalar values, empty IPC boundary
batches, duplicate variant fields, and error ordering.
Proposed implementation: https://github.com/apache/spark-connect-rust/pull/112
The PR reports about 9% lower CPU and 8.6–8.9% lower wall time for fully
consumed depth-16 mixed output versus c36c846, with 0.83–0.94 MiB higher held
RSS. Confirmation used 30 processes/build/case across 38 cases, isolated
release wheels, and a same-binary control. No memory saving or allocation-count
reduction is established.
Benchmark harness:
https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
Reported validation: 415 Rust unit/golden/sharing checks, both integration
compilation modes, formatting and core Clippy pass. A fresh release wheel
passes 273 offline Python tests, and the 108-class API snapshot is unchanged.
The PR remains draft: full live Spark/official-suite parity is pending.
Complete CPU/memory gates remain unmet due to known SPARK-60607 failures and
inconclusive regression bounds; workspace Clippy is blocked by a missing WASM
target.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
