[
https://issues.apache.org/jira/browse/SPARK-60666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-60666.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 112
[https://github.com/apache/spark-connect-rust/pull/112]
> Dispatch Arrow result decoding once per typed column in Spark Connect Rust
> --------------------------------------------------------------------------
>
> Key: SPARK-60666
> URL: https://issues.apache.org/jira/browse/SPARK-60666
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Current scope after the master rebase
> Arrow-to-Row materialization currently repeats dynamic array downcasts and
> container slicing while decoding values. Prepare a borrowed typed decoder
> once per Arrow column and nested child, then reuse it during row
> materialization.
> This is an Arrow result-decoding optimization, independent of local-input
> inference and pandas-input routing, with no intended API or behavior change.
> Returned Row and Value objects must continue to own their contents without
> retaining the Arrow arrays after decoding.
> Preserve row-major evaluation and error order, lazy errors for unused
> unsupported children, absolute child offsets in sliced and nested arrays,
> null handling, variant field selection, and exact scalar values. Regression
> coverage should include nested lists, structs and maps; sliced arrays; empty
> batches and containers; null parents; special floating-point bits; and
> conversion-error ordering.
> Rebenchmark the current head against current master before claiming CPU,
> wall-time, or memory improvements. Historical benchmark figures do not
> validate the rebased patch, and no current memory saving or allocation-count
> reduction is established. Live-Spark parity and current CI remain part of
> validation.
> Proposed patch: https://github.com/apache/spark-connect-rust/pull/112
> Historical pre-rebase description and validation
> The text below records the original proposal and its historical measurements
> and validation. It does not validate or measure the rebased patch. The
> current scope above takes precedence.
> Arrow row materialization in spark-connect-rust repeatedly downcasts column
> arrays and slices list/map children. Resolve a borrowed typed decoder once
> per Arrow column and nested child, then reuse it during row materialization.
> Preserve row-major evaluation, lazy errors, absolute child offsets, and owned
> output values. No behavior or API change is intended. Add regression coverage
> for sliced and nested arrays, nulls, exact scalar values, empty IPC boundary
> batches, duplicate variant fields, and error ordering.
> Proposed implementation: https://github.com/apache/spark-connect-rust/pull/112
> The PR reports about 9% lower CPU and 8.6–8.9% lower wall time for fully
> consumed depth-16 mixed output versus c36c846, with 0.83–0.94 MiB higher held
> RSS. Confirmation used 30 processes/build/case across 38 cases, isolated
> release wheels, and a same-binary control. No memory saving or
> allocation-count reduction is established.
> Benchmark harness:
> https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
> Reported validation: 415 Rust unit/golden/sharing checks, both integration
> compilation modes, formatting and core Clippy pass. A fresh release wheel
> passes 273 offline Python tests, and the 108-class API snapshot is unchanged.
> The PR remains draft: full live Spark/official-suite parity is pending.
> Complete CPU/memory gates remain unmet due to known SPARK-60607 failures and
> inconclusive regression bounds; workspace Clippy is blocked by a missing WASM
> target.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
