DB Tsai created SPARK-60662:
-------------------------------
Summary: Share field names across local input rows in Spark
Connect Rust
Key: SPARK-60662
URL: https://issues.apache.org/jira/browse/SPARK-60662
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
The Python binding currently clones explicit-schema or pandas column names for
every row during local createDataFrame conversion. These names are immutable
within one conversion, so repeating their String allocations wastes conversion
work and retained memory.
Share the names once per conversion using the existing StructValue/Row
shared-name representation (Arc<[String]>). Preserve explicit-schema precedence
over Row/dict/pandas names, positional defaults, input parsing, owned values,
and error order. Each call must own its names so later caller mutations cannot
change a previously created DataFrame. No intended API or behavior change.
PR: https://github.com/apache/spark-connect-rust/pull/113
Validation should cover complete IPC names and values, duplicate/empty/Unicode
names, nulls, empty input, caller mutations, and overridden-input errors. The
PR reports 410 Rust checks and 274 offline Python tests passing, including four
isolated input-name tests. Full live Spark/official-suite parity remains
pending.
On macOS arm64/Python 3.12, independent confirmation medians against c36c846
show CPU reductions of 21.50% for 10,000 x 3 explicit Rows, 7.92% for 10,000 x
32 tuples, and 7.18% for 10,000 x 32 pandas input. The latter two held-RSS
reductions are 17.367 MiB and 22.461 MiB. Each screening/confirmation phase
used 30 fresh processes per build/case across 48 cases with a same-binary
control.
These are preliminary results: the draft has not passed the complete CPU or
memory acceptance policy. Five timing metrics per phase have inconclusive guard
confidence bounds; no peak-RSS target qualifies. Both Rust builds retain
pre-existing Duration(Microsecond) warmup failures tracked by SPARK-60607. RSS
does not establish allocation counts, and effects from separate patches are not
additive.
Frozen benchmark:
https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
Related but distinct: SPARK-60585 and SPARK-60588 share decoded output
row/struct names; SPARK-60640 concerns intermediate Value clones during local
Arrow encoding.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
