[
https://issues.apache.org/jira/browse/SPARK-60662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60662:
-----------------------------------
Labels: pull-request-available (was: )
> Share field names across pandas input rows in Spark Connect Rust
> ----------------------------------------------------------------
>
> Key: SPARK-60662
> URL: https://issues.apache.org/jira/browse/SPARK-60662
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Current scope after the master rebase
> The pandas row-conversion path clones the complete field-name vector into
> each materialized row. Share one immutable field-name allocation across the
> rows of a conversion using the existing StructValue and Row shared-name
> representation.
> The remaining scope is pandas row conversion. The rewritten local-input path
> already shares names, and supported columnar pandas fast paths can bypass row
> materialization while retaining a fallback that still benefits from this
> change.
> Preserve field-name and schema precedence, duplicate and Unicode names, owned
> row values, null and empty input behavior, caller-mutation isolation, and
> conversion/error behavior. Regression coverage should exercise the pandas
> path and verify complete names and values while keeping ordinary local-input
> semantics intact.
> Rebenchmark the current head against current master before claiming CPU,
> wall-time, or memory savings. Historical local tuple and Row results do not
> validate this narrowed scope; fresh measurements should establish the
> remaining pandas benefit. Live-Spark parity and current CI remain part of
> validation.
> Proposed patch: https://github.com/apache/spark-connect-rust/pull/113
> Historical pre-rebase description and validation
> The text below records the original proposal and its historical measurements
> and validation. It does not validate or measure the rebased patch. The
> current scope above takes precedence.
> The Python binding currently clones explicit-schema or pandas column names
> for every row during local createDataFrame conversion. These names are
> immutable within one conversion, so repeating their String allocations wastes
> conversion work and retained memory.
> Share the names once per conversion using the existing StructValue/Row
> shared-name representation (Arc<[String]>). Preserve explicit-schema
> precedence over Row/dict/pandas names, positional defaults, input parsing,
> owned values, and error order. Each call must own its names so later caller
> mutations cannot change a previously created DataFrame. No intended API or
> behavior change.
> PR: https://github.com/apache/spark-connect-rust/pull/113
> Validation should cover complete IPC names and values,
> duplicate/empty/Unicode names, nulls, empty input, caller mutations, and
> overridden-input errors. The PR reports 410 Rust checks and 274 offline
> Python tests passing, including four isolated input-name tests. Full live
> Spark/official-suite parity remains pending.
> On macOS arm64/Python 3.12, independent confirmation medians against c36c846
> show CPU reductions of 21.50% for 10,000 x 3 explicit Rows, 7.92% for 10,000
> x 32 tuples, and 7.18% for 10,000 x 32 pandas input. The latter two held-RSS
> reductions are 17.367 MiB and 22.461 MiB. Each screening/confirmation phase
> used 30 fresh processes per build/case across 48 cases with a same-binary
> control.
> These are preliminary results: the draft has not passed the complete CPU or
> memory acceptance policy. Five timing metrics per phase have inconclusive
> guard confidence bounds; no peak-RSS target qualifies. Both Rust builds
> retain pre-existing Duration(Microsecond) warmup failures tracked by
> SPARK-60607. RSS does not establish allocation counts, and effects from
> separate patches are not additive.
> Frozen benchmark:
> https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
> Related but distinct: SPARK-60585 and SPARK-60588 share decoded output
> row/struct names; SPARK-60640 concerns intermediate Value clones during local
> Arrow encoding.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
