DB Tsai created SPARK-60640:
-------------------------------
Summary: Avoid intermediate Value clones during local Arrow input
encoding in Spark Connect Rust
Key: SPARK-60640
URL: https://issues.apache.org/jira/browse/SPARK-60640
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Fix For: connect-rust-4.3.0
h2. Problem
Follow-up to SPARK-60606 /
https://github.com/apache/spark-connect-rust/pull/109 (reviewed revision
f26ea7e).
rows_to_arrow_ipc gathers each column into Vec<Value> by cloning the input
values before calling values_to_arrow. For String and Binary values this copies
payloads that remain available in the original Rows throughout conversion. The
array, map and struct encoder arms also clone child Values into intermediate
vectors before recursively encoding them. Removing the redundant coercion pass
reduced copies, but these intermediate copies remain.
h2. Proposed scope
Explore a borrowed encoder interface (for example a slice of references or an
iterator) that can consume original Values and nested children without
deep-cloning their payloads. Choose the interface after accounting for null
placeholders, field-name/positional lookup, map keys, and recursive lifetimes.
Preserve current numeric conversion rules, nullability, duplicate-name lookup
semantics, schema handling, and error order.
This is an optimization of intermediate Value storage, not a claim of zero-copy
Arrow construction: Arrow output buffers and IPC serialization can still
allocate/copy.
Python spec_names copying is separate work already proposed in
https://github.com/apache/spark-connect-rust/pull/113. SPARK-60586 concerns a
broader shared Python/Arrow converter and does not cover this specific local
Row-to-Arrow allocation path.
h2. Acceptance criteria
* Regression coverage through the complete local-input IPC path for
strings/binary (including Unicode, embedded NULs, empty values and nulls),
arrays, maps, and structs; cover name/position fallback and duplicate names.
* Confirm encoded output owns its buffers and remains valid after source Rows
are dropped.
* Preserve error ordering and existing supported conversions; do not fold
unrelated conversion-policy changes into the optimization.
* Run Rust tests, Python input fixtures, and live Spark parity where applicable.
* Measure fresh release builds against a pinned baseline using
string/binary-heavy and nested inputs, with numeric-input regression guards.
Report CPU, wall time, and memory separately; RSS alone does not establish
allocation counts or zero-copy behavior.
Generated with OpenAI Codex.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
