[
https://issues.apache.org/jira/browse/SPARK-60640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60640:
-----------------------------------
Labels: pull-request-available (was: )
> Avoid intermediate Value clones during local Arrow input encoding in Spark
> Connect Rust
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-60640
> URL: https://issues.apache.org/jira/browse/SPARK-60640
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> h2. Problem
> Follow-up to SPARK-60606 /
> https://github.com/apache/spark-connect-rust/pull/109 (reviewed revision
> f26ea7e).
> rows_to_arrow_ipc gathers each column into Vec<Value> by cloning the input
> values before calling values_to_arrow. For String and Binary values this
> copies payloads that remain available in the original Rows throughout
> conversion. The array, map and struct encoder arms also clone child Values
> into intermediate vectors before recursively encoding them. Removing the
> redundant coercion pass reduced copies, but these intermediate copies remain.
> h2. Proposed scope
> Explore a borrowed encoder interface (for example a slice of references or an
> iterator) that can consume original Values and nested children without
> deep-cloning their payloads. Choose the interface after accounting for null
> placeholders, field-name/positional lookup, map keys, and recursive
> lifetimes. Preserve current numeric conversion rules, nullability,
> duplicate-name lookup semantics, schema handling, and error order.
> This is an optimization of intermediate Value storage, not a claim of
> zero-copy Arrow construction: Arrow output buffers and IPC serialization can
> still allocate/copy.
> Python spec_names copying is separate work already proposed in
> https://github.com/apache/spark-connect-rust/pull/113. SPARK-60586 concerns a
> broader shared Python/Arrow converter and does not cover this specific local
> Row-to-Arrow allocation path.
> h2. Acceptance criteria
> * Regression coverage through the complete local-input IPC path for
> strings/binary (including Unicode, embedded NULs, empty values and nulls),
> arrays, maps, and structs; cover name/position fallback and duplicate names.
> * Confirm encoded output owns its buffers and remains valid after source Rows
> are dropped.
> * Preserve error ordering and existing supported conversions; do not fold
> unrelated conversion-policy changes into the optimization.
> * Run Rust tests, Python input fixtures, and live Spark parity where
> applicable.
> * Measure fresh release builds against a pinned baseline using
> string/binary-heavy and nested inputs, with numeric-input regression guards.
> Report CPU, wall time, and memory separately; RSS alone does not establish
> allocation counts or zero-copy behavior.
> Generated with OpenAI Codex.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
