[
https://issues.apache.org/jira/browse/SPARK-60745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60745:
-----------------------------------
Labels: pull-request-available (was: )
> Avoid row materialization for Arrow-backed pandas string input in Spark
> Connect Rust
> ------------------------------------------------------------------------------------
>
> Key: SPARK-60745
> URL: https://issues.apache.org/jira/browse/SPARK-60745
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Pandas string extension columns still use Python row materialization in Spark
> Connect Rust. SPARK-60720 / PR144 deliberately supports concrete NumPy object
> storage and excludes extension dtypes. Its exact candidate
> d31870bc4ceb15c3cadc4bc0a44fcb16f7b5e723 retains the pandas 3.1 StringDtype
> benchmark at 22.310792 ms warm / 24.347666 ms first construction for 10,000 x
> 3 input. The old benchmark did not separately record the inferred backing
> class, so these timings motivate investigation rather than establish an
> ArrowStringArray speedup.
> Evaluate one conservative path for ordinary canonical Arrow-backed pandas
> StringDtype (exact ArrowStringArray with authenticated PyArrow chunks) and an
> explicit native all-StringType / UTF8_BINARY schema. Reuse the existing
> internal inline IPC transport without Python row materialization. Preserve
> field names, logical schema/metadata, nullable/missing semantics, offsets,
> chunks/slices and standard UTF8 representation. Guard customized
> DataFrame/columns/conversion/schema/storage hooks before touching data;
> Python StringArray, generic ArrowDtype, mixed and unsupported inputs retain
> the old path.
> This is a distinct extension-storage follow-up to PR144, not a widening of
> its frozen object-string measurements. SPARK-55294 concerns pandas-on-Spark
> type restoration and does not implement this Rust-client input path. Bounded
> upstream PR/JIRA searches and a source-only assessment are preserved in the
> campaign's pandas-arrow-string-feasibility directory.
> Validate canonical storage identities explicitly on pandas 2.3.3 and 3.1.0.
> Cover pd.NA/NaN, nonnullable missing values, customized aliases/hooks and
> error order, chunks/slices, string versus large_string conversion and
> overflow/fallback, Unicode and inline transport/RPC behavior. Preserve
> unchanged frozen workloads and validators, compare separate exact
> baseline/candidate wheels/environments, and report supported gains plus
> guard, cold/first-construction, fallback, memory and RPC costs. Do not route
> through the public Arrow importer if that introduces Config calls or cached
> uploads. No JVM/live Spark is required. Publish only a separately reviewed
> draft with meaningful measured gain; otherwise preserve local results.
> The campaign hard cutoff is 2026-10-30T14:00:00Z (07:00 America/Los_Angeles);
> all execution must stop by 13:50 UTC.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
