[
https://issues.apache.org/jira/browse/SPARK-60720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60720:
-----------------------------------
Labels: pull-request-available (was: )
> Use Arrow columns for supported pandas string input in Spark Connect Rust
> -------------------------------------------------------------------------
>
> Key: SPARK-60720
> URL: https://issues.apache.org/jira/browse/SPARK-60720
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Frozen combined10000x3 pandas string workload remains3.986ms vsPython1.823ms
> after SPARK-60684/PR127 exact scalar string optimization. Evaluate
> conservative columnar explicitStringType pandas object columns with exact str
> and permittedNone cells, authentic native schema/type descriptors, ordinary
> DataFrame and measured small-frame threshold. Build Arrow string columns and
> reuse importer instead of pandas row materialization/Rust scalar conversion.
> Subclasses, mutated descriptors/rebindings, inferred/DDL/mixed/extension
> dtypes and unsupportedcollation/schema cases retain fallback. Lone-surrogate
> Arrow conversion failures must return to old row path before importer,
> preserving existing error behavior/order. Validate every
> IPCcell/schema/names, Unicode/NUL/empty/all-null/nonnullable,
> metadata/collation/large-var-types and fallbackhooks. Benchmark
> identicalPR127-containing frozenbaseline/candidate with unchanged harness
> fivefreshsamples and full-stack supplement; publish only meaningful
> repeatablegain. Distinct from PR121int64/PR130timestamp/SPARK60716date.
> User-authorized revised cutoff2026-10-29T18:19:36Z; noJVM/liveSpark needed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
