[
https://issues.apache.org/jira/browse/SPARK-60687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60687:
-----------------------------------
Labels: pull-request-available (was: )
> Avoid numeric probes for exact list and tuple scalar arrays in Spark Connect
> Rust
> ---------------------------------------------------------------------------------
>
> Key: SPARK-60687
> URL: https://issues.apache.org/jira/browse/SPARK-60687
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Frozen nested array tuple input10000x3 takes39.98ms Rust vs2.93ms Python;
> pandas array42.07ms vs6.30ms. py_to_value performs Decimal/numeric probes
> before recognizing built-in sequence arrays, repeated at every nesting level.
> Investigate exact list/tuple dispatch before numeric probes with recursive
> conversion, preserving subclasses and custom numeric protocols on fallback,
> row-as-struct behavior, binary distinctions, nesting/nulls and errors.
> Separate from PR111 exact numbers, PR113 row field names, PR114 borrowed
> encoding. Compare on a baseline containing PR111 too to isolate incremental
> array-container benefit; publish only if worthwhile.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
