DB Tsai created SPARK-60684:
-------------------------------
Summary: Avoid numeric conversion probes for exact built-in string
inputs in Spark Connect Rust
Key: SPARK-60684
URL: https://issues.apache.org/jira/browse/SPARK-60684
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Assignee: DB Tsai
Fix For: connect-rust-4.3.0
Frozen 10k x3 string tuple input takes20.36ms in Rust vs2.73ms Python; pandas
string input21.85ms vs1.90ms. py_to_value currently performs
bool/Decimal/int/float probes before recognizing strings. Add a narrow exact
built-in str path that avoids numeric probes and repeated decimal module access
while preserving string subclasses and their conversion protocols on the
existing fallback. This is separate from PR111 exact int/float conversion and
PR114 borrowed Arrow encoding. Benchmark paired frozen input cases and guard
unaffected numeric/subclass/null/nested behavior.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
