DB Tsai created SPARK-60661:
-------------------------------
Summary: Avoid generic type detection for exact built-in Python
numbers in spark-connect-rust
Key: SPARK-60661
URL: https://issues.apache.org/jira/browse/SPARK-60661
Project: Spark
Issue Type: Improvement
Components: PySpark, Connect
Affects Versions: connect-rust-4.2.0
Reporter: DB Tsai
Numeric Python values in spark-connect-rust currently pass through generic
bool/Decimal detection in py_to_value. Avoid this overhead for exact built-in
int and float values by using cast_exact before the generic path. Integers that
fit i64 convert directly to Value::Long; exact floats convert to Value::Double.
Booleans, subclasses, custom conversion protocols, and overflowing integers
retain the existing fallback behavior. No behavior or API change is intended.
Add regression coverage for values and types, float bit preservation (including
NaN payloads and signed zero), subclasses, Decimal, conversion protocols, and
nested inputs. Integer overflow coercion remains a separate existing PySpark
parity gap.
Original-patch confirmation medians versus c36c846 on macOS arm64 / Python 3.12
showed CPU reductions of 74.39% for 10,000 x 32 tuples, 41.92% for 10,000 x 3
tuples, and 37.80% for numeric arrays. These measurements include conversion
and IPC serialization. The cast_exact cleanup has not been rebenchmarked; full
CPU/memory gates remain unmet because of known SPARK-60607 failures and
inconclusive regression bounds.
The PR reports 289 offline Python tests passing with an unchanged 108-class API
snapshot, plus 410 Rust unit/golden/sharing checks. Full live
Spark/official-suite parity remains pending.
Pull request: https://github.com/apache/spark-connect-rust/pull/111
Frozen benchmark harness:
https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
