[
https://issues.apache.org/jira/browse/SPARK-60661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-60661.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 111
[https://github.com/apache/spark-connect-rust/pull/111]
> Avoid generic detection for exact built-in numbers in Spark Connect Rust
> ------------------------------------------------------------------------
>
> Key: SPARK-60661
> URL: https://issues.apache.org/jira/browse/SPARK-60661
> Project: Spark
> Issue Type: Improvement
> Components: Connect, PySpark
> Affects Versions: connect-rust-4.2.0, connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Current scope after the master rebase
> In pandas row conversion and other generic callers of py_to_value, exact
> built-in int and float values still pass through generic Decimal and
> numeric-detection work. Dispatch these exact types directly while retaining
> the existing fallback for other objects.
> The remaining scope is the pandas row path and other py_to_value callers. The
> rewritten local createDataFrame inference and conversion path already
> dispatches built-in numbers directly, so this issue does not restore or
> optimize that older local-input path.
> Preserve behavior for bools, subclasses, Decimal, custom numeric protocols,
> overflowing integers, nested values, and special floating-point values,
> including NaN, infinities, and signed zero. Keep current local-input
> inference and error behavior intact.
> Regression tests should use object-backed pandas inputs where needed to
> exercise the remaining generic path. Rebenchmark the current patch against
> current master before claiming CPU, wall-time, or memory improvements; the
> historical local tuple and Row results do not validate the rebased scope.
> Live-Spark parity and current CI remain part of validation.
> Proposed patch: https://github.com/apache/spark-connect-rust/pull/111
> Historical pre-rebase description and validation
> The text below records the original proposal and its historical measurements
> and validation. It does not validate or measure the rebased patch. The
> current scope above takes precedence.
> Numeric Python values in spark-connect-rust currently pass through generic
> bool/Decimal detection in py_to_value. Avoid this overhead for exact built-in
> int and float values by using cast_exact before the generic path. Integers
> that fit i64 convert directly to Value::Long; exact floats convert to
> Value::Double. Booleans, subclasses, custom conversion protocols, and
> overflowing integers retain the existing fallback behavior. No behavior or
> API change is intended.
> Add regression coverage for values and types, float bit preservation
> (including NaN payloads and signed zero), subclasses, Decimal, conversion
> protocols, and nested inputs. Integer overflow coercion remains a separate
> existing PySpark parity gap.
> Original-patch confirmation medians versus c36c846 on macOS arm64 / Python
> 3.12 showed CPU reductions of 74.39% for 10,000 x 32 tuples, 41.92% for
> 10,000 x 3 tuples, and 37.80% for numeric arrays. These measurements include
> conversion and IPC serialization. The cast_exact cleanup has not been
> rebenchmarked; full CPU/memory gates remain unmet because of known
> SPARK-60607 failures and inconclusive regression bounds.
> The PR reports 289 offline Python tests passing with an unchanged 108-class
> API snapshot, plus 410 Rust unit/golden/sharing checks. Full live
> Spark/official-suite parity remains pending.
> Pull request: https://github.com/apache/spark-connect-rust/pull/111
> Frozen benchmark harness:
> https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
