Spenserrrr opened a new pull request, #58366:
URL: https://github.com/apache/spark/pull/58366
### What changes were proposed in this pull request?
`_fmax_func` and `_fmin_func` no longer cast their result to double. Every
branch of both expressions returns one of the operands, so the result now keeps
the operands' type instead of being widened.
### Why are the changes needed?
The cast loses precision for integral columns and reports a dtype NumPy does
not use:
```python
pdf = pd.DataFrame({"x1": [2**53 + 1], "x2": [2]})
np.fmax(ps.from_pandas(pdf).x1, ps.from_pandas(pdf).x2) #
9007199254740992.0, float64
np.fmax(pdf.x1, pdf.x2) # 9007199254740993,
int64
```
`np.fmin` loses the same value for `-(2**53 + 1)`. Dropping the cast also
makes the output dtype match NumPy for narrower integers, float32, boolean,
decimal, and datetime columns. The sibling `maximum` / `minimum` mappings never
cast and are already exact.
### Does this PR introduce _any_ user-facing change?
Yes. `np.fmax` / `np.fmin` on a non-double column now return the operands'
type rather than double, and integral values above 2^53 are exact.
Floating-point results, including the signed-zero tie, are unchanged.
### How was this patch tested?
Two new tests in `test_numpy_compat.py`, inherited by the Spark Connect
parity suite: one for the values above 2^53, one for narrower integers,
float32, boolean, decimal, and datetime columns. Both fail on the unfixed
expression.
Neither is gated by `_skip_if_numpy_differs`, because nothing they compare
is environment-dependent: the NumPy reference is identical on the minimum
dependencies (NumPy 1.23.2 / pandas 2.2.0), on NumPy 2.4.1 / pandas 2.3.3, and
on NumPy 2.5.2 / pandas 3.0.5, verified locally for every frame, and the
equal-operand rows use non-zero pairs so no signed-zero tie is involved.
Full `NumPyCompatTests` (25) and `NumPyCompatParityTests` (25) pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
