Spenserrrr opened a new pull request, #58397: URL: https://github.com/apache/spark/pull/58397
### What changes were proposed in this pull request? `IntegralOps` and `FractionalOps` computed `//` as `F.floor(a / b)`, for both the forward and the reflected operator. The four methods now reuse `_floor_divide_func`, the expression SPARK-58581 added for the `np.floor_divide` mapping, which this PR first moves from `numpy_compat.py` to `pyspark.pandas.utils` so that the core operator does not have to import the NumPy compatibility layer. That move is a pure relocation. ### Why are the changes needed? `F.floor(a / b)` is wrong in three ways and fails outright in a fourth: - the division rounds before the floor, so `ps.Series([1.0]) // 0.1` returns `10.0` where pandas returns `9.0`; - integral operands are cast to double first, so `-(2**53 + 1) // 2` returns `-4503599627370496.0` instead of `-4503599627370497`; - an infinite divisor returns `0.0` for every dividend, where pandas derives the quotient from the remainder and returns `-1.0` for opposite signs; - `F.floor` returns a bigint, so an infinite or out-of-range quotient raises `ARITHMETIC_OVERFLOW` under ANSI mode and saturates to `Long.MAX_VALUE` without it — for example `ps.Series([np.inf]) // 2.0` and `1e300 // 1e-300`. The reflected operator additionally raises `DIVIDE_BY_ZERO` for `0 // ps.Series([0])`, because its zero-divisor branch divides an infinity by the dividend. SPARK-58581 fixed the same arithmetic in the mapping, but that entry is unreachable: `np.floor_divide` is dispatched to this dunder operation before the mapping registry is consulted. Reusing the expression makes both paths agree, and the zero-divisor answers now come from literals instead of a division, so the result no longer depends on ANSI mode. A quotient above `2**53` is still rounded, since the result type has to stay double for the infinities a zero divisor produces. ### Does this PR introduce _any_ user-facing change? Yes. `//` on a pandas-on-Spark object returns pandas' result in the cases above, instead of a wrong value or an error. For example, `ps.Series([1.0, 10.0]) // 0.1` now returns `[9.0, 99.0]` rather than `[10.0, 100.0]`. ### How was this patch tested? Rows added to the existing `test_floordiv` and `test_rfloordiv` for a non-representable divisor, operands above `2**53`, infinite dividends and divisors, an overflowing quotient, negative zeros, and the most negative long. Each row was confirmed to fail or error against the unmodified source. Existing suites pass: `test_num_mul_div`, `test_num_reverse`, `test_num_arithmetic`, `test_num_ops`, `test_boolean_ops`, `test_null_ops`, `test_numpy_compat`, `computation/test_binary_ops`, `series/test_stat`, `test_utils`, and the Connect parity suites, both with ANSI mode on and with `SPARK_ANSI_SQL_MODE=false`. Also verified against pandas with a random sweep through the operator: 4000 float pairs and 1869 integer pairs whose quotient fits a double, with no mismatches, plus the 81 combinations of `[1.0, -1.0, 0.0, -0.0, 2.5, inf, -inf, nan, None]`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
