Spenserrrr opened a new pull request, #58488: URL: https://github.com/apache/spark/pull/58488
**WIP, opened for a scope discussion rather than for merge.** The change works, but the divergence it fixes is wider than what it covers. See the last section. ### What changes were proposed in this pull request? Restore NumPy's output dtype when a pandas-on-Spark ufunc receives a float32 column. The dispatch asks NumPy which dtype it would return for the operand dtypes, rather than reimplementing NumPy's promotion rules, and casts back when Spark widened the result. ### Why are the changes needed? Spark's math functions return a double for a float input, so a ufunc on a float32 column returns float64 where pandas returns float32, losing float32's range with it: `np.exp(np.float32(90))` gives `1.22e39` here and `inf` in pandas. This is long-standing rather than new, since the previous `pandas_udf` entries declared `DoubleType` and widened identically, and it affects 40 mapping entries including `sqrt`, which has no cast at all. ### Does this PR introduce _any_ user-facing change? Yes. A ufunc on a float32 column returns float32 instead of float64, and values outside the float32 range now overflow to infinity and underflow to zero as they do in pandas. ### How was this patch tested? New `test_np_float32_output_dtypes` and `test_np_float32_range` in `NumPyCompatTestsMixin`, which fail without the change; their float32 assertions are strict rather than `almost=True`, since only a strict comparison checks dtype. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) ### Open question on scope float32 is not the only divergence. Sweeping every input dtype, the output dtype also differs for narrow integer and boolean inputs, for `reciprocal`/`sign`/`square`/`trunc` on integers, for `ceil`/`floor`, and for decimal columns. Binary ufuncs across dtype pairs are not swept yet. The narrow-integer case is worth deciding with this PR, since the range argument above applies to int16 too; the rest look like separate tickets. Which scope would you prefer: 1. Restore NumPy's float output dtype wherever Spark can represent it, so float32 plus int16/Int8/Int16/boolean. One condition on top of this PR. 2. The float32 case only, as this PR stands. 3. Treat double output as intended and document it instead. int8 and boolean inputs make NumPy return float16, which Spark has no type for, so exact parity is not reachable there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
