cloud-fan opened a new pull request, #58185:
URL: https://github.com/apache/spark/pull/58185
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR:
https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g.,
'[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a
faster review.
7. If you want to add a new configuration, please read the guideline first
for naming configurations in
'common/utils/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the
guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
Followup to https://github.com/apache/spark/pull/56327.
This change makes the experimental Python UDF transpiler preserve Python
semantics more
conservatively. Numeric and non-literal unary arithmetic, modulo, and string
repetition fall back
to interpreted Python when fixed-width Catalyst operations cannot preserve
Python behavior.
String concatenation raises on null inputs as Python would, and numeric
equality and ordering
handle NaN according to Python semantics.
The transpilation unit tests are updated to assert the safe fallback paths
and the corrected NaN
behavior.
### Why are the changes needed?
Python integers have arbitrary precision, while Catalyst numeric arithmetic
uses fixed-width
types. Lowering these expressions can therefore overflow or otherwise
produce behavior different
from the original Python UDF. Spark also treats NaN differently from Python
for equality and
ordering, and Catalyst normally propagates null through concatenation where
Python raises a
TypeError. The transpiler must fail closed whenever it cannot preserve the
source UDF's behavior.
### Does this PR introduce _any_ user-facing change?
Yes. When the unreleased experimental Python UDF transpilation feature is
explicitly enabled,
unsafe arithmetic expressions now remain interpreted Python UDFs, and
transpiled comparisons and
string concatenation more closely match Python behavior.
### How was this patch tested?
Updated `pyspark.sql.tests.test_udf_transpile_unit` with positive and
negative cases covering
numeric fallback, unary operations, modulo, string operations, nulls, and
NaN comparisons.
- `build/sbt -Phive package`
- `python/run-tests --testnames pyspark.sql.tests.test_udf_transpile_unit`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
