cloud-fan commented on code in PR #58185:
URL: https://github.com/apache/spark/pull/58185#discussion_r3843398545
##########
python/pyspark/sql/transpile.py:
##########
@@ -611,64 +624,31 @@ def _convert_chunk(self, params: List[str], body: ast.AST
| None) -> Column:
"is not supported by the transpiler"
)
case ast.BinOp(left=left, op=op, right=right):
- # Operator selection is driven by the operand *categories*
under
- # the current input-type variant (see ``_category``): Python's
- # `+` / `*` are overloaded for text. `+` -> add (num,num) or
- # concat (str,str); `*` -> multiply (num,num) or repeat
(str,int
- # / int,str); `-` / `%` are numeric-only. Combos that don't fit
- # (str+int, str-str, ...) raise so this variant is dropped and
- # the JVM picks another option or falls back to the Python UDF.
- #
- # `**` is intentionally NOT lowered: Spark's `pow` is DOUBLE
and
- # loses precision for large integers, so it would silently
return
- # wrong results. TODO (SPARK-55210): add an exact integer-power
- # lowering and re-enable it.
- #
- # Value-level divergences remain documented (need runtime value
- # info, not type): overflow raises ARITHMETIC_OVERFLOW under
ANSI
- # where Python promotes to a big int; arithmetic is not
- # NULL-guarded (`x + 1` on NULL -> NULL vs Python TypeError).
- # TODO (SPARK-55210): map overflow / divide-by-zero precisely.
+ # Numeric arithmetic is never lowered: fixed-width Catalyst
Review Comment:
Agreed. For v0, I restored numeric and unary transpilation and kept ANSI
overflow as a documented divergence. Exact promotion via try_* or a dedicated
Catalyst expression can follow under SPARK-55210.
The PR still fixes NULL handling, NaN comparisons, and string concatenation;
unsafe string repetition falls back to Python. I also fixed the related CI and
formatting failures in cb4b8423fb0. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
