holdenk commented on code in PR #58185:
URL: https://github.com/apache/spark/pull/58185#discussion_r3832939548
##########
python/pyspark/sql/transpile.py:
##########
@@ -611,64 +624,31 @@ def _convert_chunk(self, params: List[str], body: ast.AST
| None) -> Column:
"is not supported by the transpiler"
)
case ast.BinOp(left=left, op=op, right=right):
- # Operator selection is driven by the operand *categories*
under
- # the current input-type variant (see ``_category``): Python's
- # `+` / `*` are overloaded for text. `+` -> add (num,num) or
- # concat (str,str); `*` -> multiply (num,num) or repeat
(str,int
- # / int,str); `-` / `%` are numeric-only. Combos that don't fit
- # (str+int, str-str, ...) raise so this variant is dropped and
- # the JVM picks another option or falls back to the Python UDF.
- #
- # `**` is intentionally NOT lowered: Spark's `pow` is DOUBLE
and
- # loses precision for large integers, so it would silently
return
- # wrong results. TODO (SPARK-55210): add an exact integer-power
- # lowering and re-enable it.
- #
- # Value-level divergences remain documented (need runtime value
- # info, not type): overflow raises ARITHMETIC_OVERFLOW under
ANSI
- # where Python promotes to a big int; arithmetic is not
- # NULL-guarded (`x + 1` on NULL -> NULL vs Python TypeError).
- # TODO (SPARK-55210): map overflow / divide-by-zero precisely.
+ # Numeric arithmetic is never lowered: fixed-width Catalyst
Review Comment:
Ok so this is a design choice we can think about, I'm in favor of
transpiling the numeric operators even if overflow is possible provided we
document that behavior. Otherwise we could catch the overflow and promote (for
example try_... and then on nulls we promote).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
