holdenk commented on code in PR #58185:
URL: https://github.com/apache/spark/pull/58185#discussion_r3832948431
##########
python/pyspark/sql/transpile.py:
##########
@@ -611,64 +624,31 @@ def _convert_chunk(self, params: List[str], body: ast.AST
| None) -> Column:
"is not supported by the transpiler"
)
case ast.BinOp(left=left, op=op, right=right):
- # Operator selection is driven by the operand *categories*
under
- # the current input-type variant (see ``_category``): Python's
- # `+` / `*` are overloaded for text. `+` -> add (num,num) or
- # concat (str,str); `*` -> multiply (num,num) or repeat
(str,int
- # / int,str); `-` / `%` are numeric-only. Combos that don't fit
- # (str+int, str-str, ...) raise so this variant is dropped and
- # the JVM picks another option or falls back to the Python UDF.
- #
- # `**` is intentionally NOT lowered: Spark's `pow` is DOUBLE
and
- # loses precision for large integers, so it would silently
return
- # wrong results. TODO (SPARK-55210): add an exact integer-power
- # lowering and re-enable it.
- #
- # Value-level divergences remain documented (need runtime value
- # info, not type): overflow raises ARITHMETIC_OVERFLOW under
ANSI
- # where Python promotes to a big int; arithmetic is not
- # NULL-guarded (`x + 1` on NULL -> NULL vs Python TypeError).
- # TODO (SPARK-55210): map overflow / divide-by-zero precisely.
+ # Numeric arithmetic is never lowered: fixed-width Catalyst
Review Comment:
Or more simply add a catalyst expr which does the automatic promotion
instead of encoding that in weird branching. But I think (for v0) document and
throw is ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
