Holden Karau created SPARK-59089:
------------------------------------
Summary: Recover FloatType columns for the transpiler by widening
operands to double
Key: SPARK-59089
URL: https://issues.apache.org/jira/browse/SPARK-59089
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 4.3.0
Reporter: Holden Karau
SPARK-55210 made FloatType columns fall back to interpreted Python entirely,
because an
expression that stays in FloatType rounds to 24 bits per step where Python
computes in double.
That is more conservative than exactness requires: casting each float operand
to DoubleType
reproduces CPython exactly, so those columns could keep their lowering.
## Why
FloatType is common in ML feature tables, and the current rule costs every such
column its
lowering permanently.
Measured:
- float32 -> Python float -> float32 round-trips **exactly** on 199163/199163
finite bit
patterns. So the value the interpreted UDF receives *is* the double a widened
expression would
compute with.
- Staying in FloatType diverged from CPython on **20000/20000** random pairs
for `(x + y) * y`.
- Casting each operand to DoubleType and computing in double matched CPython on
**20000/20000**
— for a DoubleType declared return type *and* for a FloatType one, because
`EvaluatePython.makeFromJava` narrows the interpreted double with a single
`c.toFloat`.
- Even the saturation case agrees: the double product 2.5373334837038975e44
under one `.toFloat`
is Infinity, the same as a trailing `cast(..., float)`.
The objection recorded in the current comment — "would hide the rounding for a
single operation
but not for a chain of them, and not the overflow at all" — is about declaring
a FloatType
*return* type while still computing in FloatType. It does not apply to
computing in double.
## How
Admit FloatType to the "fractional" category in
`ResolveTranspiledPythonUDFOptions`, and have the
fractional variant wrap each parameter reference in `.cast("double")` — a no-op
on a DoubleType
column that `SimplifyCasts` removes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
