tyrobro opened a new pull request, #301:
URL: https://github.com/apache/datafusion-spark/pull/301
## Which issue does this PR close?
Closes #268.
## Rationale for this change
Spark only evaluates `expr2` when `expr1` is NaN. DataFusion core's `nanvl`
evaluates both arguments eagerly, so it can't be reused directly without
surfacing errors Spark would never raise (e.g. `1 / 0` under ANSI mode).
## What changes are included in this PR?
- Adds `SparkNanvl` (`crates/spark/src/function/conditional/nanvl.rs`):
- `coerce_types`: FLOAT/FLOAT stays FLOAT; everything else (mixed
FLOAT/DOUBLE,
int, DECIMAL, STRING) coerces to DOUBLE.
- `simplify()` rewrites the call to `CASE WHEN expr1 != expr1 THEN expr2
ELSE
expr1 END`, so DataFusion's lazy `CASE` evaluation reproduces Spark's
short-circuiting of `expr2` and its NULL-propagation rule.
- `invoke_with_args()` keeps an eager fallback for physical-execution paths
that skip the logical optimizer (e.g. Comet).
- Registers `nanvl` in `conditional/mod.rs`.
- Adds `crates/sqllogictest/test_files/conditional/nanvl.slt` covering NULL
propagation, NaN bit patterns, ±Infinity/-0.0, FLOAT/DOUBLE/int/DECIMAL
coercion, scalar+array, and lazy evaluation under ANSI mode.
## Are these changes tested?
Yes, via the `.slt` file above.
## Are there any user-facing changes?
Yes — `nanvl` is now available via
`SessionStateBuilderSpark::with_spark_features()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]