david-mollitor-db opened a new pull request, #58910:
URL: https://github.com/apache/spark/pull/58910
### What changes were proposed in this pull request?
`Divide`, `Remainder`, and `IntegralDivide` (which share the `DivModLike`
trait) and `Pmod` declared `override def nullable: Boolean = true`
unconditionally. This PR changes `nullable` to reflect what the operators can
actually produce at runtime:
```scala
override def nullable: Boolean = left.nullable || right.nullable ||
!failOnError
```
Under ANSI mode (`failOnError`), divide-by-zero and integral overflow throw
rather than returning `null`, so the result is `null` only when one of the
inputs is `null`. Under LEGACY/TRY mode the behavior is unchanged (`nullable`
stays effectively `true`), because divide-by-zero returns `null` there and, for
decimals, a precision overflow can also return `null` independent of the
divisor.
The change is limited to the declared `nullable` property; `eval` and code
generation are untouched.
### Why are the changes needed?
Whole-stage and expression code generation represent SQL `NULL` with boolean
`isNull` flags. A `nullable = true` child forces the framework to materialize
an `isNull` variable and makes every parent expression emit a null-guard branch
on it (`CodegenContext.nullSafeExec`). Because nullability propagates, a single
spuriously-nullable `%` / `/` / `pmod` cascades dead `if (!isNull)` branches
into the comparisons, `CASE WHEN`s, and predicates built on top of it.
For example, `(id % 2) = 0` inside a `CASE WHEN` (ANSI) generated:
```java
boolean project_isNull_5 = false; // id % 2 -- never set to true
if (!project_isNull_5) { ... } // dead guard
...
if (!project_isNull_4 && project_value_4) // dead: !isNull_4 is always true
here
```
Correcting the nullability removes these dead guards at the source and feeds
more accurate nullability to the optimizer.
### Does this PR introduce _any_ user-facing change?
Yes (schema metadata only). Under ANSI mode, an output column that is a `/`,
`%`, `div`, or `pmod` over non-nullable inputs is now reported as non-nullable,
whereas before it was reported as nullable. Computed values and error behavior
are unchanged: ANSI divide/remainder/pmod-by-zero and integral overflow still
throw, and LEGACY/TRY still return `null`.
- Before: `spark.range(1).selectExpr("id % 2").schema.head.nullable` is
`true`
- After (ANSI): the same is `false`
### How was this patch tested?
Existing suites, all pass with no golden-file changes:
- `ArithmeticExpressionSuite`, `PredicateSuite`, `ColumnExpressionSuite`
- `SQLQueryTestSuite`: `operators.sql`, `try_arithmetic.sql`,
`decimalArithmeticOperations.sql`
`.sql.out` golden files are unaffected because their schema lines use
`df.schema.catalogString`, which does not encode nullability. Additionally
verified by inspecting generated code that the dead `isNull` guards for `%` /
`/` / `pmod` over non-null inputs are removed under ANSI.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
