vboo123 opened a new pull request, #59798:
URL: https://github.com/apache/spark/pull/59798
### What changes were proposed in this pull request?
Whole-stage codegen of an aggregate eliminates the common subexpressions of
the update expressions of all the aggregate functions together, and evaluates
them before the `do { ... } while (false)` block that checks each function's
`FILTER` clause. So a common subexpression of a function with a `FILTER` is
evaluated on a row that the `FILTER` rejects.
This PR adds `AggregateCodegenSupport.generateUpdateExprsCode`, used by the
three paths that generate the update code (`generateAggBufferUpdateCode` for
the no-keys and sort-based paths, and the regular and vectorized fast hash map
paths of `HashAggregateExec`):
- The functions without a `FILTER` share their common subexpressions,
evaluated before all the functions as before.
- Each function with a `FILTER` gets its own common subexpressions, and
their code goes at the start of the function's code, inside the `do { ... }
while (false)` block after the `FILTER` check.
Without a `FILTER`, the generated code is unchanged.
### Why are the changes needed?
With whole-stage codegen off, `AggregationIterator` evaluates `If(filter,
updateExpr, buffer)`, so the update expressions are not evaluated on a row that
the `FILTER` rejects. With codegen on, they are, so with ANSI on:
```sql
CREATE TABLE t USING parquet AS SELECT * FROM VALUES (NULL, 1, 2147483647),
(0, 1, 0) AS v(a, b, y);
SELECT avg(y + 1) FILTER (WHERE b < 0) FROM t;
```
raises `ARITHMETIC_OVERFLOW` with `spark.sql.codegen.wholeStage=true` and
returns NULL with it off. `avg` uses `y + 1` in two update expressions, so it
is a common subexpression.
### Does this PR introduce _any_ user-facing change?
Yes. With whole-stage codegen on, a query like the one above no longer
raises an error on a row that the `FILTER` rejects, as with codegen off.
### How was this patch tested?
Added a test in `WholeStageCodegenSuite` that runs `avg` and `sum` with a
`FILTER` next to `count(1)` without one, with and without grouping keys,
through the regular hash map, the row-based and vectorized fast hash maps and
the sort aggregate, with the aggregate code split into functions and not, and
with whole-stage codegen on and off.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
