zahed1994 opened a new pull request, #59460:
URL: https://github.com/apache/spark/pull/59460
### What changes were proposed in this pull request?
In `RewriteDistinctAggregates`, aggregate functions with `SortOrder` in
their children (specifically `ListAgg` with `WITHIN GROUP (ORDER BY ...)`)
caused a `ClassCastException` during distinct rewrite when combined with
`FILTER` or when regular `ListAgg` is evaluated alongside other distinct
aggregates:
```
java.lang.ClassCastException: class
org.apache.spark.sql.catalyst.expressions.AttributeReference cannot be cast to
class org.apache.spark.sql.catalyst.expressions.SortOrder
at
org.apache.spark.sql.catalyst.expressions.aggregate.ListAgg.withNewChildrenInternal(collect.scala:1051)
at
org.apache.spark.sql.catalyst.optimizer.RewriteDistinctAggregates.patchAggregateFunctionChildren(RewriteDistinctAggregates.scala:262)
```
This occurred because:
1. `SortOrder` extends `Expression with Unevaluable`, but Catalyst treats
`orderExpressions` as part of `ListAgg.children`.
2. `RewriteDistinctAggregates` treated `SortOrder` directly as an unfoldable
child expression, causing `Expand` to project `SortOrder` (an `Unevaluable`
expression).
3. `patchAggregateFunctionChildren` mapped `af.children` with
`attrs(c).getOrElse(c)`, replacing the `SortOrder` itself with an
`AttributeReference`. When `ListAgg.withNewChildrenInternal` cast
`newChildren.drop(2)` back to `SortOrder`, it threw `ClassCastException`.
This PR fixes the issue by:
1. Introducing an `evalChildren` helper that unpacks `SortOrder.children` to
operate on evaluable expressions when determining foldability, grouping sets,
and projecting attributes.
2. In `patchAggregateFunctionChildren`, unwrapping `SortOrder`, replacing
only its child expressions with rewritten attributes, and preserving the
`SortOrder` wrapper.
### Why are the changes needed?
Queries that use `LISTAGG(DISTINCT ...) WITHIN GROUP (ORDER BY ...) FILTER
(...)` or regular `LISTAGG(...) WITHIN GROUP (ORDER BY ...)` alongside distinct
aggregates fail during query planning with an internal `ClassCastException`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Added unit tests in `RewriteDistinctAggregatesSuite`.
- Added end-to-end execution tests in `DataFrameAggregateSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Yes, Generated-by: Google Antigravity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
