ctagard commented on code in PR #59403:
URL: https://github.com/apache/spark/pull/59403#discussion_r4273831018
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -2796,6 +2797,49 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
case _ => false
}
+ // Rewrites conjuncts pushed below `aggregate` in terms of the aggregate's
child and ANDs them.
+ // Expressions above the aggregate see its float/double grouping keys
normalized (see
+ // NormalizeFloatingNumbers: -0.0 becomes 0.0 and every NaN becomes one
canonical NaN), so the
+ // pushed conjuncts read them normalized too, wherever the raw value could
change the result.
+ private[optimizer] def pushBelowAggregate(
+ conds: Seq[Expression],
+ aggregate: Aggregate): Expression = {
+ val aliasMap = getAliasMap(aggregate)
+ val normalizedGrouping = aggregate.groupingExpressions
Review Comment:
Done in 3d03a16, with the condition moved to
`NormalizeFloatingNumbers.outputsNormalizedKeys` in 3c36286. Plan test in
`FilterPushdownSuite`, end-to-end test in `PythonUDFSuite`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
