liujiayi771 commented on issue #8183:
URL:
https://github.com/apache/incubator-gluten/issues/8183#issuecomment-2556516460
@taiyang-li @PHILO-HE I think it is not related to `PullOutPreProject` rule.
The original Spark plan has the duplicate `Alias(if(id % 2 = 0, id+3, id+4))`
in both `HashAggregateExec` and the `ProjectExec` before `HashAggregateExec`.
The original Spark plan not eliminate this expression in `HashAggregateExec`,
Spark should use n3 directly in `HashAggregateExec`.
After `PullOutPreProject` rule, the expression of if has been pulled out to
the preProject, and then in `CollapseProjectExecTransformer` rule, it has been
merged with the previous Project, so we can see 2 same `If` in a
`ProjectExecTransformer`.
To address this issue, we might consider adding code to deduplicate projects
in `CollapseProjectExecTransformer`, or alternatively, replace the `Alias(If)`
in `HashAggregateExec` with n3 in the initial Spark plan.
```
AdaptiveSparkPlan isFinalPlan=false
+- HashAggregate(keys=[], functions=[max(n1#0L), max(n2#1L), sum(if
((((n1#0L + n2#1L) + (n3#2L % 2)) = 0)) 1 else 0)], output=[max(n1)#11L,
max(n2)#12L, sum((IF((((n1 + n2) + (n3 % 2)) = 0), 1, 0)))#14L])
+- HashAggregate(keys=[], functions=[partial_max(n1#0L),
partial_max(n2#1L), partial_sum(if ((((n1#0L + n2#1L) + (n3#2L % 2)) = 0)) 1
else 0)], output=[max#18L, max#19L, sum#20L])
+- Project [(id#8L + 1) AS n1#0L, (id#8L + 2) AS n2#1L, if (((id#8L %
2) = 0)) (id#8L + 3) else (id#8L + 4) AS n3#2L]
+- Range (0, 10, step=1, splits=1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
