github-actions[bot] commented on code in PR #67884:
URL: https://github.com/apache/doris/pull/67884#discussion_r4070670721
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SimplifyAggGroupBy.java:
##########
@@ -19,118 +19,54 @@
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
-import org.apache.doris.nereids.trees.expressions.Add;
-import org.apache.doris.nereids.trees.expressions.BinaryArithmetic;
-import org.apache.doris.nereids.trees.expressions.Cast;
-import org.apache.doris.nereids.trees.expressions.Divide;
import org.apache.doris.nereids.trees.expressions.Expression;
-import org.apache.doris.nereids.trees.expressions.Multiply;
import org.apache.doris.nereids.trees.expressions.Slot;
-import org.apache.doris.nereids.trees.expressions.Subtract;
-import org.apache.doris.nereids.trees.expressions.literal.Literal;
-import org.apache.doris.nereids.util.ExpressionUtils;
-import org.apache.doris.nereids.util.Utils;
import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
+import java.util.LinkedHashSet;
import java.util.List;
import java.util.Set;
/**
- * Simplify Aggregate group by Multiple to One. For example
+ * Remove deterministic grouping expressions whose inputs are already bare
grouping slots.
* <p>
- * GROUP BY ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3
+ * GROUP BY ClientIP, ClientIP + 1, ClientIP + 2
* -->
* GROUP BY ClientIP
+ *
+ * <p>Retain existing bare slots so aggregate outputs can still reference
them. A cast group key
+ * cannot generally provide its original slot to those outputs. Never
synthesize a slot from
+ * derived keys, since doing so can split groups formed by non-injective
expressions.</p>
*/
public class SimplifyAggGroupBy extends OneRewriteRuleFactory {
- private static final ImmutableSet<Class<? extends Expression>>
supportedFunctions
- = ImmutableSet.of(Add.class, Subtract.class, Multiply.class,
Divide.class);
-
@Override
public Rule build() {
return logicalAggregate()
- .when(agg -> agg.getGroupByExpressions().size() > 1
- &&
ExpressionUtils.allMatch(agg.getGroupByExpressions(),
- SimplifyAggGroupBy::isBinaryArithmeticSlot))
+ .when(agg -> agg.getGroupByExpressions().size() > 1)
.then(agg -> {
- List<Expression> groupByExpressions =
agg.getGroupByExpressions();
- ImmutableSet.Builder<Expression> inputSlots
- =
ImmutableSet.builderWithExpectedSize(groupByExpressions.size());
- for (Expression groupByExpression : groupByExpressions) {
- inputSlots.addAll(groupByExpression.getInputSlots());
- }
- Set<Expression> slots = inputSlots.build();
- if (slots.size() != 1) {
+ List<Expression> simplified =
simplifyGroupBy(agg.getGroupByExpressions());
+ if (simplified == null) {
return null;
}
- return
agg.withGroupByAndOutput(Utils.fastToImmutableList(slots),
agg.getOutputExpressions());
+ return agg.withGroupByAndOutput(simplified,
agg.getOutputExpressions());
})
.toRule(RuleType.SIMPLIFY_AGG_GROUP_BY);
}
@VisibleForTesting
- protected static boolean isBinaryArithmeticSlot(Expression expr) {
- if (expr instanceof Slot) {
- return true;
- }
- if (!(expr instanceof BinaryArithmetic)) {
- return false;
- }
- if (!supportedFunctions.contains(expr.getClass())) {
- return false;
- }
-
- // Float/double arithmetic: precision loss for all operations
- if (expr.child(0).getDataType().isFloatLikeType()
- || expr.child(1).getDataType().isFloatLikeType()) {
- return false;
+ protected static List<Expression> simplifyGroupBy(List<Expression>
groupByExpressions) {
+ Set<Expression> distinctGroupBy = new
LinkedHashSet<>(groupByExpressions);
+ Set<Expression> determinants = distinctGroupBy.stream()
+
.filter(Slot.class::isInstance).collect(ImmutableSet.toImmutableSet());
+ // Keep at least one key: removing all constant keys changes the
result for empty input.
+ if (!determinants.isEmpty()) {
+ distinctGroupBy.removeIf(expression -> !(expression instanceof
Slot)
+ && !expression.containsVolatileExpression()
Review Comment:
[P1] Preserve keys with observable evaluation or validation
Consider the analyzed tree `Aggregate(count(*), groupBy=[x, assert_true(x >
0, 'bad')]) -> Scan(x)`. This predicate removes `assert_true` because it is
non-volatile and its input is the bare key `x`, producing `Aggregate(count(*),
groupBy=[x]) -> Scan(x)` and suppressing the required `InvalidArgument` for a
row with `x <= 0`. Before this change, normalization materialized the assertion
below the aggregate, where `LogicalProject.pruneOutputs` deliberately preserves
`NoneMovableFunction` evaluation even if later FD elimination drops its key.
The same check also erases `score()` from `GROUP BY id, score()`: `score()`
is non-deterministic but non-volatile and has no input slots, so the later
`CheckScoreUsage` validator can no longer reject the invalid aggregate use.
Please fence both non-movable and non-deterministic expressions (using the
existing combined non-movable helper where applicable) and add negative
coverage for both cases.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SimplifyAggGroupBy.java:
##########
@@ -19,118 +19,54 @@
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
-import org.apache.doris.nereids.trees.expressions.Add;
-import org.apache.doris.nereids.trees.expressions.BinaryArithmetic;
-import org.apache.doris.nereids.trees.expressions.Cast;
-import org.apache.doris.nereids.trees.expressions.Divide;
import org.apache.doris.nereids.trees.expressions.Expression;
-import org.apache.doris.nereids.trees.expressions.Multiply;
import org.apache.doris.nereids.trees.expressions.Slot;
-import org.apache.doris.nereids.trees.expressions.Subtract;
-import org.apache.doris.nereids.trees.expressions.literal.Literal;
-import org.apache.doris.nereids.util.ExpressionUtils;
-import org.apache.doris.nereids.util.Utils;
import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
+import java.util.LinkedHashSet;
import java.util.List;
import java.util.Set;
/**
- * Simplify Aggregate group by Multiple to One. For example
+ * Remove deterministic grouping expressions whose inputs are already bare
grouping slots.
* <p>
- * GROUP BY ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3
+ * GROUP BY ClientIP, ClientIP + 1, ClientIP + 2
* -->
* GROUP BY ClientIP
+ *
+ * <p>Retain existing bare slots so aggregate outputs can still reference
them. A cast group key
+ * cannot generally provide its original slot to those outputs. Never
synthesize a slot from
+ * derived keys, since doing so can split groups formed by non-injective
expressions.</p>
*/
public class SimplifyAggGroupBy extends OneRewriteRuleFactory {
- private static final ImmutableSet<Class<? extends Expression>>
supportedFunctions
- = ImmutableSet.of(Add.class, Subtract.class, Multiply.class,
Divide.class);
-
@Override
public Rule build() {
return logicalAggregate()
- .when(agg -> agg.getGroupByExpressions().size() > 1
- &&
ExpressionUtils.allMatch(agg.getGroupByExpressions(),
- SimplifyAggGroupBy::isBinaryArithmeticSlot))
+ .when(agg -> agg.getGroupByExpressions().size() > 1)
.then(agg -> {
- List<Expression> groupByExpressions =
agg.getGroupByExpressions();
- ImmutableSet.Builder<Expression> inputSlots
- =
ImmutableSet.builderWithExpectedSize(groupByExpressions.size());
- for (Expression groupByExpression : groupByExpressions) {
- inputSlots.addAll(groupByExpression.getInputSlots());
- }
- Set<Expression> slots = inputSlots.build();
- if (slots.size() != 1) {
+ List<Expression> simplified =
simplifyGroupBy(agg.getGroupByExpressions());
+ if (simplified == null) {
return null;
}
- return
agg.withGroupByAndOutput(Utils.fastToImmutableList(slots),
agg.getOutputExpressions());
+ return agg.withGroupByAndOutput(simplified,
agg.getOutputExpressions());
})
.toRule(RuleType.SIMPLIFY_AGG_GROUP_BY);
}
@VisibleForTesting
- protected static boolean isBinaryArithmeticSlot(Expression expr) {
- if (expr instanceof Slot) {
- return true;
- }
- if (!(expr instanceof BinaryArithmetic)) {
- return false;
- }
- if (!supportedFunctions.contains(expr.getClass())) {
- return false;
- }
-
- // Float/double arithmetic: precision loss for all operations
- if (expr.child(0).getDataType().isFloatLikeType()
- || expr.child(1).getDataType().isFloatLikeType()) {
- return false;
+ protected static List<Expression> simplifyGroupBy(List<Expression>
groupByExpressions) {
+ Set<Expression> distinctGroupBy = new
LinkedHashSet<>(groupByExpressions);
+ Set<Expression> determinants = distinctGroupBy.stream()
+
.filter(Slot.class::isInstance).collect(ImmutableSet.toImmutableSet());
+ // Keep at least one key: removing all constant keys changes the
result for empty input.
+ if (!determinants.isEmpty()) {
+ distinctGroupBy.removeIf(expression -> !(expression instanceof
Slot)
+ && !expression.containsVolatileExpression()
+ && determinants.containsAll(expression.getInputSlots()));
Review Comment:
[P1] Require dependency under grouping equality
With DOUBLE rows `x = +0.0` and `x = -0.0` (and `ELIMINATE_GROUP_BY_KEY`
disabled), `GROUP BY x, signbit(x)` must retain two composite keys:
`NormalizeAggregate` computes `signbit(x)` in the lower project while the signs
are still observable, then the aggregation sink canonicalizes the `x` key,
yielding `(+0,false)` and `(+0,true)`. This rule removes `signbit(x)` solely
because its input slot is `x`; the sink then canonicalizes both remaining `x`
values to `+0`, changing two count-1 groups into one count-2 group. The
previous allowlist did not match `signbit`.
Slot containment proves dependence on the input representation, not
constancy over Doris's grouping-equality classes. Please restrict removal to
expression/type families proven congruent with grouping equality (or retain a
conservative allowlist) and cover signed zero in a regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
