(doris) branch master updated (45a0d2f9bcb -> 3a99c83fb66)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 45a0d2f9bcb [improve](jdbc catalog) Remove all property checks during create (#35194) add 3a99c83fb66 [minor](nereids): remove useless code (#35325) No new revisions were added by this update. Summary of changes: .../nereids/rules/rewrite/AdjustConjunctsReturnType.java | 5 - .../apache/doris/nereids/rules/rewrite/OrExpansion.java| 14 -- 2 files changed, 19 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](nereids): fix PlanPostProcessor use visitor (#35244)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 0b440685d9f [fix](nereids): fix PlanPostProcessor use visitor (#35244)
0b440685d9f is described below
commit 0b440685d9f892bd0cc6ceb4aa38fe2b8d6cb9f1
Author: jakevin
AuthorDate: Thu May 23 11:55:16 2024 +0800
[fix](nereids): fix PlanPostProcessor use visitor (#35244)
(cherry picked from commit 46e004a358b9e13adb492d376f77e4317e558a6a)
---
.../doris/nereids/processor/post/AddOffsetIntoDistribute.java | 3 ++-
.../nereids/processor/post/PushDownFilterThroughProject.java | 10 ++
2 files changed, 4 insertions(+), 9 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/AddOffsetIntoDistribute.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/AddOffsetIntoDistribute.java
index deac3988698..dc817321298 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/AddOffsetIntoDistribute.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/AddOffsetIntoDistribute.java
@@ -31,11 +31,12 @@ import
org.apache.doris.nereids.trees.plans.physical.PhysicalLimit;
public class AddOffsetIntoDistribute extends PlanPostProcessor {
@Override
public Plan visitPhysicalLimit(PhysicalLimit limit,
CascadesContext context) {
+limit = (PhysicalLimit) super.visit(limit, context);
if (limit.getPhase().isLocal() || limit.getOffset() == 0) {
return limit;
}
return new PhysicalDistribute<>(DistributionSpecGather.INSTANCE,
-limit.withLimit(limit.getLimit() + limit.getOffset()));
+limit.withLimit(limit.getLimit() +
limit.getOffset())).copyStatsAndGroupIdFrom(limit);
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PushDownFilterThroughProject.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PushDownFilterThroughProject.java
index 99b7d8a9572..e0ee6bbd6fd 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PushDownFilterThroughProject.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PushDownFilterThroughProject.java
@@ -19,7 +19,6 @@ package org.apache.doris.nereids.processor.post;
import org.apache.doris.nereids.CascadesContext;
import org.apache.doris.nereids.trees.plans.Plan;
-import org.apache.doris.nereids.trees.plans.physical.AbstractPhysicalPlan;
import org.apache.doris.nereids.trees.plans.physical.PhysicalFilter;
import org.apache.doris.nereids.trees.plans.physical.PhysicalProject;
import org.apache.doris.nereids.util.ExpressionUtils;
@@ -30,15 +29,10 @@ import org.apache.doris.nereids.util.ExpressionUtils;
public class PushDownFilterThroughProject extends PlanPostProcessor {
@Override
public Plan visitPhysicalFilter(PhysicalFilter filter,
CascadesContext context) {
+filter = (PhysicalFilter) super.visit(filter, context);
Plan child = filter.child();
if (!(child instanceof PhysicalProject)) {
-Plan newChild = child.accept(this, context);
-if (newChild == child) {
-return filter;
-} else {
-return ((AbstractPhysicalPlan)
filter.withChildren(child.accept(this, context)))
-.copyStatsAndGroupIdFrom(filter);
-}
+return filter;
}
PhysicalProject project = (PhysicalProject) child;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [fix](nereids): fix PlanPostProcessor use visitor (#35244) (#35265)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new e4c3533053f [fix](nereids): fix PlanPostProcessor use visitor (#35244)
(#35265)
e4c3533053f is described below
commit e4c3533053fe8461285f8f9743094f635178fdcd
Author: jakevin
AuthorDate: Thu May 23 14:08:24 2024 +0800
[fix](nereids): fix PlanPostProcessor use visitor (#35244) (#35265)
(cherry picked from commit 46e004a358b9e13adb492d376f77e4317e558a6a)
---
.../doris/nereids/processor/post/AddOffsetIntoDistribute.java | 3 ++-
.../nereids/processor/post/PushdownFilterThroughProject.java | 10 ++
2 files changed, 4 insertions(+), 9 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/AddOffsetIntoDistribute.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/AddOffsetIntoDistribute.java
index deac3988698..dc817321298 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/AddOffsetIntoDistribute.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/AddOffsetIntoDistribute.java
@@ -31,11 +31,12 @@ import
org.apache.doris.nereids.trees.plans.physical.PhysicalLimit;
public class AddOffsetIntoDistribute extends PlanPostProcessor {
@Override
public Plan visitPhysicalLimit(PhysicalLimit limit,
CascadesContext context) {
+limit = (PhysicalLimit) super.visit(limit, context);
if (limit.getPhase().isLocal() || limit.getOffset() == 0) {
return limit;
}
return new PhysicalDistribute<>(DistributionSpecGather.INSTANCE,
-limit.withLimit(limit.getLimit() + limit.getOffset()));
+limit.withLimit(limit.getLimit() +
limit.getOffset())).copyStatsAndGroupIdFrom(limit);
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PushdownFilterThroughProject.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PushdownFilterThroughProject.java
index 31ea757a036..87ca77c42c6 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PushdownFilterThroughProject.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PushdownFilterThroughProject.java
@@ -19,7 +19,6 @@ package org.apache.doris.nereids.processor.post;
import org.apache.doris.nereids.CascadesContext;
import org.apache.doris.nereids.trees.plans.Plan;
-import org.apache.doris.nereids.trees.plans.physical.AbstractPhysicalPlan;
import org.apache.doris.nereids.trees.plans.physical.PhysicalFilter;
import org.apache.doris.nereids.trees.plans.physical.PhysicalProject;
import org.apache.doris.nereids.util.ExpressionUtils;
@@ -30,15 +29,10 @@ import org.apache.doris.nereids.util.ExpressionUtils;
public class PushdownFilterThroughProject extends PlanPostProcessor {
@Override
public Plan visitPhysicalFilter(PhysicalFilter filter,
CascadesContext context) {
+filter = (PhysicalFilter) super.visit(filter, context);
Plan child = filter.child();
if (!(child instanceof PhysicalProject)) {
-Plan newChild = child.accept(this, context);
-if (newChild == child) {
-return filter;
-} else {
-return ((AbstractPhysicalPlan)
filter.withChildren(child.accept(this, context)))
-.copyStatsAndGroupIdFrom(filter);
-}
+return filter;
}
PhysicalProject project = (PhysicalProject) child;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (5bbc149285d -> 961bd36043f)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 5bbc149285d [feat](nereids) support Iceberg time travel syntax (#34681) add 961bd36043f [fix](Nereids): use project input slots as require slots when eliminating group by key (#35247) No new revisions were added by this update. Summary of changes: .../nereids/rules/rewrite/EliminateGroupByKey.java | 2 +- .../rules/rewrite/EliminateGroupByKeyTest.java | 22 ++ 2 files changed, 23 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](nereids): fix PlanPostProcessor use visitor (#35244)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 46e004a358b [fix](nereids): fix PlanPostProcessor use visitor (#35244)
46e004a358b is described below
commit 46e004a358b9e13adb492d376f77e4317e558a6a
Author: jakevin
AuthorDate: Thu May 23 11:55:16 2024 +0800
[fix](nereids): fix PlanPostProcessor use visitor (#35244)
---
.../doris/nereids/processor/post/AddOffsetIntoDistribute.java | 3 ++-
.../nereids/processor/post/PushDownFilterThroughProject.java | 10 ++
2 files changed, 4 insertions(+), 9 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/AddOffsetIntoDistribute.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/AddOffsetIntoDistribute.java
index deac3988698..dc817321298 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/AddOffsetIntoDistribute.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/AddOffsetIntoDistribute.java
@@ -31,11 +31,12 @@ import
org.apache.doris.nereids.trees.plans.physical.PhysicalLimit;
public class AddOffsetIntoDistribute extends PlanPostProcessor {
@Override
public Plan visitPhysicalLimit(PhysicalLimit limit,
CascadesContext context) {
+limit = (PhysicalLimit) super.visit(limit, context);
if (limit.getPhase().isLocal() || limit.getOffset() == 0) {
return limit;
}
return new PhysicalDistribute<>(DistributionSpecGather.INSTANCE,
-limit.withLimit(limit.getLimit() + limit.getOffset()));
+limit.withLimit(limit.getLimit() +
limit.getOffset())).copyStatsAndGroupIdFrom(limit);
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PushDownFilterThroughProject.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PushDownFilterThroughProject.java
index 99b7d8a9572..e0ee6bbd6fd 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PushDownFilterThroughProject.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PushDownFilterThroughProject.java
@@ -19,7 +19,6 @@ package org.apache.doris.nereids.processor.post;
import org.apache.doris.nereids.CascadesContext;
import org.apache.doris.nereids.trees.plans.Plan;
-import org.apache.doris.nereids.trees.plans.physical.AbstractPhysicalPlan;
import org.apache.doris.nereids.trees.plans.physical.PhysicalFilter;
import org.apache.doris.nereids.trees.plans.physical.PhysicalProject;
import org.apache.doris.nereids.util.ExpressionUtils;
@@ -30,15 +29,10 @@ import org.apache.doris.nereids.util.ExpressionUtils;
public class PushDownFilterThroughProject extends PlanPostProcessor {
@Override
public Plan visitPhysicalFilter(PhysicalFilter filter,
CascadesContext context) {
+filter = (PhysicalFilter) super.visit(filter, context);
Plan child = filter.child();
if (!(child instanceof PhysicalProject)) {
-Plan newChild = child.accept(this, context);
-if (newChild == child) {
-return filter;
-} else {
-return ((AbstractPhysicalPlan)
filter.withChildren(child.accept(this, context)))
-.copyStatsAndGroupIdFrom(filter);
-}
+return filter;
}
PhysicalProject project = (PhysicalProject) child;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [feat](Nereids): Refactor Eliminate_Group_By_Key by functional dependencies (#34948)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 01fa2e6a555 [feat](Nereids): Refactor Eliminate_Group_By_Key by
functional dependencies (#34948)
01fa2e6a555 is described below
commit 01fa2e6a55511f1173f958293b091ba7e97bf74e
Author: 谢健
AuthorDate: Wed May 22 10:26:46 2024 +0800
[feat](Nereids): Refactor Eliminate_Group_By_Key by functional dependencies
(#34948)
---
.../apache/doris/nereids/properties/FuncDeps.java | 34 +++
.../nereids/properties/FunctionalDependencies.java | 29 ++-
.../org/apache/doris/nereids/rules/RuleType.java | 1 +
.../nereids/rules/rewrite/EliminateGroupByKey.java | 236 ++-
.../plans/logical/LogicalCatalogRelation.java | 9 -
.../nereids/trees/plans/logical/LogicalPlan.java | 13 +-
.../trees/plans/logical/LogicalProject.java| 5 +
.../properties/{FuncDepsTest.java => FdTest.java} | 6 +-
.../doris/nereids/properties/FuncDepsTest.java | 260 ++---
.../properties/FunctionalDependenciesTest.java | 6 +-
.../rules/rewrite/EliminateGroupByKeyTest.java | 99
.../eliminate_gby_key/eliminate_gby_key.groovy | 22 +-
12 files changed, 329 insertions(+), 391 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/FuncDeps.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/FuncDeps.java
index 8c84d7a1755..9ecfbea6504 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/FuncDeps.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/FuncDeps.java
@@ -20,8 +20,11 @@ package org.apache.doris.nereids.properties;
import org.apache.doris.nereids.trees.expressions.Slot;
import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Sets;
+import java.util.ArrayList;
import java.util.HashSet;
+import java.util.List;
import java.util.Objects;
import java.util.Set;
@@ -71,6 +74,37 @@ public class FuncDeps {
return items.size();
}
+public boolean isEmpty() {
+return items.isEmpty();
+}
+
+/**
+ * Eliminate all deps in slots
+ */
+public Set> eliminateDeps(Set> slots) {
+Set> minSlotSet = slots;
+List>> reduceSlotSets = new ArrayList<>();
+reduceSlotSets.add(slots);
+while (!reduceSlotSets.isEmpty()) {
+List>> newReduceSlotSets = new ArrayList<>();
+for (Set> slotSet : reduceSlotSets) {
+for (FuncDepsItem funcDepsItem : items) {
+if (slotSet.contains(funcDepsItem.dependencies)
+&& slotSet.contains(funcDepsItem.determinants)) {
+Set> newSet = Sets.newHashSet(slotSet);
+newSet.remove(funcDepsItem.dependencies);
+if (minSlotSet.size() > newSet.size()) {
+minSlotSet = newSet;
+}
+newReduceSlotSets.add(newSet);
+}
+}
+}
+reduceSlotSets = newReduceSlotSets;
+}
+return minSlotSet;
+}
+
public boolean isFuncDeps(Set dominate, Set dependency) {
return items.contains(new FuncDepsItem(dominate, dependency));
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/FunctionalDependencies.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/FunctionalDependencies.java
index 079657df9b6..cee2be1c918 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/FunctionalDependencies.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/FunctionalDependencies.java
@@ -18,6 +18,7 @@
package org.apache.doris.nereids.properties;
import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.functions.ExpressionTrait;
import org.apache.doris.nereids.util.ImmutableEqualSet;
import com.google.common.collect.ImmutableSet;
@@ -196,6 +197,9 @@ public class FunctionalDependencies {
}
public void addDeps(Set dominate, Set dependency) {
+if (dominate.containsAll(dependency)) {
+return;
+}
fdDgBuilder.addDeps(dominate, dependency);
}
@@ -265,10 +269,17 @@ public class FunctionalDependencies {
/**
* get all unique slots
*/
-public List> getAllUnique() {
-List> res = new ArrayList<>(uniqueSet.slotSets);
-for (Slot s : uniqueSet.slots) {
-res.add(ImmutableSet.of(s));
+public List> getAllUniqueAndNotNull() {
+List> res = new ArrayList<
(doris) branch master updated: [fix](Nereids): fix some bugs in or expansion (#34837)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1e4a69c2a43 [fix](Nereids): fix some bugs in or expansion (#34837)
1e4a69c2a43 is described below
commit 1e4a69c2a43ae0078092bd709008a313cad280a4
Author: 谢健
AuthorDate: Wed May 15 15:27:05 2024 +0800
[fix](Nereids): fix some bugs in or expansion (#34837)
add unit test
---
.../doris/nereids/rules/rewrite/OrExpansion.java | 25 ++
.../trees/copier/LogicalPlanDeepCopier.java| 8 ++-
.../nereids/rules/rewrite/OrExpansionTest.java | 4
3 files changed, 36 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/OrExpansion.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/OrExpansion.java
index 9f9257f5f60..61a90e3aea4 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/OrExpansion.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/OrExpansion.java
@@ -104,6 +104,31 @@ public class OrExpansion extends
DefaultPlanRewriter implem
return hasNewChildren ? plan.withChildren(newChildren) : plan;
}
+@Override
+public Plan visitLogicalCTEAnchor(
+LogicalCTEAnchor anchor,
OrExpandsionContext ctx) {
+Plan child1 = anchor.child(0).accept(this, ctx);
+// Consumer's CTE must be child of the cteAnchor in this case:
+// anchor
+// +-producer1
+// +-agg(consumer1) join agg(consumer1)
+// >
+// anchor
+// +-producer1
+// +-anchor
+// +--producer2(agg2(consumer1))
+// +--producer3(agg3(consumer1))
+// +-consumer2 join consumer3
+OrExpandsionContext consumerContext =
+new OrExpandsionContext(ctx.statementContext,
ctx.cascadesContext);
+Plan child2 = anchor.child(1).accept(this, consumerContext);
+for (int i = consumerContext.cteProducerList.size() - 1; i >= 0; i--) {
+LogicalCTEProducer producer =
consumerContext.cteProducerList.get(i);
+child2 = new LogicalCTEAnchor<>(producer.getCteId(), producer,
child2);
+}
+return anchor.withChildren(ImmutableList.of(child1, child2));
+}
+
@Override
public Plan visitLogicalJoin(LogicalJoin
join, OrExpandsionContext ctx) {
join = (LogicalJoin) this.visit(join,
ctx);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/copier/LogicalPlanDeepCopier.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/copier/LogicalPlanDeepCopier.java
index dfa881aa2e6..197de0089a3 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/copier/LogicalPlanDeepCopier.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/copier/LogicalPlanDeepCopier.java
@@ -340,8 +340,14 @@ public class LogicalPlanDeepCopier extends
DefaultPlanRewriter markJoinConjuncts =
join.getMarkJoinConjuncts().stream()
.map(c -> ExpressionDeepCopier.INSTANCE.deepCopy(c, context))
.collect(ImmutableList.toImmutableList());
+Optional markJoinSlotReference =
Optional.empty();
+if (join.getMarkJoinSlotReference().isPresent()) {
+markJoinSlotReference = Optional.of((MarkJoinSlotReference)
ExpressionDeepCopier.INSTANCE
+.deepCopy(join.getMarkJoinSlotReference().get(), context));
+
+}
return new LogicalJoin<>(join.getJoinType(), hashJoinConjuncts,
otherJoinConjuncts, markJoinConjuncts,
-join.getDistributeHint(), join.getMarkJoinSlotReference(),
children, join.getJoinReorderContext());
+join.getDistributeHint(), markJoinSlotReference, children,
join.getJoinReorderContext());
}
@Override
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/OrExpansionTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/OrExpansionTest.java
index 9f8bd8bcc55..0f2d9418bff 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/OrExpansionTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/OrExpansionTest.java
@@ -19,6 +19,7 @@ package org.apache.doris.nereids.rules.rewrite;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.logical.LogicalCTEAnchor;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCTEConsumer;
import org.apache.doris.nereids.util.MemoPatternMatchSupported;
import org.apache.doris.nereids.util.PlanChecker;
import org.apache.doris.utframe.TestWithFeService;
@@ -81,6 +82,9 @@ class OrExpansionTest extends TestWithFeService implements
MemoPatternMatchSuppo
Assertions.
(doris) branch master updated: [feat](Nereids): Reject Commutativity Swap for Nested Loop Joins Affecting Parallelism (#34639)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 141c3330b4e [feat](Nereids): Reject Commutativity Swap for Nested Loop
Joins Affecting Parallelism (#34639)
141c3330b4e is described below
commit 141c3330b4ebf5aaff874e72c711a6b2f6efddf5
Author: 谢健
AuthorDate: Mon May 13 13:55:56 2024 +0800
[feat](Nereids): Reject Commutativity Swap for Nested Loop Joins Affecting
Parallelism (#34639)
---
.../rules/exploration/join/JoinCommute.java| 7 +++
.../apache/doris/planner/NestedLoopJoinNode.java | 6 +-
.../rules/exploration/join/JoinCommuteTest.java| 24 +-
3 files changed, 35 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinCommute.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinCommute.java
index 49c91b92942..73fb853082c 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinCommute.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinCommute.java
@@ -25,9 +25,11 @@ import org.apache.doris.nereids.trees.expressions.Not;
import org.apache.doris.nereids.trees.expressions.Slot;
import
org.apache.doris.nereids.trees.expressions.functions.scalar.BitmapContains;
import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.JoinType;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.planner.NestedLoopJoinNode;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.thrift.TRuntimeFilterType;
@@ -66,6 +68,11 @@ public class JoinCommute extends OneExplorationRuleFactory {
// commuting nest loop mark join or left anti mark join is not
supported by be
.whenNot(join -> join.isMarkJoin() &&
(join.getHashJoinConjuncts().isEmpty()
|| join.getJoinType().isLeftAntiJoin()))
+// For a nested loop join, if commutativity causes a join that
could originally be executed
+// in parallel to become non-parallelizable, then we reject
this swap.
+.whenNot(join -> JoinUtils.shouldNestedLoopJoin(join)
+&&
NestedLoopJoinNode.canParallelize(JoinType.toJoinOperator(join.getJoinType()))
+&&
!NestedLoopJoinNode.canParallelize(JoinType.toJoinOperator(join.getJoinType().swap(
.then(join -> {
LogicalJoin newJoin =
join.withTypeChildren(join.getJoinType().swap(),
join.right(), join.left(), null);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/NestedLoopJoinNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/NestedLoopJoinNode.java
index 983cbfd5884..30c0a2d0394 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/NestedLoopJoinNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/NestedLoopJoinNode.java
@@ -73,12 +73,16 @@ public class NestedLoopJoinNode extends JoinNodeBase {
tupleIds.addAll(inner.getOutputTupleIds());
}
-public boolean canParallelize() {
+public static boolean canParallelize(JoinOperator joinOp) {
return joinOp == JoinOperator.CROSS_JOIN || joinOp ==
JoinOperator.INNER_JOIN
|| joinOp == JoinOperator.LEFT_OUTER_JOIN || joinOp ==
JoinOperator.LEFT_SEMI_JOIN
|| joinOp == JoinOperator.LEFT_ANTI_JOIN || joinOp ==
JoinOperator.NULL_AWARE_LEFT_ANTI_JOIN;
}
+public boolean canParallelize() {
+return canParallelize(joinOp);
+}
+
public void setJoinConjuncts(List joinConjuncts) {
this.joinConjuncts = joinConjuncts;
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/join/JoinCommuteTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/join/JoinCommuteTest.java
index 20323d108e6..18235b3ce4c 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/join/JoinCommuteTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/join/JoinCommuteTest.java
@@ -18,7 +18,9 @@
package org.apache.doris.nereids.rules.exploration.join;
import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.trees.expressions.GreaterThan;
import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
import org.apache.doris.nereids.trees.p
(doris) branch master updated (e8288f18f72 -> 68fefbc6648)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from e8288f18f72 [FIX](array_index)fix array tokenized index parser and array_contains bugs (#34683) add 68fefbc6648 [feature](Nereids): eliminate useless project (#34611) No new revisions were added by this update. Summary of changes: .../nereids/processor/post/ColumnPruningPostProcessor.java| 11 +-- .../org/apache/doris/nereids/trees/plans/algebra/Project.java | 10 ++ .../doris/nereids/trees/plans/logical/LogicalProject.java | 10 -- 3 files changed, 19 insertions(+), 12 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](Nereids): fix choose id when there is enforcer in group (#34573)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 9da023362d0 [fix](Nereids): fix choose id when there is enforcer in
group (#34573)
9da023362d0 is described below
commit 9da023362d08e2dd92dca2ecae23917a4d406468
Author: 谢健
AuthorDate: Fri May 10 10:18:29 2024 +0800
[fix](Nereids): fix choose id when there is enforcer in group (#34573)
---
fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java
index a9d3d881491..c9beb29c0dc 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java
@@ -73,7 +73,7 @@ public class Group {
private PhysicalProperties chosenProperties;
-private int chosenGroupExpressionId = -1;
+private List chosenGroupExpressionId = new ArrayList<>();
private StructInfoMap structInfoMap = new StructInfoMap();
@@ -215,13 +215,13 @@ public class Group {
public Optional>
getLowestCostPlan(PhysicalProperties physicalProperties) {
chosenProperties = physicalProperties;
if (physicalProperties == null || lowestCostPlans.isEmpty()) {
-chosenGroupExpressionId = -1;
+chosenGroupExpressionId.clear();
return Optional.empty();
}
Optional> costAndGroupExpression =
Optional.ofNullable(lowestCostPlans.get(physicalProperties));
if (costAndGroupExpression.isPresent()) {
-chosenGroupExpressionId =
costAndGroupExpression.get().second.getId().asInt();
+
chosenGroupExpressionId.add(costAndGroupExpression.get().second.getId().asInt());
}
return costAndGroupExpression;
}
@@ -463,7 +463,7 @@ public class Group {
for (GroupExpression enforcer : enforcers) {
str.append("").append(enforcer).append("\n");
}
-if (chosenGroupExpressionId != -1) {
+if (!chosenGroupExpressionId.isEmpty()) {
str.append(" chosen expression id:
").append(chosenGroupExpressionId).append("\n");
str.append(" chosen properties:
").append(chosenProperties).append("\n");
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (517247e73fe -> 7763d0cfc91)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 517247e73fe [feat](Nereids): Put the Child with Least Row Count in the
First Position of Intersect (#34290)
add 7763d0cfc91 [feat](Nereids): compute func deps in logical plan (#34233)
No new revisions were added by this update.
Summary of changes:
.../apache/doris/nereids/properties/FuncDeps.java | 31 +++-
.../doris/nereids/properties/FuncDepsDAG.java | 99 ---
.../doris/nereids/properties/FuncDepsDG.java | 198 +
.../nereids/properties/FunctionalDependencies.java | 91 --
.../trees/plans/BlockFuncDepsPropagation.java | 5 +
.../nereids/trees/plans/PropagateFuncDeps.java | 5 +
.../trees/plans/logical/LogicalAggregate.java | 5 +
.../trees/plans/logical/LogicalAssertNumRows.java | 5 +
.../plans/logical/LogicalCatalogRelation.java | 5 +
.../plans/logical/LogicalDeferMaterializeTopN.java | 5 +
.../nereids/trees/plans/logical/LogicalExcept.java | 14 ++
.../nereids/trees/plans/logical/LogicalFilter.java | 5 +
.../trees/plans/logical/LogicalGenerate.java | 5 +
.../nereids/trees/plans/logical/LogicalHaving.java | 5 +
.../trees/plans/logical/LogicalIntersect.java | 9 +
.../nereids/trees/plans/logical/LogicalJoin.java | 10 ++
.../nereids/trees/plans/logical/LogicalLimit.java | 5 +
.../trees/plans/logical/LogicalOneRowRelation.java | 5 +
.../nereids/trees/plans/logical/LogicalPlan.java | 20 +++
.../trees/plans/logical/LogicalProject.java| 5 +
.../nereids/trees/plans/logical/LogicalRepeat.java | 5 +
.../trees/plans/logical/LogicalSubQueryAlias.java | 5 +
.../nereids/trees/plans/logical/LogicalTopN.java | 5 +
.../nereids/trees/plans/logical/LogicalUnion.java | 84 +
.../nereids/trees/plans/logical/LogicalView.java | 5 +
.../nereids/trees/plans/logical/LogicalWindow.java | 5 +
.../doris/nereids/util/ImmutableEqualSet.java | 20 +++
.../{FuncDepsDAGTest.java => FuncDepsDGTest.java} | 35 ++--
.../{EqualSetTest.java => FuncDepsTest.java} | 125 ++---
.../properties/FunctionalDependenciesTest.java | 10 +-
.../doris/nereids/properties/UniformTest.java | 4 +-
.../doris/nereids/properties/UniqueTest.java | 32 ++--
32 files changed, 616 insertions(+), 251 deletions(-)
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/FuncDepsDAG.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/FuncDepsDG.java
rename
fe/fe-core/src/test/java/org/apache/doris/nereids/properties/{FuncDepsDAGTest.java
=> FuncDepsDGTest.java} (66%)
copy
fe/fe-core/src/test/java/org/apache/doris/nereids/properties/{EqualSetTest.java
=> FuncDepsTest.java} (58%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (a4fca4e0a04 -> 517247e73fe)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from a4fca4e0a04 [Bug](RegressionTest) fix regresstion test failed (#34466)
add 517247e73fe [feat](Nereids): Put the Child with Least Row Count in the
First Position of Intersect (#34290)
No new revisions were added by this update.
Summary of changes:
.../org/apache/doris/nereids/cost/CostModelV1.java | 11
.../org/apache/doris/nereids/rules/RuleSet.java| 2 +
.../org/apache/doris/nereids/rules/RuleType.java | 1 +
.../rules/exploration/IntersectReorder.java| 69 ++
...eProjectTest.java => IntersectReorderTest.java} | 49 ---
.../doris/nereids/util/LogicalPlanBuilder.java | 6 ++
.../bs_downgrade_shape/query8.out | 8 +--
.../shape/query38.out | 12 ++--
.../nereids_tpcds_shape_sf1000_p0/shape/query8.out | 8 +--
.../noStatsRfPrune/query14.out | 16 ++---
.../noStatsRfPrune/query38.out | 12 ++--
.../noStatsRfPrune/query8.out | 8 +--
.../no_stats_shape/query14.out | 16 ++---
.../no_stats_shape/query38.out | 12 ++--
.../no_stats_shape/query8.out | 8 +--
.../rf_prune/query38.out | 12 ++--
.../rf_prune/query8.out| 8 +--
.../nereids_tpcds_shape_sf100_p0/shape/query38.out | 12 ++--
.../nereids_tpcds_shape_sf100_p0/shape/query8.out | 8 +--
19 files changed, 187 insertions(+), 91 deletions(-)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/IntersectReorder.java
copy
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/{join/InnerJoinRightAssociateProjectTest.java
=> IntersectReorderTest.java} (55%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (a0606975722 -> db2760c13b1)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from a0606975722 [fix](memory) Fix page cache memory tracker consumption in prune (#34320) add db2760c13b1 [fix](Fe): remove code List.of in java11 grammar (#34407) No new revisions were added by this update. Summary of changes: fe/fe-core/src/main/java/org/apache/doris/backup/Repository.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (785ffb2aeb1 -> 073fec33b96)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 785ffb2aeb1 [Fix](nereids) fix cases unstable of hint (#34101)
add 073fec33b96 [feature](Nereids): add equal set in functional
dependencies (#33642)
No new revisions were added by this update.
Summary of changes:
.../nereids/properties/FunctionalDependencies.java | 42 +-
.../trees/plans/BlockFuncDepsPropagation.java | 5 +
.../nereids/trees/plans/PropagateFuncDeps.java | 5 +
.../trees/plans/logical/LogicalAggregate.java | 5 +
.../trees/plans/logical/LogicalAssertNumRows.java | 6 +
.../plans/logical/LogicalCatalogRelation.java | 5 +
.../plans/logical/LogicalDeferMaterializeTopN.java | 5 +
.../nereids/trees/plans/logical/LogicalExcept.java | 15 ++
.../nereids/trees/plans/logical/LogicalFilter.java | 10 ++
.../trees/plans/logical/LogicalGenerate.java | 6 +
.../nereids/trees/plans/logical/LogicalHaving.java | 10 ++
.../trees/plans/logical/LogicalIntersect.java | 9 ++
.../nereids/trees/plans/logical/LogicalJoin.java | 16 ++
.../nereids/trees/plans/logical/LogicalLimit.java | 5 +
.../trees/plans/logical/LogicalOneRowRelation.java | 16 ++
.../nereids/trees/plans/logical/LogicalPlan.java | 3 +
.../trees/plans/logical/LogicalProject.java| 21 +++
.../nereids/trees/plans/logical/LogicalRepeat.java | 5 +
.../trees/plans/logical/LogicalSqlCache.java | 21 +--
.../trees/plans/logical/LogicalSubQueryAlias.java | 11 ++
.../nereids/trees/plans/logical/LogicalTopN.java | 5 +
.../nereids/trees/plans/logical/LogicalUnion.java | 61
.../nereids/trees/plans/logical/LogicalView.java | 6 +
.../nereids/trees/plans/logical/LogicalWindow.java | 6 +
.../apache/doris/nereids/util/ExpressionUtils.java | 8 +
.../doris/nereids/util/ImmutableEqualSet.java | 69 +++--
.../{UniformTest.java => EqualSetTest.java}| 161 -
.../suites/nereids_syntax_p0/join_order.groovy | 8 +-
28 files changed, 437 insertions(+), 108 deletions(-)
copy
fe/fe-core/src/test/java/org/apache/doris/nereids/properties/{UniformTest.java
=> EqualSetTest.java} (50%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [feature](Nereids): add ColumnPruningPostProcessor. (#32800) (#34077)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 991b6a72741 [feature](Nereids): add ColumnPruningPostProcessor.
(#32800) (#34077)
991b6a72741 is described below
commit 991b6a727413d1e22d291f92cb4a1974bebbaa1a
Author: jakevin
AuthorDate: Thu Apr 25 11:26:14 2024 +0800
[feature](Nereids): add ColumnPruningPostProcessor. (#32800) (#34077)
(cherry picked from commit 5970f983db5dc95341c16f3459d883ce0f9676b3)
---
.../processor/post/ColumnPruningPostProcessor.java | 107 +
.../nereids/processor/post/PlanPostProcessors.java | 1 +
.../doris/nereids/processor/post/TopNScanOpt.java | 6 +-
.../trees/plans/physical/AbstractPhysicalPlan.java | 2 +-
.../ColumnPruningPostProcessorTest.java| 62
.../data/nereids_ssb_shape_sf100_p0/shape/q3.4.out | 13 +--
.../data/nereids_ssb_shape_sf100_p0/shape/q4.3.out | 25 ++---
7 files changed, 195 insertions(+), 21 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/ColumnPruningPostProcessor.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/ColumnPruningPostProcessor.java
new file mode 100644
index 000..40d25ddd748
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/ColumnPruningPostProcessor.java
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.processor.post;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.physical.AbstractPhysicalJoin;
+import org.apache.doris.nereids.trees.plans.physical.AbstractPhysicalPlan;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalDistribute;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalProject;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Prune column for Join-Cluster
+ */
+@DependsRules({
+MergeProjectPostProcessor.class
+})
+public class ColumnPruningPostProcessor extends PlanPostProcessor {
+@Override
+public PhysicalProject visitPhysicalProject(PhysicalProject project, CascadesContext ctx) {
+Plan child = project.child();
+Plan newChild = child.accept(this, ctx);
+if (newChild instanceof AbstractPhysicalJoin) {
+AbstractPhysicalJoin join =
(AbstractPhysicalJoin) newChild;
+Plan left = join.left();
+Plan right = join.right();
+Set leftOutput = left.getOutputSet();
+Set rightOutput = right.getOutputSet();
+
+Set usedSlots = project.getProjects().stream().flatMap(ne ->
ne.getInputSlots().stream())
+.collect(Collectors.toSet());
+
+Stream.concat(join.getHashJoinConjuncts().stream(),
join.getOtherJoinConjuncts().stream())
+.flatMap(expr -> expr.getInputSlots().stream())
+.forEach(usedSlots::add);
+join.getMarkJoinSlotReference().ifPresent(usedSlots::add);
+
+List leftNewProjections = new ArrayList<>();
+List rightNewProjections = new ArrayList<>();
+
+for (Slot usedSlot : usedSlots) {
+if (leftOutput.contains(usedSlot)) {
+leftNewProjections.add(usedSlot);
+} else if (rightOutput.contains(usedSlot)) {
+rightNewProjections.add(usedSlot);
+}
+}
+
+Plan newLeft;
+if (left instanceof PhysicalDistribute) {
+newLeft = leftNewProjections.size() != leftOutput.size() &&
!leftNewProjections.isEmpty()
+
(doris) branch master updated (df07e1a4079 -> bf2cb5c604d)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from df07e1a4079 [fix](routine-load) fix routine load lag is negative
(#33846)
add bf2cb5c604d [feature](Nereids): Add DAG to store function dependencies
(#33882)
No new revisions were added by this update.
Summary of changes:
...UnboundLogicalProperties.java => FuncDeps.java} | 45 ++
.../doris/nereids/properties/FuncDepsDAG.java | 99 ++
.../doris/nereids/properties/FuncDepsDAGTest.java | 75
3 files changed, 202 insertions(+), 17 deletions(-)
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/{UnboundLogicalProperties.java
=> FuncDeps.java} (52%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/FuncDepsDAG.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/properties/FuncDepsDAGTest.java
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (f9e9c36bcb4 -> a16d11ea610)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from f9e9c36bcb4 [refactor](filesystem)refactor `filesystem` interface (#33361) add a16d11ea610 [minor](Nereids): remove useless getFilterConjuncts() filter() in Translator (#33801) No new revisions were added by this update. Summary of changes: .../glue/translator/PhysicalPlanTranslator.java| 317 ++--- .../java/org/apache/doris/nereids/memo/Memo.java | 10 - .../nereids/rules/rewrite/EliminateFilterTest.java | 42 ++- 3 files changed, 191 insertions(+), 178 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (975bf73ec21 -> 3dd0033d4b2)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 975bf73ec21 [fix](cloud) Disable enable_feature_binlog in the cloud mode (#33719) add 3dd0033d4b2 [fix](Nereids): add whole tree rewriter when root is not CTEAnchor (#33591) No new revisions were added by this update. Summary of changes: .../nereids/jobs/rewrite/CostBasedRewriteJob.java | 18 +-- .../runtime_filter/test_pushdown_setop.out | 35 -- 2 files changed, 19 insertions(+), 34 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (de6334e3a4f -> 5cd490121cb)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from de6334e3a4f [chore](Nereids)disable fallback_to_original_planner by default (#32882) add 5cd490121cb [minor](Nereids): remove useless stream filter() in Translator (#33758) No new revisions were added by this update. Summary of changes: .../doris/nereids/glue/translator/PhysicalPlanTranslator.java | 6 ++ 1 file changed, 2 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [minor](Nereids): remove useless override (#33651)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 45b7eee4429 [minor](Nereids): remove useless override (#33651)
45b7eee4429 is described below
commit 45b7eee44290ff0c7442328b37cc788d9f42990d
Author: jakevin
AuthorDate: Tue Apr 16 12:30:19 2024 +0800
[minor](Nereids): remove useless override (#33651)
---
.../org/apache/doris/nereids/trees/plans/FakePlan.java | 11 ++-
.../java/org/apache/doris/nereids/trees/plans/Plan.java | 4 +---
.../doris/nereids/trees/plans/commands/Command.java | 6 ++
.../trees/plans/physical/PhysicalOlapTableSink.java | 16
.../nereids/trees/plans/physical/PhysicalProject.java| 5 -
5 files changed, 17 insertions(+), 25 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/FakePlan.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/FakePlan.java
index ad186c680ce..4846fcd448f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/FakePlan.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/FakePlan.java
@@ -26,9 +26,13 @@ import
org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
import org.apache.doris.nereids.util.MutableState;
import org.apache.doris.nereids.util.MutableState.MultiMutableState;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
+import java.util.Set;
/**
* Used for unit test only.
@@ -88,7 +92,12 @@ public class FakePlan implements Plan {
@Override
public List getOutput() {
-return new ArrayList<>();
+return ImmutableList.of();
+}
+
+@Override
+public Set getOutputSet() {
+return ImmutableSet.of();
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/Plan.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/Plan.java
index d73b7390ce8..a4a9ec00def 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/Plan.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/Plan.java
@@ -98,9 +98,7 @@ public interface Plan extends TreeNode {
/**
* Get output slot set of the plan.
*/
-default Set getOutputSet() {
-return ImmutableSet.copyOf(getOutput());
-}
+Set getOutputSet();
/** getOutputExprIds */
default List getOutputExprIds() {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/Command.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/Command.java
index a87aaf3d302..4dcf3017097 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/Command.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/Command.java
@@ -33,6 +33,7 @@ import com.google.common.collect.ImmutableList;
import java.util.List;
import java.util.Optional;
+import java.util.Set;
/**
* All DDL and DML commands' super class.
@@ -101,6 +102,11 @@ public abstract class Command extends AbstractPlan
implements LogicalPlan, Block
throw new RuntimeException("Command do not implement getOutput");
}
+@Override
+public Set getOutputSet() {
+throw new RuntimeException("Command do not implement getOutputSet");
+}
+
@Override
public String treeString() {
throw new RuntimeException("Command do not implement treeString");
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalOlapTableSink.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalOlapTableSink.java
index debc0ece6c2..04cdc347db7 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalOlapTableSink.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalOlapTableSink.java
@@ -29,7 +29,6 @@ import org.apache.doris.nereids.properties.LogicalProperties;
import org.apache.doris.nereids.properties.PhysicalProperties;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.NamedExpression;
-import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.PlanType;
import org.apache.doris.nereids.trees.plans.algebra.Sink;
@@ -40,12 +39,10 @@ import org.apache.doris.statistics.Statistics;
import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
import java.util
(doris) branch branch-2.0 updated: [feature](Nereids): date literal suppose Zone (#33534) (#33683)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new ee474044293 [feature](Nereids): date literal suppose Zone (#33534)
(#33683)
ee474044293 is described below
commit ee47404429377a9100125a32998295826b194619
Author: jakevin
AuthorDate: Tue Apr 16 09:27:48 2024 +0800
[feature](Nereids): date literal suppose Zone (#33534) (#33683)
---
.../trees/expressions/literal/DateLiteral.java | 13 ---
.../doris/nereids/util/DateTimeFormatterUtils.java | 1 -
.../apache/doris/nereids/util/ExpressionUtils.java | 7
.../expressions/literal/DateTimeLiteralTest.java | 41 --
4 files changed, 22 insertions(+), 40 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java
index c6e19bb96cb..9693858fb7e 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java
@@ -244,19 +244,6 @@ public class DateLiteral extends Literal {
sb.append(s.substring(i));
-// Zone Part
-// while(i < s.length()) {
-//
-// }
-
-// add missing :00 in Zone part
-// int len = sb.length();
-// int signIdx = sb.indexOf("+", 10); // from index:10, skip date part
(it contains '-')
-// signIdx = signIdx == -1 ? sb.indexOf("-", 10) : signIdx;
-// if (signIdx != -1 && len - signIdx == 3) {
-// sb.append(":00");
-// }
-
return sb.toString();
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/DateTimeFormatterUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/DateTimeFormatterUtils.java
index 50d0f7169a5..6b52fb24a93 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/DateTimeFormatterUtils.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/DateTimeFormatterUtils.java
@@ -41,7 +41,6 @@ import java.time.temporal.ChronoField;
public class DateTimeFormatterUtils {
public static final DateTimeFormatter ZONE_FORMATTER = new
DateTimeFormatterBuilder()
.optionalStart()
-// .appendZoneText(TextStyle.FULL)
.appendZoneOrOffsetId()
.optionalEnd()
.toFormatter()
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java
index 46990827633..8bec208896a 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java
@@ -536,13 +536,6 @@ public class ExpressionUtils {
.collect(ImmutableSet.toImmutableSet());
}
-public static boolean checkTypeSkipCast(Expression expression, Class cls) {
-while (expression instanceof Cast) {
-expression = ((Cast) expression).child();
-}
-return cls.isInstance(expression);
-}
-
public static Expression getExpressionCoveredByCast(Expression expression)
{
while (expression instanceof Cast) {
expression = ((Cast) expression).child();
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteralTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteralTest.java
index c5e0b5eee6b..93039c4ad7f 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteralTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteralTest.java
@@ -137,25 +137,7 @@ class DateTimeLiteralTest {
new DateTimeV2Literal("2022-08-01 01:01:01Z");
new DateTimeV2Literal("2022-08-01 01:01:01Europe/Berlin");
new DateTimeV2Literal("2022-08-01 01:01:01Europe/London");
-}
-
-@Test
-@Disabled("Test results can change over time")
-void testZoneOrOffsetRight() {
-java.util.function.BiConsumer assertHour =
(dateTimeV2Literal, expectHour) -> {
-Assertions.assertEquals(dateTimeV2Literal.hour, expectHour);
-};
-DateTimeV2Literal dateTimeV2Literal;
-dateTimeV2Literal = new DateTimeV2Literal("2022-08-01
00:00:00Europe/London"); // +01:00
-assertHour.accept(dateTimeV2Literal, 7L);
-dateTimeV2Literal = new DateTimeV2Literal("2022-08-01
00:00:00America/New_York"); // -04:00
-asser
(doris) branch master updated: [feature](Nereids): date literal suppose Zone (#33534)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 96eb7ec9a21 [feature](Nereids): date literal suppose Zone (#33534)
96eb7ec9a21 is described below
commit 96eb7ec9a21a792a2d091c73dacf13d2c6f611ea
Author: jakevin
AuthorDate: Mon Apr 15 10:18:05 2024 +0800
[feature](Nereids): date literal suppose Zone (#33534)
support
```
'2022-05-01 01:02:55+02:30
'2022-05-01 01:02:55Asia/Shanghai
```
---
.../trees/expressions/literal/DateLiteral.java | 13 ---
.../doris/nereids/util/DateTimeFormatterUtils.java | 1 -
.../apache/doris/nereids/util/ExpressionUtils.java | 33 -
.../expressions/literal/DateTimeLiteralTest.java | 41 --
4 files changed, 22 insertions(+), 66 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java
index 89910b74374..93127933ea7 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java
@@ -246,19 +246,6 @@ public class DateLiteral extends Literal {
sb.append(s.substring(i));
-// Zone Part
-// while(i < s.length()) {
-//
-// }
-
-// add missing :00 in Zone part
-// int len = sb.length();
-// int signIdx = sb.indexOf("+", 10); // from index:10, skip date part
(it contains '-')
-// signIdx = signIdx == -1 ? sb.indexOf("-", 10) : signIdx;
-// if (signIdx != -1 && len - signIdx == 3) {
-// sb.append(":00");
-// }
-
return sb.toString();
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/DateTimeFormatterUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/DateTimeFormatterUtils.java
index 50d0f7169a5..6b52fb24a93 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/DateTimeFormatterUtils.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/DateTimeFormatterUtils.java
@@ -41,7 +41,6 @@ import java.time.temporal.ChronoField;
public class DateTimeFormatterUtils {
public static final DateTimeFormatter ZONE_FORMATTER = new
DateTimeFormatterBuilder()
.optionalStart()
-// .appendZoneText(TextStyle.FULL)
.appendZoneOrOffsetId()
.optionalEnd()
.toFormatter()
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java
index 401b8e0736b..e5e79f7b725 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java
@@ -427,32 +427,6 @@ public class ExpressionUtils {
}
}
-private static class ExpressionReplacerContext {
-private final Map
replaceMap;
-// if the key of replaceMap is named expr and withAlias is true, we
should
-// add alias after replaced
-private final boolean withAliasIfKeyNamed;
-
-private ExpressionReplacerContext(Map replaceMap,
-boolean withAliasIfKeyNamed) {
-this.replaceMap = replaceMap;
-this.withAliasIfKeyNamed = withAliasIfKeyNamed;
-}
-
-public static ExpressionReplacerContext of(Map replaceMap,
-boolean withAliasIfKeyNamed) {
-return new ExpressionReplacerContext(replaceMap,
withAliasIfKeyNamed);
-}
-
-public Map getReplaceMap()
{
-return replaceMap;
-}
-
-public boolean isWithAliasIfKeyNamed() {
-return withAliasIfKeyNamed;
-}
-}
-
/**
* merge arguments into an expression array
*
@@ -833,13 +807,6 @@ public class ExpressionUtils {
return set;
}
-public static boolean checkTypeSkipCast(Expression expression, Class cls) {
-while (expression instanceof Cast) {
-expression = ((Cast) expression).child();
-}
-return cls.isInstance(expression);
-}
-
public static Expression getExpressionCoveredByCast(Expression expression)
{
while (expression instanceof Cast) {
expression = ((Cast) expression).child();
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteralTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteralTest.java
index 8555d44d634..ea930e2b73d 100644
---
a/fe/fe-
(doris) branch master updated: [refactor](Nereids): compute unique and uniform property respectively (#32908)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 007e8158f45 [refactor](Nereids): compute unique and uniform property
respectively (#32908)
007e8158f45 is described below
commit 007e8158f4506e89f870097b4f1e28f94f24dae6
Author: 谢健
AuthorDate: Thu Apr 11 15:57:38 2024 +0800
[refactor](Nereids): compute unique and uniform property respectively
(#32908)
---
.../nereids/properties/FunctionalDependencies.java | 4 +
.../doris/nereids/trees/plans/AbstractPlan.java| 2 +-
.../trees/plans/BlockFuncDepsPropagation.java | 18 +-
.../nereids/trees/plans/PropagateFuncDeps.java | 20 +-
.../trees/plans/logical/LogicalAggregate.java | 108 +++---
.../trees/plans/logical/LogicalAssertNumRows.java | 33 +-
.../plans/logical/LogicalCatalogRelation.java | 23 +-
.../plans/logical/LogicalDeferMaterializeTopN.java | 29 +-
.../nereids/trees/plans/logical/LogicalExcept.java | 59 ++--
.../nereids/trees/plans/logical/LogicalFilter.java | 25 +-
.../trees/plans/logical/LogicalGenerate.java | 20 +-
.../nereids/trees/plans/logical/LogicalHaving.java | 17 +-
.../trees/plans/logical/LogicalIntersect.java | 27 +-
.../nereids/trees/plans/logical/LogicalJoin.java | 130 ---
.../nereids/trees/plans/logical/LogicalLimit.java | 30 +-
.../trees/plans/logical/LogicalOneRowRelation.java | 21 +-
.../nereids/trees/plans/logical/LogicalPlan.java | 17 +-
.../trees/plans/logical/LogicalProject.java| 65 ++--
.../nereids/trees/plans/logical/LogicalRepeat.java | 19 +-
.../trees/plans/logical/LogicalSubQueryAlias.java | 26 +-
.../nereids/trees/plans/logical/LogicalTopN.java | 30 +-
.../nereids/trees/plans/logical/LogicalUnion.java | 22 +-
.../nereids/trees/plans/logical/LogicalView.java | 20 +-
.../nereids/trees/plans/logical/LogicalWindow.java | 73 ++--
.../apache/doris/nereids/util/ExpressionUtils.java | 4 +-
.../properties/FunctionalDependenciesTest.java | 4 +-
.../doris/nereids/properties/UniformTest.java | 207
...tionalDependenciesTest.java => UniqueTest.java} | 376 ++---
28 files changed, 866 insertions(+), 563 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/FunctionalDependencies.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/FunctionalDependencies.java
index c7e6030e137..2b0c1f5c914 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/FunctionalDependencies.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/FunctionalDependencies.java
@@ -136,6 +136,10 @@ public class FunctionalDependencies {
uniqueSet.add(slotSet);
}
+public void addUniqueSlot(FunctionalDependencies
functionalDependencies) {
+uniqueSet.add(functionalDependencies.uniqueSet);
+}
+
public void addFdItems(ImmutableSet items) {
fdItems = ImmutableSet.copyOf(items);
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/AbstractPlan.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/AbstractPlan.java
index 286a92aab76..4747aa84898 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/AbstractPlan.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/AbstractPlan.java
@@ -179,7 +179,7 @@ public abstract class AbstractPlan extends
AbstractTreeNode implements Pla
} else {
Supplier> outputSupplier =
Suppliers.memoize(this::computeOutput);
Supplier fdSupplier = () -> this
instanceof LogicalPlan
-? ((LogicalPlan) this).computeFuncDeps(outputSupplier)
+? ((LogicalPlan) this).computeFuncDeps()
: FunctionalDependencies.EMPTY_FUNC_DEPS;
return new LogicalProperties(outputSupplier, fdSupplier);
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/BlockFuncDepsPropagation.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/BlockFuncDepsPropagation.java
index 07804587e3b..a679ad0f7d2 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/BlockFuncDepsPropagation.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/BlockFuncDepsPropagation.java
@@ -19,25 +19,31 @@ package org.apache.doris.nereids.trees.plans;
import org.apache.doris.nereids.properties.FdItem;
import org.apache.doris.nereids.properties.FunctionalDependencies;
-import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
import com.google.common.collect.ImmutableSet;
-import java.util.List;
-import java.util.function
(doris) branch master updated (65b382c17d4 -> e8e08f70051)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 65b382c17d4 [Fix](Outfile) Fixed the problem that the concurrent Outfile wrote multiple Success files (#33016) add e8e08f70051 [fix](Nereids): add order for constraint test (#33323) No new revisions were added by this update. Summary of changes: regression-test/data/nereids_syntax_p0/constraint.out | 8 regression-test/suites/nereids_syntax_p0/constraint.groovy | 14 +++--- 2 files changed, 11 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (b7e44d84750 -> ee48fa2305c)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from b7e44d84750 [Fix](grace-exit) Fix threadpool hang on grace exit (#33132) add ee48fa2305c [fix](Nereids): don't transpose agg and join if join is mark join (#33312) No new revisions were added by this update. Summary of changes: .../rules/exploration/TransposeAggSemiJoin.java| 4 ++-- .../rules/exploration/TransposeAggSemiJoinProject.java | 4 ++-- .../rules/exploration/TransposeAggSemiJoinTest.java| 18 ++ .../rules/rewrite/TransposeSemiJoinAggProjectTest.java | 15 +++ 4 files changed, 37 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (8a4180d0cb4 -> 2e8bdfb0711)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 8a4180d0cb4 [Update](inverted index) add config for inverted index query cache shards (#32666) add 2e8bdfb0711 [fix](Nereids): fix elimiate join by pkfk when there are multi joins (#32703) No new revisions were added by this update. Summary of changes: .../nereids/rules/rewrite/ForeignKeyContext.java | 2 +- .../data/nereids_rules_p0/pkfk/eliminate_inner.out | 28 ++ .../nereids_rules_p0/pkfk/eliminate_inner.groovy | 8 ++- 3 files changed, 36 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated (49bbdebf8a8 -> 409a44b76cd)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git from 49bbdebf8a8 [fix](invert index) Fixed compatibility issues with tii tis (#32651) add 409a44b76cd pick (#32305) (#31858) (#32476) No new revisions were added by this update. Summary of changes: .../doris/nereids/jobs/executor/Rewriter.java | 6 ++ .../org/apache/doris/nereids/rules/RuleType.java | 2 + .../rules/rewrite/PullUpProjectUnderLimit.java | 57 +++ .../rules/rewrite/PullUpProjectUnderTopN.java | 81 ++ .../trees/expressions/literal/DateLiteral.java | 46 ++-- .../apache/doris/nereids/trees/plans/JoinType.java | 4 ++ .../apache/doris/nereids/util/ExpressionUtils.java | 30 .../expressions/literal/DateTimeLiteralTest.java | 7 ++ 8 files changed, 208 insertions(+), 25 deletions(-) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectUnderLimit.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectUnderTopN.java - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated (1e4fa20d55b -> f94e14d9c54)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git discard 1e4fa20d55b [fix](Nereids): slot set in condition can be empty (#32169) omit 27973b6 [fix](schema-change) fix the bug of handling empty blocks in schema change (#32396) new 88170a94e87 (invert index) modify of time series compaction policy new 99847923413 (jdbc catalog) Add a property to test the connection when creating a Jdbc catalog (#32125) new c472746c2a0 (Nereids): remove test infer_set_operator_distinct(CBO) new e912c107db4 (jni) remove 'push_down_predicates' and fix BE crash with decimal predicate (#32253) new f94e14d9c54 (kerberos)fix hive keberos principal usage This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (1e4fa20d55b) \ N -- N -- N refs/heads/branch-2.1 (f94e14d9c54) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 5 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: be/src/agent/task_worker_pool.cpp | 23 +- be/src/olap/cumulative_compaction.cpp | 8 +- be/src/olap/cumulative_compaction_policy.h | 8 + .../cumulative_compaction_time_series_policy.cpp | 108 +- .../cumulative_compaction_time_series_policy.h | 4 + be/src/olap/full_compaction.cpp| 6 +- be/src/olap/rowset/rowset_meta.h | 6 + be/src/olap/schema_change.cpp | 12 +- be/src/olap/tablet_meta.cpp| 17 +- be/src/olap/tablet_meta.h | 10 +- be/src/service/internal_service.cpp| 60 be/src/service/internal_service.h | 5 + be/src/vec/exec/jni_connector.cpp | 17 +- be/src/vec/exec/jni_connector.h| 3 + be/src/vec/exec/vjdbc_connector.cpp| 27 +- be/src/vec/exec/vjdbc_connector.h | 5 + docs/en/docs/lakehouse/multi-catalog/jdbc.md | 2 + .../Create/CREATE-TABLE.md | 9 + docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md| 3 +- .../Create/CREATE-TABLE.md | 8 + .../java/org/apache/doris/avro/AvroJNIScanner.java | 3 +- .../java/org/apache/doris/hudi/HudiJniScanner.java | 16 +- .../org/apache/doris/common/jni/JniScanner.java| 12 +- .../apache/doris/common/jni/MockJniScanner.java| 11 +- .../org/apache/doris/jdbc/BaseJdbcExecutor.java| 20 ++ .../org/apache/doris/jdbc/DefaultJdbcExecutor.java | 20 ++ .../doris/maxcompute/MaxComputeJniScanner.java | 16 +- .../org/apache/doris/paimon/PaimonJniScanner.java | 3 +- .../main/java/org/apache/doris/alter/Alter.java| 4 +- .../java/org/apache/doris/alter/RollupJobV2.java | 1 + .../apache/doris/alter/SchemaChangeHandler.java| 7 + .../org/apache/doris/alter/SchemaChangeJobV2.java | 1 + .../analysis/ModifyTablePropertiesClause.java | 17 + .../java/org/apache/doris/backup/RestoreJob.java | 1 + .../main/java/org/apache/doris/catalog/Env.java| 11 +- .../org/apache/doris/catalog/JdbcResource.java | 9 +- .../java/org/apache/doris/catalog/OlapTable.java | 14 + .../org/apache/doris/catalog/TableProperty.java| 18 +- .../apache/doris/common/util/PropertyAnalyzer.java | 29 +- .../apache/doris/datasource/InternalCatalog.java | 20 +- .../doris/datasource/hive/HMSExternalCatalog.java | 4 +- .../datasource/hive/HiveMetaStoreClientHelper.java | 11 +- .../doris/datasource/jdbc/JdbcExternalCatalog.java | 99 ++ .../doris/datasource/jdbc/client/JdbcClient.java | 18 + .../datasource/jdbc/client/JdbcOracleClient.java | 5 + .../datasource/jdbc/client/JdbcSapHanaClient.java | 5 + .../org/apache/doris/master/ReportHandler.java | 1 + .../rules/rewrite/PullUpProjectUnderLimit.java | 6 +- .../rules/rewrite/PullUpProjectUnderTopN.java | 38 +-- .../apache/doris/nereids/trees/plans/JoinType.java | 4 - .../org/apache/doris/rpc/BackendServiceClient.java | 5 + .../org/apache/doris/rpc/BackendServiceProxy.java | 12 + .../org/apache/doris/tas
(doris) 02/05: (jdbc catalog) Add a property to test the connection when creating a Jdbc catalog (#32125)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 9984792341350337ddd5490046c99a23dd083d95
Author: zy-kkk
AuthorDate: Wed Mar 13 12:43:00 2024 +0800
(jdbc catalog) Add a property to test the connection when creating a Jdbc
catalog (#32125)
---
be/src/service/internal_service.cpp| 60 +
be/src/service/internal_service.h | 5 ++
be/src/vec/exec/vjdbc_connector.cpp| 27 +-
be/src/vec/exec/vjdbc_connector.h | 5 ++
docs/en/docs/lakehouse/multi-catalog/jdbc.md | 2 +
docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md| 3 +-
.../org/apache/doris/jdbc/BaseJdbcExecutor.java| 20 +
.../org/apache/doris/jdbc/DefaultJdbcExecutor.java | 20 +
.../org/apache/doris/catalog/JdbcResource.java | 9 +-
.../doris/datasource/jdbc/JdbcExternalCatalog.java | 99 +
.../doris/datasource/jdbc/client/JdbcClient.java | 18
.../datasource/jdbc/client/JdbcOracleClient.java | 5 ++
.../datasource/jdbc/client/JdbcSapHanaClient.java | 5 ++
.../org/apache/doris/rpc/BackendServiceClient.java | 5 ++
.../org/apache/doris/rpc/BackendServiceProxy.java | 12 +++
gensrc/proto/internal_service.proto| 11 +++
.../jdbc/test_clickhouse_jdbc_catalog.out | Bin 4991 -> 5426 bytes
.../jdbc/test_clickhouse_jdbc_catalog.groovy | 30 ++-
18 files changed, 331 insertions(+), 5 deletions(-)
diff --git a/be/src/service/internal_service.cpp
b/be/src/service/internal_service.cpp
index 7a37d613263..65cc431b4d5 100644
--- a/be/src/service/internal_service.cpp
+++ b/be/src/service/internal_service.cpp
@@ -39,6 +39,7 @@
#include
#include
#include
+#include
#include
#include
@@ -786,6 +787,65 @@ void
PInternalServiceImpl::tablet_fetch_data(google::protobuf::RpcController* co
}
}
+void PInternalService::test_jdbc_connection(google::protobuf::RpcController*
controller,
+const PJdbcTestConnectionRequest*
request,
+PJdbcTestConnectionResult* result,
+google::protobuf::Closure* done) {
+bool ret = _heavy_work_pool.try_offer([request, result, done]() {
+VLOG_RPC << "test jdbc connection";
+brpc::ClosureGuard closure_guard(done);
+TTableDescriptor table_desc;
+vectorized::JdbcConnectorParam jdbc_param;
+Status st = Status::OK();
+{
+const uint8_t* buf = (const uint8_t*)request->jdbc_table().data();
+uint32_t len = request->jdbc_table().size();
+st = deserialize_thrift_msg(buf, &len, false, &table_desc);
+if (!st.ok()) {
+LOG(WARNING) << "test jdbc connection failed, errmsg=" << st;
+st.to_protobuf(result->mutable_status());
+return;
+}
+}
+TJdbcTable jdbc_table = (table_desc.jdbcTable);
+jdbc_param.catalog_id = jdbc_table.catalog_id;
+jdbc_param.driver_class = jdbc_table.jdbc_driver_class;
+jdbc_param.driver_path = jdbc_table.jdbc_driver_url;
+jdbc_param.driver_checksum = jdbc_table.jdbc_driver_checksum;
+jdbc_param.jdbc_url = jdbc_table.jdbc_url;
+jdbc_param.user = jdbc_table.jdbc_user;
+jdbc_param.passwd = jdbc_table.jdbc_password;
+jdbc_param.query_string = request->query_str();
+jdbc_param.table_type =
static_cast(request->jdbc_table_type());
+jdbc_param.use_transaction = false;
+jdbc_param.connection_pool_min_size =
jdbc_table.connection_pool_min_size;
+jdbc_param.connection_pool_max_size =
jdbc_table.connection_pool_max_size;
+jdbc_param.connection_pool_max_life_time =
jdbc_table.connection_pool_max_life_time;
+jdbc_param.connection_pool_max_wait_time =
jdbc_table.connection_pool_max_wait_time;
+jdbc_param.connection_pool_keep_alive =
jdbc_table.connection_pool_keep_alive;
+
+std::unique_ptr jdbc_connector;
+jdbc_connector.reset(new (std::nothrow)
vectorized::JdbcConnector(jdbc_param));
+
+st = jdbc_connector->test_connection();
+st.to_protobuf(result->mutable_status());
+
+Status clean_st = jdbc_connector->clean_datasource();
+if (!clean_st.ok()) {
+LOG(WARNING) << "Failed to clean JDBC datasource: " <<
clean_st.msg();
+}
+Status close_st = jdbc_connector->close();
+if (!close_st.ok()) {
+LOG(WARNING) << "Failed to close JDBC connector: " <<
close_st.msg();
+}
+});
+
+if (!ret) {
+offer_failed(result, done, _heavy_work_pool);
+re
(doris) 03/05: (Nereids): remove test infer_set_operator_distinct(CBO)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a commit to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git commit c472746c2a03f1c03a191a6cf465e6b6b98a48e5 Author: jackwener AuthorDate: Wed Mar 20 17:08:06 2024 +0800 (Nereids): remove test infer_set_operator_distinct(CBO) --- .../infer_set_operator_distinct.out| 372 - .../infer_set_operator_distinct.groovy | 172 -- 2 files changed, 544 deletions(-) diff --git a/regression-test/data/nereids_rules_p0/infer_set_operator_distinct/infer_set_operator_distinct.out b/regression-test/data/nereids_rules_p0/infer_set_operator_distinct/infer_set_operator_distinct.out deleted file mode 100644 index 786072eae0d..000 --- a/regression-test/data/nereids_rules_p0/infer_set_operator_distinct/infer_set_operator_distinct.out +++ /dev/null @@ -1,372 +0,0 @@ --- This file is automatically generated. You should know what you did if you want to edit this --- !union_distinct -- -PhysicalResultSink ---PhysicalDistribute[DistributionSpecGather] -hashAgg[GLOBAL] ---PhysicalDistribute[DistributionSpecHash] -hashAgg[LOCAL] ---PhysicalUnion -PhysicalDistribute[DistributionSpecExecutionAny] ---PhysicalProject -PhysicalOlapScan[t1] -PhysicalDistribute[DistributionSpecExecutionAny] ---PhysicalProject -PhysicalOlapScan[t2] - --- !union_complex_conditions -- -PhysicalResultSink ---PhysicalDistribute[DistributionSpecGather] -hashAgg[GLOBAL] ---PhysicalDistribute[DistributionSpecHash] -hashAgg[LOCAL] ---PhysicalUnion -PhysicalDistribute[DistributionSpecExecutionAny] ---PhysicalProject -filter((t1.score > 10)) ---PhysicalOlapScan[t1] -PhysicalDistribute[DistributionSpecExecutionAny] ---PhysicalProject -filter((t2.name = 'Test')) ---PhysicalOlapScan[t2] - --- !multi_union -- -PhysicalResultSink ---PhysicalDistribute[DistributionSpecGather] -hashAgg[GLOBAL] ---PhysicalDistribute[DistributionSpecHash] -hashAgg[LOCAL] ---PhysicalUnion -PhysicalDistribute[DistributionSpecExecutionAny] ---PhysicalProject -PhysicalOlapScan[t1] -PhysicalDistribute[DistributionSpecExecutionAny] ---PhysicalProject -PhysicalOlapScan[t2] -PhysicalDistribute[DistributionSpecExecutionAny] ---PhysicalProject -PhysicalOlapScan[t3] - --- !except_distinct -- -PhysicalResultSink ---PhysicalDistribute[DistributionSpecGather] -PhysicalExcept ---PhysicalDistribute[DistributionSpecHash] -PhysicalProject ---PhysicalOlapScan[t1] ---PhysicalDistribute[DistributionSpecHash] -PhysicalProject ---PhysicalOlapScan[t2] - --- !except_with_filter -- -PhysicalResultSink ---PhysicalDistribute[DistributionSpecGather] -PhysicalExcept ---PhysicalDistribute[DistributionSpecHash] -PhysicalProject ---filter((t1.id > 100)) -PhysicalOlapScan[t1] ---PhysicalDistribute[DistributionSpecHash] -PhysicalProject ---filter((t2.id < 50)) -PhysicalOlapScan[t2] - --- !intersect_distinct -- -PhysicalResultSink ---PhysicalDistribute[DistributionSpecGather] -PhysicalIntersect ---PhysicalDistribute[DistributionSpecHash] -PhysicalProject ---PhysicalOlapScan[t1] ---PhysicalDistribute[DistributionSpecHash] -PhysicalProject ---PhysicalOlapScan[t2] - --- !intersect_with_aggregate -- -PhysicalResultSink ---PhysicalDistribute[DistributionSpecGather] -PhysicalIntersect ---PhysicalDistribute[DistributionSpecHash] -PhysicalProject ---hashAgg[GLOBAL] -PhysicalDistribute[DistributionSpecGather] ---hashAgg[LOCAL] -PhysicalProject ---PhysicalOlapScan[t1] ---PhysicalDistribute[DistributionSpecHash] -PhysicalProject ---hashAgg[GLOBAL] -PhysicalDistribute[DistributionSpecGather] ---hashAgg[LOCAL] -PhysicalProject ---PhysicalOlapScan[t2] - --- !mixed_set_operators -- -PhysicalResultSink ---PhysicalDistribute[DistributionSpecGather] -PhysicalIntersect ---PhysicalExcept -hashAgg[GLOBAL] ---PhysicalDistribute[DistributionSpecHash] -hashAgg[LOCAL] ---PhysicalUnion -PhysicalDistribute[DistributionSpecExecutionAny] ---PhysicalProject -PhysicalOlapScan[t1] -PhysicalDistribute[DistributionSpecExecutionAny] ---PhysicalProject -PhysicalOlapScan[t2] -PhysicalDistribute[D
(doris) 04/05: (jni) remove 'push_down_predicates' and fix BE crash with decimal predicate (#32253)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit e912c107db436b269eb3d646539b8d118927348c
Author: Mingyu Chen
AuthorDate: Fri Mar 15 22:23:06 2024 +0800
(jni) remove 'push_down_predicates' and fix BE crash with decimal predicate
(#32253)
---
be/src/vec/exec/jni_connector.cpp | 17 ++---
be/src/vec/exec/jni_connector.h | 3 +++
.../main/java/org/apache/doris/avro/AvroJNIScanner.java | 3 +--
.../main/java/org/apache/doris/hudi/HudiJniScanner.java | 16 +---
.../java/org/apache/doris/common/jni/JniScanner.java| 12 +---
.../org/apache/doris/common/jni/MockJniScanner.java | 11 +--
.../apache/doris/maxcompute/MaxComputeJniScanner.java | 16 +++-
.../java/org/apache/doris/paimon/PaimonJniScanner.java | 3 +--
8 files changed, 29 insertions(+), 52 deletions(-)
diff --git a/be/src/vec/exec/jni_connector.cpp
b/be/src/vec/exec/jni_connector.cpp
index ffe2742b117..7f29c023e7c 100644
--- a/be/src/vec/exec/jni_connector.cpp
+++ b/be/src/vec/exec/jni_connector.cpp
@@ -102,13 +102,16 @@ Status JniConnector::open(RuntimeState* state,
RuntimeProfile* profile) {
Status JniConnector::init(
std::unordered_map*
colname_to_value_range) {
-_generate_predicates(colname_to_value_range);
-if (_predicates_length != 0 && _predicates != nullptr) {
-int64_t predicates_address = (int64_t)_predicates.get();
-// We can call
org.apache.doris.common.jni.vec.ScanPredicate#parseScanPredicates to parse the
-// serialized predicates in java side.
-_scanner_params.emplace("push_down_predicates",
std::to_string(predicates_address));
-}
+// TODO: This logic need to be changed.
+// See the comment of "predicates" field in JniScanner.java
+
+// _generate_predicates(colname_to_value_range);
+// if (_predicates_length != 0 && _predicates != nullptr) {
+// int64_t predicates_address = (int64_t)_predicates.get();
+// // We can call
org.apache.doris.common.jni.vec.ScanPredicate#parseScanPredicates to parse the
+// // serialized predicates in java side.
+// _scanner_params.emplace("push_down_predicates",
std::to_string(predicates_address));
+// }
return Status::OK();
}
diff --git a/be/src/vec/exec/jni_connector.h b/be/src/vec/exec/jni_connector.h
index ed282fc0c40..dc21be85b77 100644
--- a/be/src/vec/exec/jni_connector.h
+++ b/be/src/vec/exec/jni_connector.h
@@ -163,6 +163,9 @@ public:
char_ptr += s->size;
}
} else {
+// FIXME: it can not handle decimal type correctly.
+// but this logic is deprecated and not used.
+// so may be deleted or fixed later.
for (const CppType* v : values) {
int type_len = sizeof(CppType);
*reinterpret_cast(char_ptr) = type_len;
diff --git
a/fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/AvroJNIScanner.java
b/fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/AvroJNIScanner.java
index 17a185d03ae..dc845f43cb8 100644
---
a/fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/AvroJNIScanner.java
+++
b/fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/AvroJNIScanner.java
@@ -19,7 +19,6 @@ package org.apache.doris.avro;
import org.apache.doris.common.jni.JniScanner;
import org.apache.doris.common.jni.vec.ColumnType;
-import org.apache.doris.common.jni.vec.ScanPredicate;
import org.apache.doris.common.jni.vec.TableSchema;
import org.apache.doris.thrift.TFileType;
@@ -173,7 +172,7 @@ public class AvroJNIScanner extends JniScanner {
try {
initAvroFileContext();
initFieldInspector();
-initTableInfo(requiredTypes, requiredFields, new ScanPredicate[0],
fetchSize);
+initTableInfo(requiredTypes, requiredFields, fetchSize);
} catch (Exception e) {
LOG.warn("Failed to init avro scanner. ", e);
throw new RuntimeException(e);
diff --git
a/fe/be-java-extensions/hudi-scanner/src/main/java/org/apache/doris/hudi/HudiJniScanner.java
b/fe/be-java-extensions/hudi-scanner/src/main/java/org/apache/doris/hudi/HudiJniScanner.java
index 932b53b9a7c..0a1b69fcfb0 100644
---
a/fe/be-java-extensions/hudi-scanner/src/main/java/org/apache/doris/hudi/HudiJniScanner.java
+++
b/fe/be-java-extensions/hudi-scanner/src/main/java/org/apache/doris/hudi/HudiJniScanner.java
@@ -20,7 +20,6 @@ package org.apache.doris.hudi;
import org.apache.doris.common.jni.JniScanner;
import org.apache.doris.common.jni.vec.ColumnType;
-import org.apache.doris.common.jni.vec.ScanPredicate;
import
(doris) 05/05: (kerberos)fix hive keberos principal usage
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit f94e14d9c54b11e020d2511cf9c2f48d336d56f6
Author: slothever
AuthorDate: Thu Mar 21 15:34:57 2024 +0800
(kerberos)fix hive keberos principal usage
---
.../org/apache/doris/datasource/hive/HMSExternalCatalog.java | 4 ++--
.../doris/datasource/hive/HiveMetaStoreClientHelper.java | 11 +++
2 files changed, 5 insertions(+), 10 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java
index 793a8a24228..0f2a7bb2acb 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java
@@ -142,8 +142,8 @@ public class HMSExternalCatalog extends ExternalCatalog {
hiveConf.set(HiveConf.ConfVars.METASTORE_CLIENT_SOCKET_TIMEOUT.name(),
String.valueOf(Config.hive_metastore_client_timeout_second));
HadoopUGI.tryKrbLogin(this.getName(),
AuthenticationConfig.getKerberosConfig(hiveConf,
-AuthenticationConfig.HIVE_KERBEROS_PRINCIPAL,
-AuthenticationConfig.HIVE_KERBEROS_KEYTAB));
+AuthenticationConfig.HADOOP_KERBEROS_PRINCIPAL,
+AuthenticationConfig.HADOOP_KERBEROS_KEYTAB));
}
metadataOps = ExternalMetadataOperations.newHiveMetadataOps(hiveConf,
jdbcClientConfig, this);
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreClientHelper.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreClientHelper.java
index 23c83a11146..4cbad277d43 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreClientHelper.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreClientHelper.java
@@ -817,15 +817,10 @@ public class HiveMetaStoreClientHelper {
}
public static T ugiDoAs(Configuration conf,
PrivilegedExceptionAction action) {
+// if hive config is not ready, then use hadoop kerberos to login
AuthenticationConfig krbConfig =
AuthenticationConfig.getKerberosConfig(conf,
-AuthenticationConfig.HIVE_KERBEROS_PRINCIPAL,
-AuthenticationConfig.HIVE_KERBEROS_KEYTAB);
-if (!krbConfig.isValid()) {
-// if hive config is not ready, then use hadoop kerberos to login
-krbConfig = AuthenticationConfig.getKerberosConfig(conf,
-AuthenticationConfig.HADOOP_KERBEROS_PRINCIPAL,
-AuthenticationConfig.HADOOP_KERBEROS_KEYTAB);
-}
+AuthenticationConfig.HADOOP_KERBEROS_PRINCIPAL,
+AuthenticationConfig.HADOOP_KERBEROS_KEYTAB);
return HadoopUGI.ugiDoAs(krbConfig, action);
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 01/05: (invert index) modify of time series compaction policy
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 88170a94e8737e19696f614b1ea21ca5d0f407d3
Author: zzzxl1993 <474696...@qq.com>
AuthorDate: Tue Mar 19 18:49:43 2024 +0800
(invert index) modify of time series compaction policy
---
be/src/agent/task_worker_pool.cpp | 23 +++--
be/src/olap/cumulative_compaction.cpp | 8 +-
be/src/olap/cumulative_compaction_policy.h | 8 ++
.../cumulative_compaction_time_series_policy.cpp | 108 +++--
.../cumulative_compaction_time_series_policy.h | 4 +
be/src/olap/full_compaction.cpp| 6 +-
be/src/olap/rowset/rowset_meta.h | 6 ++
be/src/olap/tablet_meta.cpp| 17 +++-
be/src/olap/tablet_meta.h | 10 +-
.../Create/CREATE-TABLE.md | 9 ++
.../Create/CREATE-TABLE.md | 8 ++
.../main/java/org/apache/doris/alter/Alter.java| 4 +-
.../java/org/apache/doris/alter/RollupJobV2.java | 1 +
.../apache/doris/alter/SchemaChangeHandler.java| 7 ++
.../org/apache/doris/alter/SchemaChangeJobV2.java | 1 +
.../analysis/ModifyTablePropertiesClause.java | 17
.../java/org/apache/doris/backup/RestoreJob.java | 1 +
.../main/java/org/apache/doris/catalog/Env.java| 11 ++-
.../java/org/apache/doris/catalog/OlapTable.java | 14 +++
.../org/apache/doris/catalog/TableProperty.java| 18 +++-
.../apache/doris/common/util/PropertyAnalyzer.java | 29 +-
.../apache/doris/datasource/InternalCatalog.java | 20 +++-
.../org/apache/doris/master/ReportHandler.java | 1 +
.../org/apache/doris/task/CreateReplicaTask.java | 5 +
.../doris/task/UpdateTabletMetaInfoTask.java | 5 +
.../java/org/apache/doris/task/AgentTaskTest.java | 2 +-
gensrc/proto/olap_file.proto | 2 +
gensrc/thrift/AgentService.thrift | 2 +
28 files changed, 320 insertions(+), 27 deletions(-)
diff --git a/be/src/agent/task_worker_pool.cpp
b/be/src/agent/task_worker_pool.cpp
index 00c16cfa12d..34918a5b0a2 100644
--- a/be/src/agent/task_worker_pool.cpp
+++ b/be/src/agent/task_worker_pool.cpp
@@ -52,6 +52,7 @@
#include "io/fs/local_file_system.h"
#include "io/fs/path.h"
#include "io/fs/s3_file_system.h"
+#include "olap/cumulative_compaction_time_series_policy.h"
#include "olap/data_dir.h"
#include "olap/olap_common.h"
#include "olap/rowset/rowset_meta.h"
@@ -691,8 +692,8 @@ void update_tablet_meta_callback(StorageEngine& engine,
const TAgentTaskRequest&
need_to_save = true;
}
if (tablet_meta_info.__isset.compaction_policy) {
-if (tablet_meta_info.compaction_policy != "size_based" &&
-tablet_meta_info.compaction_policy != "time_series") {
+if (tablet_meta_info.compaction_policy !=
CUMULATIVE_SIZE_BASED_POLICY &&
+tablet_meta_info.compaction_policy !=
CUMULATIVE_TIME_SERIES_POLICY) {
status = Status::InvalidArgument(
"invalid compaction policy, only support for
size_based or "
"time_series");
@@ -702,7 +703,7 @@ void update_tablet_meta_callback(StorageEngine& engine,
const TAgentTaskRequest&
need_to_save = true;
}
if (tablet_meta_info.__isset.time_series_compaction_goal_size_mbytes) {
-if (tablet->tablet_meta()->compaction_policy() != "time_series") {
+if (tablet->tablet_meta()->compaction_policy() !=
CUMULATIVE_TIME_SERIES_POLICY) {
status = Status::InvalidArgument(
"only time series compaction policy support time
series config");
continue;
@@ -712,7 +713,7 @@ void update_tablet_meta_callback(StorageEngine& engine,
const TAgentTaskRequest&
need_to_save = true;
}
if
(tablet_meta_info.__isset.time_series_compaction_file_count_threshold) {
-if (tablet->tablet_meta()->compaction_policy() != "time_series") {
+if (tablet->tablet_meta()->compaction_policy() !=
CUMULATIVE_TIME_SERIES_POLICY) {
status = Status::InvalidArgument(
"only time series compaction policy support time
series config");
continue;
@@ -722,7 +723,7 @@ void update_tablet_meta_callback(StorageEngine& engine,
const TAgentTaskRequest&
need_to_save = true;
}
if
(tablet_meta_info.__isset.time_series_compaction_time_threshold_seconds) {
-if (tablet->tablet_meta()-
(doris) branch branch-2.1 updated (f4c00535f43 -> 1e4fa20d55b)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git omit f4c00535f43 [fix](kerberos)fix hive keberos principal usage (#32612) omit bc279f0ce7b [test](Nereids): remove test infer_set_operator_distinct(CBO) (#32547) omit 2a7db2a4ff1 [opt](invert index) modify of time series compaction policy #31488 (#32483) omit 4c8aaa156ae [fix](jni) remove 'push_down_predicates' and fix BE crash with decimal predicate (#32253) (#32599) omit a7fa9f290ca [fix](test)change stream load of auto partition (#32544) omit 48d7585d85f [enhancement](test) unique model by modify a value type from INT or BIGINT to other type (#32553) omit dea6859e0d7 [refactor](jdbc catalog) refactor jdbc catalog get databases logic (#32579) omit 043113c89e6 [enhancement](test) unique model by modify a key type from SMALLINT to other type (#32033) omit 617cc667fe1 [Fix](Variant) fix variant serialize root node (#31769) omit 02ef02402ad [pipelineX](debug) Add debug logs for long-running load task (#32534) omit 2c87238504f [enhance](S3) Print the oss request id for each error s3 request (#32491) omit 60a673979e5 [case](Cloud) Add ssb case for hdfs vault (#32567) omit 02430e6e53b [enhance](S3) Print the oss request id for each error s3 request (#32499) omit d2968dcf994 [fix](jdbc catalog) Fixed the problem that the metadata_refresh_interval_sec properties cannot be set (#32516) omit 085696744dd [Enhancement] when partition column is datetime, date can work in create table command (#32335) omit 7486e96b121 [improve](function) add error msg if exceeded maximum default value in repeat function (#32219) omit 8e3d28b93a9 [Feature] Support Array Type compare function for nereids planner (#31701) omit 6d076f99476 [improvement](group_comit) Add bvar to monitor the total wal count on disk (#31646) omit 09be4dc7ee0 [fix](random-bucket) tabletindex when there is no cached value in memory (#32336) omit 06bf5541f27 [pipelineX](fix) Fix running tasks API core dump (#32503) omit 0db402e154c [expr](fix) Not to throw exception when close failed (#32287) omit a40463617e8 [feature](cpu cores) get the cores when running within a cgroup. (#32370) omit 95322e2ebeb [opt](variable) user variable support expression rather than literal (#32492) omit 2a1d2ad647c [fix](Nereids) decimalv2 and int/date bit arithmetic need cast to bigint (#32311) omit b92a7646658 [feature](function) Support for aggregate function foreach combiner for some error function (#31913) omit b6a35d68b0b [code](Refactor) Del unless filter id in runtime filter func (#32502) omit 0e493add692 [regression-test](case) forbid test_stream_stub_fault_injection (#32540) omit 22782fec925 [opt](routine-load) optimize allocate task to be algorithm for load balance (#32021) omit 6871c964af9 [fix](nereids)NullSafeEqualToEqual rule only change to equal if both children are not nullable (#32374) omit 1b0c4d3aa25 [chor](log) Change log4j rollover strategy to 'max' (#32116) omit 6d401451cc3 [fix](stats) sort partitions when do sample analyze (#32185) omit 9f98200f27e [case](mtmv)MTMV external catalog case (#32390) omit 4b21c5c1cff [Enhencement](Nereids) add eliminate distinct constant rule (#32307) omit 4efeb6618a8 [Fix](inverted index) fix inappropriate use of macro in inverted index fs directory error process (#32472) omit ea8d4f2d0bb [fix][regression]update ccr test project (#32445) omit 50c247e08cd [fix](snapshot-loader) Fix be crash caused by deref end() iterator (#32489) omit 612d3595e4e [improvement](spill) optimize the spilling logic of hash join operator (#32202) omit e892774c9a5 [improvement](agg) streaming agg should not take too much memory when spilling enabled (#32426) omit 7484a7ba5f8 [fix](broker load) improve the checking of overlapping partitions of same table (#32254) omit 2196c534e81 [fix](group commit) Fix compatibility issues on serializing and deserializing wal file (#32299) omit 2057886d30b [fix](group commit) Fix invalid function problem on p2 regression-test (#32481) omit 3c377a89577 [fix](group commit) Fix group commit connect to observer fe (#3) omit 14c9537679a [fix](decimal) fix Arithmetic Overflow error of converting string to decimal (#32246) omit ab512f935c5 [pipelineX](api) Add api for long-running tasks (#32459) omit 66fe61b5910 [fix](nereids)support topn-filter for non pipeline engine #32397 omit f99db389983 [fix](ParquetReader) Fix Parquet Reader to read `int96` parquet type problem (#32394) omit e541ca9f11d [Fix](Job)When jobname is the do keyword, parsing errors will occur when executing SQL. (#32379) omit 725f86a27b2 [fix](group commit) Fix p2 regression-test (#32270) omit 0635a8716ca [improve](group commit)
(doris) 01/01: [fix](Nereids): slot set in condition can be empty (#32169)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 1e4fa20d55bb3c7b44df7231df78fe594b59f3b3
Author: jackwener
AuthorDate: Tue Mar 19 17:36:50 2024 +0800
[fix](Nereids): slot set in condition can be empty (#32169)
(cherry picked from commit ca092139a007dca99006d8247d9926947b1490d3)
---
.../rules/rewrite/PullUpProjectUnderLimit.java | 6 ++--
.../rules/rewrite/PullUpProjectUnderTopN.java | 38 +-
.../apache/doris/nereids/trees/plans/JoinType.java | 4 +++
3 files changed, 37 insertions(+), 11 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectUnderLimit.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectUnderLimit.java
index a1a4ac4d219..cc2e6388a60 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectUnderLimit.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectUnderLimit.java
@@ -21,6 +21,7 @@ import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import org.apache.doris.nereids.util.PlanUtils;
@@ -36,9 +37,10 @@ import java.util.stream.Collectors;
public class PullUpProjectUnderLimit extends OneRewriteRuleFactory {
@Override
public Rule build() {
-return logicalLimit(logicalProject().whenNot(p -> p.isAllSlots()))
+return logicalLimit(logicalProject(logicalJoin().when(j ->
j.getJoinType().isLeftRightOuterOrCrossJoin()))
+.whenNot(p -> p.isAllSlots()))
.then(limit -> {
-LogicalProject project = limit.child();
+LogicalProject> project =
limit.child();
Set allUsedSlots =
project.getProjects().stream().flatMap(ne -> ne.getInputSlots().stream())
.collect(Collectors.toSet());
Set outputSet = project.child().getOutputSet();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectUnderTopN.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectUnderTopN.java
index b02edd9033b..469444006d1 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectUnderTopN.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectUnderTopN.java
@@ -20,41 +20,61 @@ package org.apache.doris.nereids.rules.rewrite;
import org.apache.doris.nereids.properties.OrderKey;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
+import org.apache.doris.nereids.util.ExpressionUtils;
import org.apache.doris.nereids.util.PlanUtils;
import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableList;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;
/**
- * Pull up Project under TopN.
+ * Pull up Project under TopN for PushDownTopNThroughJoin
*/
public class PullUpProjectUnderTopN extends OneRewriteRuleFactory {
@Override
public Rule build() {
-return logicalTopN(logicalProject().whenNot(p -> p.isAllSlots()))
-.whenNot(topN ->
topN.child().hasPushedDownToProjectionFunctions())
+return logicalTopN(
+logicalProject(logicalJoin().when(j ->
j.getJoinType().isLeftRightOuterOrCrossJoin()))
+.whenNot(p -> p.isAllSlots()))
.then(topN -> {
-LogicalProject project = topN.child();
+LogicalProject> project =
topN.child();
Set outputSet = project.child().getOutputSet();
-if
(!topN.getOrderKeys().stream().map(OrderKey::getExpr).flatMap(e ->
e.getInputSlots().stream())
-.allMatch(outputSet::contains)) {
-return null;
+
+Map slotMap =
ExpressionUtils.generateReplaceMap(project.getProjects());
+List newOrderKeys = new ArrayList<>();
+for (OrderKey orderKey : topN.getOrderKeys()) {
+
(doris) branch branch-2.1 updated: [test](Nereids): remove test infer_set_operator_distinct(CBO) (#32547)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a commit to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-2.1 by this push: new bc279f0ce7b [test](Nereids): remove test infer_set_operator_distinct(CBO) (#32547) bc279f0ce7b is described below commit bc279f0ce7b5232a62172aee65cb35b820b38a76 Author: jakevin AuthorDate: Thu Mar 21 16:11:44 2024 +0800 [test](Nereids): remove test infer_set_operator_distinct(CBO) (#32547) --- .../infer_set_operator_distinct.out| 372 - .../infer_set_operator_distinct.groovy | 172 -- 2 files changed, 544 deletions(-) diff --git a/regression-test/data/nereids_rules_p0/infer_set_operator_distinct/infer_set_operator_distinct.out b/regression-test/data/nereids_rules_p0/infer_set_operator_distinct/infer_set_operator_distinct.out deleted file mode 100644 index 786072eae0d..000 --- a/regression-test/data/nereids_rules_p0/infer_set_operator_distinct/infer_set_operator_distinct.out +++ /dev/null @@ -1,372 +0,0 @@ --- This file is automatically generated. You should know what you did if you want to edit this --- !union_distinct -- -PhysicalResultSink ---PhysicalDistribute[DistributionSpecGather] -hashAgg[GLOBAL] ---PhysicalDistribute[DistributionSpecHash] -hashAgg[LOCAL] ---PhysicalUnion -PhysicalDistribute[DistributionSpecExecutionAny] ---PhysicalProject -PhysicalOlapScan[t1] -PhysicalDistribute[DistributionSpecExecutionAny] ---PhysicalProject -PhysicalOlapScan[t2] - --- !union_complex_conditions -- -PhysicalResultSink ---PhysicalDistribute[DistributionSpecGather] -hashAgg[GLOBAL] ---PhysicalDistribute[DistributionSpecHash] -hashAgg[LOCAL] ---PhysicalUnion -PhysicalDistribute[DistributionSpecExecutionAny] ---PhysicalProject -filter((t1.score > 10)) ---PhysicalOlapScan[t1] -PhysicalDistribute[DistributionSpecExecutionAny] ---PhysicalProject -filter((t2.name = 'Test')) ---PhysicalOlapScan[t2] - --- !multi_union -- -PhysicalResultSink ---PhysicalDistribute[DistributionSpecGather] -hashAgg[GLOBAL] ---PhysicalDistribute[DistributionSpecHash] -hashAgg[LOCAL] ---PhysicalUnion -PhysicalDistribute[DistributionSpecExecutionAny] ---PhysicalProject -PhysicalOlapScan[t1] -PhysicalDistribute[DistributionSpecExecutionAny] ---PhysicalProject -PhysicalOlapScan[t2] -PhysicalDistribute[DistributionSpecExecutionAny] ---PhysicalProject -PhysicalOlapScan[t3] - --- !except_distinct -- -PhysicalResultSink ---PhysicalDistribute[DistributionSpecGather] -PhysicalExcept ---PhysicalDistribute[DistributionSpecHash] -PhysicalProject ---PhysicalOlapScan[t1] ---PhysicalDistribute[DistributionSpecHash] -PhysicalProject ---PhysicalOlapScan[t2] - --- !except_with_filter -- -PhysicalResultSink ---PhysicalDistribute[DistributionSpecGather] -PhysicalExcept ---PhysicalDistribute[DistributionSpecHash] -PhysicalProject ---filter((t1.id > 100)) -PhysicalOlapScan[t1] ---PhysicalDistribute[DistributionSpecHash] -PhysicalProject ---filter((t2.id < 50)) -PhysicalOlapScan[t2] - --- !intersect_distinct -- -PhysicalResultSink ---PhysicalDistribute[DistributionSpecGather] -PhysicalIntersect ---PhysicalDistribute[DistributionSpecHash] -PhysicalProject ---PhysicalOlapScan[t1] ---PhysicalDistribute[DistributionSpecHash] -PhysicalProject ---PhysicalOlapScan[t2] - --- !intersect_with_aggregate -- -PhysicalResultSink ---PhysicalDistribute[DistributionSpecGather] -PhysicalIntersect ---PhysicalDistribute[DistributionSpecHash] -PhysicalProject ---hashAgg[GLOBAL] -PhysicalDistribute[DistributionSpecGather] ---hashAgg[LOCAL] -PhysicalProject ---PhysicalOlapScan[t1] ---PhysicalDistribute[DistributionSpecHash] -PhysicalProject ---hashAgg[GLOBAL] -PhysicalDistribute[DistributionSpecGather] ---hashAgg[LOCAL] -PhysicalProject ---PhysicalOlapScan[t2] - --- !mixed_set_operators -- -PhysicalResultSink ---PhysicalDistribute[DistributionSpecGather] -PhysicalIntersect ---PhysicalExcept -hashAgg[GLOBAL] ---PhysicalDistribute[DistributionSpecHash] -hashAgg[LOCAL] ---PhysicalUnion -PhysicalDistribute[DistributionSpecExecutionAny] ---PhysicalProject -
(doris) branch master updated (99680e9ec13 -> ca092139a00)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 99680e9ec13 [Bug](top-n) do not update topn filter when sort node and scan node are not in the… (#32159) add ca092139a00 [fix](Nereids): slot set in condition can be empty (#32169) No new revisions were added by this update. Summary of changes: .../doris/nereids/rules/exploration/join/InnerJoinLAsscom.java | 6 +++--- .../nereids/rules/rewrite/TransposeSemiJoinLogicalJoin.java| 2 +- .../rules/rewrite/TransposeSemiJoinLogicalJoinProject.java | 10 +- 3 files changed, 9 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (684766e8ce0 -> 7786f95682a)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 684766e8ce0 [Enhancement](jdbc catalog) Add a property to test the connection when creating a Jdbc catalog (#32125) add 7786f95682a [fix](Nereids): ignore project and distribute in test_cte_filter_pushdown and push_down_expression_in_hash_join (#32083) No new revisions were added by this update. Summary of changes: .../nereids_p0/cte/test_cte_filter_pushdown.out| 69 - .../cte/test_cte_filter_pushdown.out | 69 - .../push_down_expression_in_hash_join.out | 158 ++--- .../nereids_p0/cte/test_cte_filter_pushdown.groovy | 1 + .../cte/test_cte_filter_pushdown.groovy| 1 + .../push_down_expression_in_hash_join.groovy | 1 + 6 files changed, 95 insertions(+), 204 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (6c6466b465d -> 5242a80d2bf)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 6c6466b465d [release](build script) fix build release bug when build multiple version on same machine #32104 add 5242a80d2bf [feat](Nereids): use table map to construct struct info (#32058) No new revisions were added by this update. Summary of changes: .../apache/doris/nereids/memo/StructInfoMap.java | 46 - .../doris/nereids/memo/StructInfoMapTest.java | 48 ++ .../apache/doris/utframe/TestWithFeService.java| 25 +++ 3 files changed, 108 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [feat](Nereids): add struct info map in group (#31800)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new dbccbe0fbba [feat](Nereids): add struct info map in group (#31800)
dbccbe0fbba is described below
commit dbccbe0fbbaaa6550eee7b3234c15910615c63a5
Author: 谢健
AuthorDate: Wed Mar 6 16:49:04 2024 +0800
[feat](Nereids): add struct info map in group (#31800)
---
.../org/apache/doris/mtmv/MTMVRelationManager.java | 10 +-
.../java/org/apache/doris/nereids/memo/Group.java | 7 +-
.../apache/doris/nereids/memo/StructInfoMap.java | 123 +
.../mv/InitMaterializationContextHook.java | 7 +-
.../doris/nereids/memo/StructInfoMapTest.java | 81 ++
.../org/apache/doris/nereids/util/PlanChecker.java | 2 +
.../apache/doris/utframe/TestWithFeService.java| 28 +
7 files changed, 252 insertions(+), 6 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java
index aa7ffd2426d..723deaff740 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java
@@ -35,6 +35,7 @@ import
org.apache.doris.nereids.trees.plans.commands.info.TableNameInfo;
import org.apache.doris.persist.AlterMTMV;
import org.apache.doris.qe.ConnectContext;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Maps;
import com.google.common.collect.Sets;
@@ -71,8 +72,7 @@ public class MTMVRelationManager implements MTMVHookService {
for (BaseTableInfo tableInfo : mvInfos) {
try {
MTMV mtmv = (MTMV) MTMVUtil.getTable(tableInfo);
-if (!CollectionUtils
-
.isEmpty(MTMVRewriteUtil.getMTMVCanRewritePartitions(mtmv, ctx,
System.currentTimeMillis( {
+if (isMVPartitionValid(mtmv, ctx)) {
res.add(mtmv);
}
} catch (AnalysisException e) {
@@ -83,6 +83,12 @@ public class MTMVRelationManager implements MTMVHookService {
return res;
}
+@VisibleForTesting
+public boolean isMVPartitionValid(MTMV mtmv, ConnectContext ctx) {
+return !CollectionUtils
+.isEmpty(MTMVRewriteUtil.getMTMVCanRewritePartitions(mtmv,
ctx, System.currentTimeMillis()));
+}
+
private Set getMTMVInfos(List tableInfos) {
Set mvInfos = Sets.newHashSet();
for (BaseTableInfo tableInfo : tableInfos) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java
index 2a0e83c72a4..5a5abd56f95 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java
@@ -19,7 +19,6 @@ package org.apache.doris.nereids.memo;
import org.apache.doris.common.Pair;
import org.apache.doris.nereids.cost.Cost;
-import org.apache.doris.nereids.jobs.joinorder.hypergraph.HyperGraph;
import org.apache.doris.nereids.properties.LogicalProperties;
import org.apache.doris.nereids.properties.PhysicalProperties;
import org.apache.doris.nereids.rules.exploration.mv.StructInfo;
@@ -78,6 +77,8 @@ public class Group {
private List structInfos = new ArrayList<>();
+private StructInfoMap structInfoMap = new StructInfoMap();
+
/**
* Constructor for Group.
*
@@ -418,8 +419,8 @@ public class Group {
return false;
}
-public List getHyperGraphs() {
-return new ArrayList<>();
+public StructInfoMap getstructInfoMap() {
+return structInfoMap;
}
public boolean isProjectGroup() {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/StructInfoMap.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/StructInfoMap.java
new file mode 100644
index 000..f7e09fd44d9
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/StructInfoMap.java
@@ -0,0 +1,123 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WIT
(doris) branch branch-2.0 updated: [fix](Nereids): OuterJoinAssoc reject literal condition (#31172) (#31291)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 7154fdfd1a7 [fix](Nereids): OuterJoinAssoc reject literal condition
(#31172) (#31291)
7154fdfd1a7 is described below
commit 7154fdfd1a7b8eca0e3fb50f387e537336c118f4
Author: jakevin
AuthorDate: Thu Feb 22 20:31:41 2024 +0800
[fix](Nereids): OuterJoinAssoc reject literal condition (#31172) (#31291)
(cherry picked from commit e0e3c3873d15de9a92840af9629c094381824588)
---
.../org/apache/doris/nereids/rules/exploration/join/OuterJoinAssoc.java | 2 +-
.../doris/nereids/rules/exploration/join/OuterJoinAssocProject.java | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssoc.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssoc.java
index 2080cfce93d..3208654fa78 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssoc.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssoc.java
@@ -79,7 +79,7 @@ public class OuterJoinAssoc extends OneExplorationRuleFactory
{
.addAll(topJoin.getOtherJoinConjuncts()).build();
Set notNullSlots =
ExpressionUtils.inferNotNullSlots(on,
ctx.cascadesContext);
-if (!conditionSlot.equals(notNullSlots)) {
+if (conditionSlot.isEmpty() ||
!conditionSlot.equals(notNullSlots)) {
return null;
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssocProject.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssocProject.java
index cbbd72b5c87..96dadc2135d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssocProject.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/OuterJoinAssocProject.java
@@ -80,7 +80,7 @@ public class OuterJoinAssocProject extends
OneExplorationRuleFactory {
.addAll(topJoin.getOtherJoinConjuncts()).build();
Set notNullSlots =
ExpressionUtils.inferNotNullSlots(on,
ctx.cascadesContext);
-if (!conditionSlot.equals(notNullSlots)) {
+if (conditionSlot.isEmpty() ||
!conditionSlot.equals(notNullSlots)) {
return null;
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (2d2e9e661cb -> e0e3c3873d1)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 2d2e9e661cb [fix](stream-load) print stream load profile for pipeline and pipelinex (#31198) add e0e3c3873d1 [fix](Nereids): OuterJoinAssoc reject literal condition (#31172) No new revisions were added by this update. Summary of changes: .../org/apache/doris/nereids/rules/exploration/join/OuterJoinAssoc.java | 2 +- .../doris/nereids/rules/exploration/join/OuterJoinAssocProject.java | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](Nereids): log constraint editlog in lock scope (#30630)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c76a1401282 [fix](Nereids): log constraint editlog in lock scope
(#30630)
c76a1401282 is described below
commit c76a14012824c7d81c467c2f41267306ce1ba7fa
Author: 谢健
AuthorDate: Tue Feb 20 14:39:39 2024 +0800
[fix](Nereids): log constraint editlog in lock scope (#30630)
---
.../java/org/apache/doris/catalog/TableIf.java | 113 +
.../trees/plans/commands/AddConstraintCommand.java | 13 +--
.../plans/commands/DropConstraintCommand.java | 5 +-
.../java/org/apache/doris/persist/EditLog.java | 2 +-
.../catalog/constraint/ConstraintPersistTest.java | 55 +-
5 files changed, 127 insertions(+), 61 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
index 7c1943c69a6..29c7d6b83e2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
@@ -26,6 +26,7 @@ import org.apache.doris.cluster.ClusterNamespace;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.MetaNotFoundException;
import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.persist.AlterConstraintLog;
import org.apache.doris.statistics.AnalysisInfo;
import org.apache.doris.statistics.BaseAnalysisTask;
import org.apache.doris.statistics.ColumnStatistic;
@@ -227,7 +228,7 @@ public interface TableIf {
}
// Note this function is not thread safe
-default void checkConstraintNotExistence(String name, Constraint
primaryKeyConstraint,
+default void checkConstraintNotExistenceUnsafe(String name, Constraint
primaryKeyConstraint,
Map constraintMap) {
if (constraintMap.containsKey(name)) {
throw new RuntimeException(String.format("Constraint name %s has
existed", name));
@@ -240,87 +241,108 @@ public interface TableIf {
}
}
-default Constraint addUniqueConstraint(String name, ImmutableList
columns) {
+default void addUniqueConstraint(String name, ImmutableList
columns, boolean replay) {
writeLock();
try {
Map constraintMap = getConstraintsMapUnsafe();
UniqueConstraint uniqueConstraint = new UniqueConstraint(name,
ImmutableSet.copyOf(columns));
-checkConstraintNotExistence(name, uniqueConstraint, constraintMap);
+checkConstraintNotExistenceUnsafe(name, uniqueConstraint,
constraintMap);
constraintMap.put(name, uniqueConstraint);
-return uniqueConstraint;
+if (!replay) {
+Env.getCurrentEnv().getEditLog().logAddConstraint(
+new AlterConstraintLog(uniqueConstraint, this));
+}
} finally {
writeUnlock();
}
}
-default Constraint addPrimaryKeyConstraint(String name,
ImmutableList columns) {
+default void addPrimaryKeyConstraint(String name, ImmutableList
columns, boolean replay) {
writeLock();
try {
Map constraintMap = getConstraintsMapUnsafe();
PrimaryKeyConstraint primaryKeyConstraint = new
PrimaryKeyConstraint(name, ImmutableSet.copyOf(columns));
-checkConstraintNotExistence(name, primaryKeyConstraint,
constraintMap);
+checkConstraintNotExistenceUnsafe(name, primaryKeyConstraint,
constraintMap);
constraintMap.put(name, primaryKeyConstraint);
-return primaryKeyConstraint;
+if (!replay) {
+Env.getCurrentEnv().getEditLog().logAddConstraint(
+new AlterConstraintLog(primaryKeyConstraint, this));
+}
} finally {
writeUnlock();
}
}
-default void updatePrimaryKeyForForeignKey(PrimaryKeyConstraint
requirePrimaryKey, TableIf referencedTable) {
-referencedTable.writeLock();
-try {
-Optional primaryKeyConstraint =
referencedTable.getConstraintsMapUnsafe().values().stream()
-.filter(requirePrimaryKey::equals)
-.findFirst();
-if (!primaryKeyConstraint.isPresent()) {
-throw new AnalysisException(String.format(
-"Foreign key constraint requires a primary key
constraint %s in %s",
-requirePrimaryKey.getPrimaryKeyNames(),
referencedTable.getName()));
-}
-((PrimaryKeyConstraint)
(primaryKeyConstraint.get())).addForeignTable(this);
-} finally {
-referencedTable.writeUnlock();
+default PrimaryKeyConstraint tryGetPrimaryKe
(doris) branch master updated: [fix](Nereids): fix wrong case in TransposeSemiJoinLogicalJoinProject (#30874)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f8ab1d6dac9 [fix](Nereids): fix wrong case in
TransposeSemiJoinLogicalJoinProject (#30874)
f8ab1d6dac9 is described below
commit f8ab1d6dac9d425eb7bb3441c7184b987862e8d0
Author: jakevin
AuthorDate: Tue Feb 6 11:18:21 2024 +0800
[fix](Nereids): fix wrong case in TransposeSemiJoinLogicalJoinProject
(#30874)
---
.../rules/rewrite/TransposeSemiJoinLogicalJoinProject.java | 12 ++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/TransposeSemiJoinLogicalJoinProject.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/TransposeSemiJoinLogicalJoinProject.java
index 37b3e027ff7..783dca5f9dc 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/TransposeSemiJoinLogicalJoinProject.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/TransposeSemiJoinLogicalJoinProject.java
@@ -76,7 +76,11 @@ public class TransposeSemiJoinLogicalJoinProject extends
OneRewriteRuleFactory {
* A B AC
*/
// RIGHT_OUTER_JOIN should be eliminated in rewrite
phase
-Preconditions.checkState(bottomJoin.getJoinType() !=
JoinType.RIGHT_OUTER_JOIN);
+// TODO: when top join is ANTI JOIN, bottomJoin may
be RIGHT_OUTER_JOIN
+// Can we also do the transformation?
+if (bottomJoin.getJoinType() ==
JoinType.RIGHT_OUTER_JOIN) {
+return null;
+}
Plan newBottomSemiJoin = topSemiJoin.withChildren(a,
c);
Plan newTopJoin =
bottomJoin.withChildren(newBottomSemiJoin, b);
@@ -92,7 +96,11 @@ public class TransposeSemiJoinLogicalJoinProject extends
OneRewriteRuleFactory {
* A B B C
*/
// LEFT_OUTER_JOIN should be eliminated in rewrite
phase
-Preconditions.checkState(bottomJoin.getJoinType() !=
JoinType.LEFT_OUTER_JOIN);
+// TODO: when top join is ANTI JOIN, bottomJoin may
be RIGHT_OUTER_JOIN
+// Can we also do the transformation?
+if (bottomJoin.getJoinType() ==
JoinType.LEFT_OUTER_JOIN) {
+return null;
+}
Plan newBottomSemiJoin = topSemiJoin.withChildren(b,
c);
Plan newTopJoin = bottomJoin.withChildren(a,
newBottomSemiJoin);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [enhance](Nereids): support more DateLiteral (#30876)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 35289ee2449 [enhance](Nereids): support more DateLiteral (#30876)
35289ee2449 is described below
commit 35289ee2449a10d01b528d071c1821753db86da3
Author: jakevin
AuthorDate: Mon Feb 5 23:26:33 2024 +0800
[enhance](Nereids): support more DateLiteral (#30876)
* [enhance](Nereids): support DateLiteral with suffix space (#30583)
(cherry picked from commit ab5bddba39c1e8fc0b07255b85dfbbc492d9ce80)
* [enhancement](Nereids): datetime support microsecond overflow (#30744)
(cherry picked from commit 3c6206355b79084927f1fe0117770988312c0683)
---
.../trees/expressions/literal/DateLiteral.java | 17 +++
.../trees/expressions/literal/DateTimeLiteral.java | 16 +-
.../expressions/literal/DateTimeV2Literal.java | 9 --
.../apache/doris/nereids/types/DateTimeV2Type.java | 5 +++-
.../doris/nereids/util/DateTimeFormatterUtils.java | 5 ++--
.../trees/expressions/literal/DateLiteralTest.java | 16 +-
.../expressions/literal/DateTimeLiteralTest.java | 34 ++
.../nereids/util/DateTimeFormatterUtilsTest.java | 4 +--
8 files changed, 79 insertions(+), 27 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java
index 9fbece96756..58c252b4b98 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java
@@ -36,6 +36,7 @@ import java.time.Year;
import java.time.temporal.ChronoField;
import java.time.temporal.TemporalAccessor;
import java.util.Set;
+import java.util.function.UnaryOperator;
/**
* Date literal in Nereids.
@@ -100,7 +101,7 @@ public class DateLiteral extends Literal {
// normalize yymmdd -> mmdd
static String normalizeBasic(String s) {
-java.util.function.UnaryOperator normalizeTwoDigit = (input)
-> {
+UnaryOperator normalizeTwoDigit = (input) -> {
String yy = input.substring(0, 2);
int year = Integer.parseInt(yy);
if (year >= 0 && year <= 69) {
@@ -182,9 +183,9 @@ public class DateLiteral extends Literal {
// normalize leading 0 for date and time
// date and time contains 6 number part at most, so we just need
normal 6 number part
int partNumber = 0;
-while (i < s.length()) {
+while (i < s.length() && partNumber < 6) {
char c = s.charAt(i);
-if (Character.isDigit(c) && partNumber < 6) {
+if (Character.isDigit(c)) {
// find consecutive digit
int j = i + 1;
while (j < s.length() && Character.isDigit(s.charAt(j))) {
@@ -233,11 +234,14 @@ public class DateLiteral extends Literal {
}
// parse MicroSecond
+// Keep up to 7 digits at most, 7th digit is use for overflow.
if (partNumber == 6 && i < s.length() && s.charAt(i) == '.') {
sb.append(s.charAt(i));
i += 1;
while (i < s.length() && Character.isDigit(s.charAt(i))) {
-sb.append(s.charAt(i));
+if (i - 19 <= 7) {
+sb.append(s.charAt(i));
+}
i += 1;
}
}
@@ -265,9 +269,12 @@ public class DateLiteral extends Literal {
try {
TemporalAccessor dateTime;
+// remove suffix/prefix ' '
+s = s.trim();
// parse condition without '-' and ':'
boolean containsPunctuation = false;
-for (int i = 0; i < s.length(); i++) {
+int len = Math.min(s.length(), 11);
+for (int i = 0; i < len; i++) {
if (isPunctuation(s.charAt(i))) {
containsPunctuation = true;
break;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteral.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteral.java
index 36131db238f..3f96ef52e68 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteral.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteral.java
@@ -136,7 +136,6 @@ public class DateTimeLiteral extends DateLiteral {
hour = DateUtils.getOrDefault(temporal, Chron
(doris) branch master updated (f4803e6602e -> 3c6206355b7)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from f4803e6602e [test](Nereids): add push_down_distinct_through_join regression test (#30760) add 3c6206355b7 [enhancement](Nereids): datetime support microsecond overflow (#30744) No new revisions were added by this update. Summary of changes: .../trees/expressions/literal/DateLiteral.java | 14 + .../trees/expressions/literal/DateTimeLiteral.java | 16 +- .../expressions/literal/DateTimeV2Literal.java | 9 -- .../apache/doris/nereids/types/DateTimeV2Type.java | 5 +++- .../doris/nereids/util/DateTimeFormatterUtils.java | 5 ++-- .../rules/SimplifyComparisonPredicateSqlTest.java | 9 -- .../trees/expressions/literal/DateLiteralTest.java | 10 +-- .../expressions/literal/DateTimeLiteralTest.java | 34 ++ .../nereids/util/DateTimeFormatterUtilsTest.java | 4 +-- .../data/nereids_syntax_p0/test_literal.out| 22 ++ .../suites/nereids_syntax_p0/test_literal.groovy | 28 ++ 11 files changed, 120 insertions(+), 36 deletions(-) create mode 100644 regression-test/data/nereids_syntax_p0/test_literal.out - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [test](Nereids): add push_down_distinct_through_join regression test (#30760)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f4803e6602e [test](Nereids): add push_down_distinct_through_join
regression test (#30760)
f4803e6602e is described below
commit f4803e6602ed6afa144afcbb0282f7f4fadbb976
Author: jakevin
AuthorDate: Fri Feb 2 23:04:22 2024 +0800
[test](Nereids): add push_down_distinct_through_join regression test
(#30760)
---
.../push_down_distinct_through_join.out| 42 +
.../push_down_distinct_through_join.groovy | 70 ++
2 files changed, 112 insertions(+)
diff --git
a/regression-test/data/nereids_rules_p0/push_down_distinct_through_join/push_down_distinct_through_join.out
b/regression-test/data/nereids_rules_p0/push_down_distinct_through_join/push_down_distinct_through_join.out
new file mode 100644
index 000..1e6ea0878c7
--- /dev/null
+++
b/regression-test/data/nereids_rules_p0/push_down_distinct_through_join/push_down_distinct_through_join.out
@@ -0,0 +1,42 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !basic_not --
+PhysicalResultSink
+--hashAgg[LOCAL]
+hashJoin[INNER_JOIN] hashCondition=((t1.id = t2.id)) otherCondition=()
+--PhysicalOlapScan[t]
+--PhysicalOlapScan[t]
+
+-- !basic --
+PhysicalResultSink
+--hashAgg[LOCAL]
+hashJoin[INNER_JOIN] hashCondition=((t1.id = t3.id)) otherCondition=()
+--hashAgg[LOCAL]
+hashJoin[INNER_JOIN] hashCondition=((t1.id = t2.id)) otherCondition=()
+--PhysicalOlapScan[t]
+--PhysicalOlapScan[t]
+--PhysicalOlapScan[t]
+
+-- !basic_sql --
+1
+10
+2
+3
+4
+5
+6
+7
+8
+9
+
+-- !basic_sql_disable --
+1
+10
+2
+3
+4
+5
+6
+7
+8
+9
+
diff --git
a/regression-test/suites/nereids_rules_p0/push_down_distinct_through_join/push_down_distinct_through_join.groovy
b/regression-test/suites/nereids_rules_p0/push_down_distinct_through_join/push_down_distinct_through_join.groovy
new file mode 100644
index 000..1903f99e7a2
--- /dev/null
+++
b/regression-test/suites/nereids_rules_p0/push_down_distinct_through_join/push_down_distinct_through_join.groovy
@@ -0,0 +1,70 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("push_down_distinct_through_join") {
+sql "SET enable_nereids_planner=true"
+sql "set runtime_filter_mode=OFF"
+sql "SET enable_fallback_to_original_planner=false"
+sql "SET ignore_shape_nodes='PhysicalDistribute,PhysicalProject'"
+sql "SET disable_join_reorder=true"
+
+sql """
+DROP TABLE IF EXISTS t;
+"""
+
+sql """
+CREATE TABLE IF NOT EXISTS t(
+ `id` int(32),
+ `score` int(64) NULL,
+ `name` varchar(64) NULL
+) ENGINE = OLAP
+DISTRIBUTED BY HASH(id) BUCKETS 4
+PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+);
+"""
+
+sql "insert into t values (1, 1, 'a')"
+sql "insert into t values (2, null, 'a')"
+sql "insert into t values (3, 1, null)"
+sql "insert into t values (4, 2, 'b')"
+sql "insert into t values (5, null, 'b')"
+sql "insert into t values (6, 2, null)"
+sql "insert into t values (7, 3, 'c')"
+sql "insert into t values (8, null, 'c')"
+sql "insert into t values (9, 3, null)"
+sql "insert into t values (10, null, null)"
+
+sql "SET ENABLE_NEREIDS_RULES=push_down_distinct_through_join"
+
+qt_basic_not """
+explain shape plan select distinct * from (select t1.id from t t1 join
t t2 on t1.id = t2.id) t;
+"""
+
+qt_basic """
+explain shape plan select distinct * from (select t1.id from t t1 join
t t2 on t1.id = t2.id join t t3 on t1.i
(doris) branch branch-2.0 updated: [fix](Nereids): add def in test (#29796) (#30685)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 49fa9fd1c7f [fix](Nereids): add def in test (#29796) (#30685)
49fa9fd1c7f is described below
commit 49fa9fd1c7fab11c8203e23c9e11d4da3c7c1658
Author: jakevin
AuthorDate: Thu Feb 1 16:01:52 2024 +0800
[fix](Nereids): add def in test (#29796) (#30685)
---
.../suites/nereids_p0/expression/case_when_to_if.groovy| 10 +-
.../suites/nereids_p0/expression/topn_to_max.groovy| 6 +++---
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git
a/regression-test/suites/nereids_p0/expression/case_when_to_if.groovy
b/regression-test/suites/nereids_p0/expression/case_when_to_if.groovy
index 8c41b8d1c67..c83ce27c017 100644
--- a/regression-test/suites/nereids_p0/expression/case_when_to_if.groovy
+++ b/regression-test/suites/nereids_p0/expression/case_when_to_if.groovy
@@ -30,7 +30,7 @@ suite("test_case_when_to_if") {
from test_case_when_to_if
group by k2;
'''
-res = sql '''
+def res = sql '''
explain rewritten plan select k2,
sum(case when (k1=1) then 1 end) sum1
from test_case_when_to_if
@@ -45,13 +45,13 @@ suite("test_case_when_to_if") {
from test_case_when_to_if
group by k2;
'''
-res = sql '''
+def res1 = sql '''
explain rewritten plan select k2,
sum(case when (k1=1) then 1 else null end) sum1
from test_case_when_to_if
group by k2;
'''
-assertTrue(res.toString().contains("if"))
+assertTrue(res1.toString().contains("if"))
sql '''
select k2,
@@ -59,11 +59,11 @@ suite("test_case_when_to_if") {
from test_case_when_to_if
group by k2;
'''
-res = sql '''
+def res2 = sql '''
explain rewritten plan select k2,
sum(case when (k1>0) then k1 else abs(k1) end) sum1
from test_case_when_to_if
group by k2;
'''
-assertTrue(res.toString().contains("if"))
+assertTrue(res2.toString().contains("if"))
}
diff --git a/regression-test/suites/nereids_p0/expression/topn_to_max.groovy
b/regression-test/suites/nereids_p0/expression/topn_to_max.groovy
index ae848b5a244..60c603e7ef6 100644
--- a/regression-test/suites/nereids_p0/expression/topn_to_max.groovy
+++ b/regression-test/suites/nereids_p0/expression/topn_to_max.groovy
@@ -41,9 +41,9 @@ suite("test_topn_to_max") {
select topn(k2, 1)
from test_topn_to_max;
'''
-res = sql '''
-explain rewritten plan select max(k2)
+def res1 = sql '''
+explain rewritten plan select topn(k2, 1)
from test_topn_to_max;
'''
-assertTrue(res.toString().contains("max"))
+assertTrue(res1.toString().contains("max"), res1.toString() + " should
contain max")
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (9692f31715e -> be44013890e)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 9692f31715e [doc](partial update) update plan of flexible column updates (#30682) add be44013890e [fix](Nereids) add logical project to prevent extra wrong column (#30459) No new revisions were added by this update. Summary of changes: .../apache/doris/nereids/jobs/executor/Analyzer.java | 4 +++- .../nereids/rules/analysis/SubqueryToApply.java | 10 ++ .../sub_query_diff_old_optimize.out | 3 +++ .../sub_query_diff_old_optimize.groovy | 20 4 files changed, 28 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated (0d619fb0f4e -> 77042235c96)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a change to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
from 0d619fb0f4e Sync stats cache while task finished, doesn't need to
query column_statistics table. (#30609) (#30666)
add 77042235c96 [fix](Nereids): fix offset in PlanTranslator (#29789)
(#30684)
No new revisions were added by this update.
Summary of changes:
.../glue/translator/PhysicalPlanTranslator.java | 21 +
...sProcessor.java => AddOffsetIntoDistribute.java} | 21 ++---
.../nereids/processor/post/PlanPostProcessors.java | 1 +
.../doris/nereids/properties/DistributionSpec.java | 1 -
.../trees/plans/physical/PhysicalDistribute.java| 4 ++--
.../nereids/trees/plans/physical/PhysicalLimit.java | 5 +
.../org/apache/doris/nereids/util/PlanChecker.java | 1 -
7 files changed, 35 insertions(+), 19 deletions(-)
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/{RecomputeLogicalPropertiesProcessor.java
=> AddOffsetIntoDistribute.java} (56%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](Nereids): fix wrong regression test (#30520)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a04fd237fae [fix](Nereids): fix wrong regression test (#30520)
a04fd237fae is described below
commit a04fd237fae2a95baf47dac8c5d5cd6f5b1ff799
Author: jakevin
AuthorDate: Tue Jan 30 14:26:25 2024 +0800
[fix](Nereids): fix wrong regression test (#30520)
---
.../push_down_top_n/push_down_top_n_distinct_through_join.out | 2 +-
.../push_down_top_n/push_down_top_n_distinct_through_join.groovy | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git
a/regression-test/data/nereids_rules_p0/push_down_top_n/push_down_top_n_distinct_through_join.out
b/regression-test/data/nereids_rules_p0/push_down_top_n/push_down_top_n_distinct_through_join.out
index 72452ed3c14..5c3ec397a99 100644
---
a/regression-test/data/nereids_rules_p0/push_down_top_n/push_down_top_n_distinct_through_join.out
+++
b/regression-test/data/nereids_rules_p0/push_down_top_n/push_down_top_n_distinct_through_join.out
@@ -5,7 +5,7 @@ PhysicalResultSink
PhysicalTopN[LOCAL_SORT]
--hashAgg[GLOBAL]
hashAgg[LOCAL]
---NestedLoopJoin[LEFT_OUTER_JOIN](t1.id = t1.id)
+--hashJoin[LEFT_OUTER_JOIN] hashCondition=((t1.id = t2.id))
otherCondition=()
PhysicalTopN[MERGE_SORT]
--PhysicalTopN[LOCAL_SORT]
hashAgg[LOCAL]
diff --git
a/regression-test/suites/nereids_rules_p0/push_down_top_n/push_down_top_n_distinct_through_join.groovy
b/regression-test/suites/nereids_rules_p0/push_down_top_n/push_down_top_n_distinct_through_join.groovy
index 890e2ba897d..3ee3a916e7b 100644
---
a/regression-test/suites/nereids_rules_p0/push_down_top_n/push_down_top_n_distinct_through_join.groovy
+++
b/regression-test/suites/nereids_rules_p0/push_down_top_n/push_down_top_n_distinct_through_join.groovy
@@ -51,11 +51,11 @@ suite("push_down_top_n_distinct_through_join") {
"""
qt_push_down_topn_through_join """
-explain shape plan select distinct * from (select t1.id from
table_join t1 left join table_join t2 on t1.id = t1.id) t order by id limit 10;
+explain shape plan select distinct * from (select t1.id from
table_join t1 left join table_join t2 on t1.id = t2.id) t order by id limit 10;
"""
qt_push_down_topn_through_join_data """
-select distinct * from (select t1.id from table_join t1 left join
table_join t2 on t1.id = t1.id) t order by id limit 10;
+select distinct * from (select t1.id from table_join t1 left join
table_join t2 on t1.id = t2.id) t order by id limit 10;
"""
qt_push_down_topn_through_join """
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [feature](Nereids): InferPredicates support In (#29458) (#30525)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new c028679c134 [feature](Nereids): InferPredicates support In (#29458)
(#30525)
c028679c134 is described below
commit c028679c1340d085c5c239f590f05c8611831012
Author: jakevin
AuthorDate: Tue Jan 30 14:23:29 2024 +0800
[feature](Nereids): InferPredicates support In (#29458) (#30525)
(cherry picked from commit 7a0734dbd60effa676d87bf5a5b7ca516e134d52)
---
.../doris/nereids/jobs/scheduler/SimpleJobScheduler.java | 12
.../src/main/java/org/apache/doris/qe/SessionVariable.java | 3 +++
2 files changed, 11 insertions(+), 4 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/scheduler/SimpleJobScheduler.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/scheduler/SimpleJobScheduler.java
index c52e2e90f13..ec751bdab2d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/scheduler/SimpleJobScheduler.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/scheduler/SimpleJobScheduler.java
@@ -19,6 +19,7 @@ package org.apache.doris.nereids.jobs.scheduler;
import org.apache.doris.nereids.CascadesContext;
import org.apache.doris.nereids.jobs.Job;
+import org.apache.doris.qe.SessionVariable;
import java.util.concurrent.TimeUnit;
@@ -29,11 +30,14 @@ public class SimpleJobScheduler implements JobScheduler {
@Override
public void executeJobPool(ScheduleContext scheduleContext) {
JobPool pool = scheduleContext.getJobPool();
+CascadesContext context = (CascadesContext) scheduleContext;
+SessionVariable sessionVariable =
context.getConnectContext().getSessionVariable();
while (!pool.isEmpty()) {
-CascadesContext context = (CascadesContext) scheduleContext;
-if
(context.getConnectContext().getSessionVariable().enableNereidsTimeout
-&&
context.getStatementContext().getStopwatch().elapsed(TimeUnit.MILLISECONDS) >
5000) {
-throw new RuntimeException("Nereids cost too much time ( > 5s
)");
+if (sessionVariable.enableNereidsTimeout
+&&
context.getStatementContext().getStopwatch().elapsed(TimeUnit.MILLISECONDS)
+> sessionVariable.nereidsTimeoutSecond * 1000L) {
+throw new RuntimeException(
+"Nereids cost too much time ( > " +
sessionVariable.nereidsTimeoutSecond + "s )");
}
Job job = pool.pop();
job.execute();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index 036d8da6836..616368a3778 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@@ -972,6 +972,9 @@ public class SessionVariable implements Serializable,
Writable {
@VariableMgr.VarAttr(name = ENABLE_NEW_SHUFFLE_HASH_METHOD)
public boolean enableNewShuffleHashMethod = true;
+@VariableMgr.VarAttr(name = "nereids_timeout_second", needForward = true)
+public int nereidsTimeoutSecond = 5;
+
@VariableMgr.VarAttr(name = ENABLE_PUSH_DOWN_NO_GROUP_AGG)
public boolean enablePushDownNoGroupAgg = true;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [fix](Nereids): Except just can merge with left deep shape (#30270) (#30524)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 9f8a11c26ad [fix](Nereids): Except just can merge with left deep shape
(#30270) (#30524)
9f8a11c26ad is described below
commit 9f8a11c26add04a582ef541c1a61e7d939fd1234
Author: jakevin
AuthorDate: Tue Jan 30 14:21:59 2024 +0800
[fix](Nereids): Except just can merge with left deep shape (#30270) (#30524)
(cherry picked from commit aaa854f8ba8d31fc6eec219fb9da97bff4128b55)
---
.../doris/nereids/jobs/executor/Rewriter.java | 3 +-
.../org/apache/doris/nereids/rules/RuleType.java | 2 +
.../nereids/rules/rewrite/MergeSetOperations.java | 22 ++-
.../rules/rewrite/MergeSetOperationsExcept.java| 64 +
.../data/nereids_p0/set_operations/except.out | 148 +
.../suites/nereids_p0/set_operations/except.groovy | 126 ++
6 files changed, 358 insertions(+), 7 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
index 1f17d15ae61..881750f23dd 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
@@ -77,6 +77,7 @@ import org.apache.doris.nereids.rules.rewrite.MergeFilters;
import org.apache.doris.nereids.rules.rewrite.MergeOneRowRelationIntoUnion;
import org.apache.doris.nereids.rules.rewrite.MergeProjects;
import org.apache.doris.nereids.rules.rewrite.MergeSetOperations;
+import org.apache.doris.nereids.rules.rewrite.MergeSetOperationsExcept;
import org.apache.doris.nereids.rules.rewrite.NormalizeSort;
import org.apache.doris.nereids.rules.rewrite.OrExpansion;
import org.apache.doris.nereids.rules.rewrite.PruneEmptyPartition;
@@ -253,7 +254,7 @@ public class Rewriter extends AbstractBatchJobExecutor {
topic("Set operation optimization",
// Do MergeSetOperation first because we hope to match
pattern of Distinct SetOperator.
topDown(new PushProjectThroughUnion(), new
MergeProjects()),
-bottomUp(new MergeSetOperations()),
+bottomUp(new MergeSetOperations(), new
MergeSetOperationsExcept()),
bottomUp(new PushProjectIntoOneRowRelation()),
topDown(new MergeOneRowRelationIntoUnion()),
topDown(new BuildAggForUnion())
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
index 3bdcf18fd44..a0c32509a65 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
@@ -236,6 +236,8 @@ public enum RuleType {
MERGE_ONE_ROW_RELATION_INTO_UNION(RuleTypeClass.REWRITE),
PUSH_PROJECT_INTO_ONE_ROW_RELATION(RuleTypeClass.REWRITE),
MERGE_SET_OPERATION(RuleTypeClass.REWRITE),
+MERGE_SET_OPERATION_EXCEPT(RuleTypeClass.REWRITE),
+MERGE_TOP_N(RuleTypeClass.REWRITE),
BUILD_AGG_FOR_UNION(RuleTypeClass.REWRITE),
COUNT_DISTINCT_REWRITE(RuleTypeClass.REWRITE),
INNER_TO_CROSS_JOIN(RuleTypeClass.REWRITE),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MergeSetOperations.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MergeSetOperations.java
index 152409d2b53..33d9d50acaa 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MergeSetOperations.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MergeSetOperations.java
@@ -22,6 +22,7 @@ import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.trees.expressions.SlotReference;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.algebra.SetOperation.Qualifier;
+import org.apache.doris.nereids.trees.plans.logical.LogicalExcept;
import org.apache.doris.nereids.trees.plans.logical.LogicalSetOperation;
import com.google.common.collect.ImmutableList;
@@ -43,12 +44,22 @@ import java.util.List;
* / | \
* scan1 scan2 scan3
*
+ * Notice: this rule ignore Except.
+ * Relational Algebra: Union (R U S), Intersect Syntax: (R ∩ S), Except
Syntax: (R - S)
+ * TODO: Except need other Rewrite.
+ * (R - S) U T = (R U T) - S
+ * (R - S) U (T - U) = (R U T) - (S U U)
+ * R - (S U T) = (R - S) - T
+ * R - (S - T) = (R - S) U T
+ * .. and so on
*/
-public class MergeSetOperations implements RewriteRuleFactory {
+public class MergeSetOperations extends OneRewriteRuleFactory {
@Override
-public List buildRules() {
-
(doris) branch master updated: [feat](Nereids): drop foreign key after dropping primary key that is referenced by the foreign key (#30417)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 2c4f6ce1b96 [feat](Nereids): drop foreign key after dropping primary
key that is referenced by the foreign key (#30417)
2c4f6ce1b96 is described below
commit 2c4f6ce1b960a3d09724c1236e171be8bd209fa1
Author: 谢健
AuthorDate: Mon Jan 29 18:55:44 2024 +0800
[feat](Nereids): drop foreign key after dropping primary key that is
referenced by the foreign key (#30417)
---
.../java/org/apache/doris/catalog/TableIf.java | 10 --
.../catalog/constraint/ForeignKeyConstraint.java | 6 +++-
.../doris/nereids/trees/plans/ConstraintTest.java | 31
.../data/nereids_syntax_p0/constraint.out | 23 +---
.../suites/nereids_syntax_p0/constraint.groovy | 42 +++---
5 files changed, 101 insertions(+), 11 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
index b59aedfdbf3..67efd98fec6 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
@@ -45,9 +45,11 @@ import java.io.IOException;
import java.util.Collections;
import java.util.List;
import java.util.Map;
+import java.util.Map.Entry;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
public interface TableIf {
Logger LOG = LogManager.getLogger(TableIf.class);
@@ -341,8 +343,12 @@ public interface TableIf {
writeLock();
try {
Map constraintMap = getConstraintsMapUnsafe();
-constraintMap.entrySet().removeIf(e -> e.getValue() instanceof
ForeignKeyConstraint
-&& ((ForeignKeyConstraint)
e.getValue()).isReferringPK(table, constraint));
+Set fkName = constraintMap.entrySet().stream()
+.filter(e -> e.getValue() instanceof ForeignKeyConstraint
+&& ((ForeignKeyConstraint)
e.getValue()).isReferringPK(table, constraint))
+.map(Entry::getKey)
+.collect(Collectors.toSet());
+fkName.forEach(constraintMap::remove);
} finally {
writeUnlock();
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/constraint/ForeignKeyConstraint.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/constraint/ForeignKeyConstraint.java
index 66d8e0a3706..69d16c07532 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/catalog/constraint/ForeignKeyConstraint.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/catalog/constraint/ForeignKeyConstraint.java
@@ -58,6 +58,10 @@ public class ForeignKeyConstraint extends Constraint {
return foreignToReference.keySet();
}
+public Set getPrimaryKeyNames() {
+return ImmutableSet.copyOf(foreignToReference.values());
+}
+
public Set getReferencedColumnNames() {
return ImmutableSet.copyOf(foreignToReference.values());
}
@@ -87,7 +91,7 @@ public class ForeignKeyConstraint extends Constraint {
}
public Boolean isReferringPK(TableIf table, PrimaryKeyConstraint
constraint) {
-return constraint.getPrimaryKeyNames().equals(getForeignKeyNames())
+return constraint.getPrimaryKeyNames().equals(getPrimaryKeyNames())
&& getReferencedTable().equals(table);
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/ConstraintTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/ConstraintTest.java
index 6d5600909f3..ea9d3ab30d6 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/ConstraintTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/ConstraintTest.java
@@ -61,6 +61,15 @@ class ConstraintTest extends TestWithFeService implements
PlanPatternMatchSuppor
+ "properties(\n"
+ "\"replication_num\"=\"1\"\n"
+ ")");
+createTable("create table t3 (\n"
++ "k1 int,\n"
++ "k2 int\n"
++ ")\n"
++ "unique key(k1, k2)\n"
++ "distributed by hash(k1) buckets 4\n"
++ "properties(\n"
++ "\"replication_num\"=\"1\"\n"
++ ")");
}
@Test
@@ -162,4 +171,26 @@ class ConstraintTest extends TestWithFeService implements
PlanPatternMatchSuppor
PlanChecker.from(connectContext).parse("select
(doris) branch master updated: [fix](Nereids) should not generate same exprId for diff column when sink (#30501)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 608fede57a7 [fix](Nereids) should not generate same exprId for diff
column when sink (#30501)
608fede57a7 is described below
commit 608fede57a78f77933a7b4ebd80b523cbccc0848
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Mon Jan 29 18:51:49 2024 +0800
[fix](Nereids) should not generate same exprId for diff column when sink
(#30501)
---
.../java/org/apache/doris/nereids/rules/analysis/BindSink.java | 9 -
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java
index 7dbb6b3e1e1..c0e8162f08f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java
@@ -213,7 +213,14 @@ public class BindSink implements AnalysisRuleFactory {
+ " target table " +
table.getName());
}
if (columnToOutput.get(seqCol.get().getName())
!= null) {
-columnToOutput.put(column.getName(),
columnToOutput.get(seqCol.get().getName()));
+// should generate diff exprId for seq
column
+NamedExpression seqColumn =
columnToOutput.get(seqCol.get().getName());
+if (seqColumn instanceof Alias) {
+seqColumn = new Alias(((Alias)
seqColumn).child(), column.getName());
+} else {
+seqColumn = new Alias(seqColumn,
column.getName());
+}
+columnToOutput.put(column.getName(),
seqColumn);
}
} else if (isPartialUpdate) {
// If the current load is a partial update,
the values of unmentioned
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [feature](Nereids): double eager support mix function (#30468)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c583563087b [feature](Nereids): double eager support mix function
(#30468)
c583563087b is described below
commit c583563087bc5a0db9920aa88aafb63a5bd61e19
Author: jakevin
AuthorDate: Mon Jan 29 13:08:09 2024 +0800
[feature](Nereids): double eager support mix function (#30468)
---
.../doris/nereids/jobs/executor/Rewriter.java | 6 +-
.../org/apache/doris/nereids/rules/RuleType.java | 3 +-
...hroughJoin.java => PushDownAggThroughJoin.java} | 107 +--
.../rules/rewrite/PushDownSumThroughJoin.java | 212 -
.../rewrite/PushDownCountThroughJoinTest.java | 13 +-
.../rules/rewrite/PushDownSumThroughJoinTest.java | 29 ++-
.../eager_aggregate/push_down_sum_through_join.out | 12 +-
.../nereids_rules_p0/eager_aggregate/basic.groovy | 3 +-
.../push_down_count_through_join.groovy| 2 +-
.../push_down_sum_through_join.groovy | 4 +-
10 files changed, 101 insertions(+), 290 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
index 2c0e57b715e..34f7afe4995 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
@@ -98,14 +98,13 @@ import
org.apache.doris.nereids.rules.rewrite.PullUpProjectUnderTopN;
import org.apache.doris.nereids.rules.rewrite.PushConjunctsIntoEsScan;
import org.apache.doris.nereids.rules.rewrite.PushConjunctsIntoJdbcScan;
import org.apache.doris.nereids.rules.rewrite.PushConjunctsIntoOdbcScan;
+import org.apache.doris.nereids.rules.rewrite.PushDownAggThroughJoin;
import org.apache.doris.nereids.rules.rewrite.PushDownAggThroughJoinOneSide;
-import org.apache.doris.nereids.rules.rewrite.PushDownCountThroughJoin;
import org.apache.doris.nereids.rules.rewrite.PushDownDistinctThroughJoin;
import org.apache.doris.nereids.rules.rewrite.PushDownFilterThroughProject;
import org.apache.doris.nereids.rules.rewrite.PushDownLimit;
import org.apache.doris.nereids.rules.rewrite.PushDownLimitDistinctThroughJoin;
import
org.apache.doris.nereids.rules.rewrite.PushDownLimitDistinctThroughUnion;
-import org.apache.doris.nereids.rules.rewrite.PushDownSumThroughJoin;
import org.apache.doris.nereids.rules.rewrite.PushDownTopNDistinctThroughJoin;
import org.apache.doris.nereids.rules.rewrite.PushDownTopNDistinctThroughUnion;
import org.apache.doris.nereids.rules.rewrite.PushDownTopNThroughJoin;
@@ -288,9 +287,8 @@ public class Rewriter extends AbstractBatchJobExecutor {
topic("Eager aggregation",
topDown(
-new PushDownSumThroughJoin(),
new PushDownAggThroughJoinOneSide(),
-new PushDownCountThroughJoin()
+new PushDownAggThroughJoin()
),
custom(RuleType.PUSH_DOWN_DISTINCT_THROUGH_JOIN,
PushDownDistinctThroughJoin::new)
),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
index b35c7e03b72..594f49a3b70 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
@@ -167,8 +167,7 @@ public enum RuleType {
ELIMINATE_SORT(RuleTypeClass.REWRITE),
PUSH_DOWN_AGG_THROUGH_JOIN_ONE_SIDE(RuleTypeClass.REWRITE),
-PUSH_DOWN_SUM_THROUGH_JOIN(RuleTypeClass.REWRITE),
-PUSH_DOWN_COUNT_THROUGH_JOIN(RuleTypeClass.REWRITE),
+PUSH_DOWN_AGG_THROUGH_JOIN(RuleTypeClass.REWRITE),
TRANSPOSE_LOGICAL_SEMI_JOIN_LOGICAL_JOIN(RuleTypeClass.REWRITE),
TRANSPOSE_LOGICAL_SEMI_JOIN_LOGICAL_JOIN_PROJECT(RuleTypeClass.REWRITE),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownCountThroughJoin.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownAggThroughJoin.java
similarity index 69%
rename from
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownCountThroughJoin.java
rename to
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownAggThroughJoin.java
index 462180ab7a6..f003d2ac2cc 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownCountThroughJoin.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownAggThroughJoin.java
@@ -67,7 +67,7 @@ import java.util.Set;
*
* Notice: rule can't optimize condition that groupby is empty whe
(doris) branch master updated (35c2a529fac -> 8db0cf35f2d)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 35c2a529fac [fix](Nereids): when predicate contains right output, don't convert outer to anti join (#30276) add 8db0cf35f2d [feat](Nereids): eliminate inner join by pk fk when comparing mv (#30258) No new revisions were added by this update. Summary of changes: .../doris/nereids/jobs/executor/Rewriter.java | 2 +- .../joinorder/hypergraph/node/StructInfoNode.java | 23 -- .../rules/exploration/mv/HyperGraphComparator.java | 53 +++- .../nereids/rules/rewrite/EliminateJoinByFK.java | 319 + .../rules/rewrite/EliminateJoinByUnique.java | 1 - .../nereids/rules/rewrite/ForeignKeyContext.java | 184 .../org/apache/doris/nereids/util/JoinUtils.java | 42 ++- .../rules/exploration/mv/EliminateJoinTest.java| 103 +++ 8 files changed, 443 insertions(+), 284 deletions(-) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ForeignKeyContext.java - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](Nereids): when predicate contains right output, don't convert outer to anti join (#30276)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 35c2a529fac [fix](Nereids): when predicate contains right output,
don't convert outer to anti join (#30276)
35c2a529fac is described below
commit 35c2a529fac1b3aa805cb7fd365565c86f327844
Author: 谢健
AuthorDate: Thu Jan 25 14:01:27 2024 +0800
[fix](Nereids): when predicate contains right output, don't convert outer
to anti join (#30276)
---
.../rules/rewrite/ConvertOuterJoinToAntiJoin.java | 6 +-
.../rewrite/ConvertOuterJoinToAntiJoinTest.java| 82 +-
2 files changed, 84 insertions(+), 4 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConvertOuterJoinToAntiJoin.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConvertOuterJoinToAntiJoin.java
index ebc4630d78e..74bd7e29142 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConvertOuterJoinToAntiJoin.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConvertOuterJoinToAntiJoin.java
@@ -81,7 +81,8 @@ public class ConvertOuterJoinToAntiJoin extends
OneRewriteRuleFactory {
&&
rightAlwaysNullSlots.containsAll(p.getInputSlots(
.collect(ImmutableSet.toImmutableSet());
boolean containRightSlot = predicates.stream()
-.anyMatch(s ->
join.right().getOutputSet().containsAll(s.getInputSlots()));
+.flatMap(p -> p.getInputSlots().stream())
+.anyMatch(join.right().getOutputSet()::contains);
if (!containRightSlot) {
res = join.withJoinType(JoinType.LEFT_ANTI_JOIN);
res = predicates.isEmpty() ? res :
filter.withConjuncts(predicates).withChildren(res);
@@ -94,7 +95,8 @@ public class ConvertOuterJoinToAntiJoin extends
OneRewriteRuleFactory {
&&
leftAlwaysNullSlots.containsAll(p.getInputSlots(
.collect(ImmutableSet.toImmutableSet());
boolean containLeftSlot = predicates.stream()
-.anyMatch(s ->
join.left().getOutputSet().containsAll(s.getInputSlots()));
+.flatMap(p -> p.getInputSlots().stream())
+.anyMatch(join.left().getOutputSet()::contains);
if (!containLeftSlot) {
res = join.withJoinType(JoinType.RIGHT_ANTI_JOIN);
res = predicates.isEmpty() ? res :
filter.withConjuncts(predicates).withChildren(res);
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/ConvertOuterJoinToAntiJoinTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/ConvertOuterJoinToAntiJoinTest.java
index 49960c77ee4..20b36d3272e 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/ConvertOuterJoinToAntiJoinTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/ConvertOuterJoinToAntiJoinTest.java
@@ -18,8 +18,12 @@
package org.apache.doris.nereids.rules.rewrite;
import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
import org.apache.doris.nereids.trees.expressions.IsNull;
+import org.apache.doris.nereids.trees.expressions.Or;
import org.apache.doris.nereids.trees.expressions.StatementScopeIdGenerator;
+import org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral;
import org.apache.doris.nereids.trees.plans.JoinType;
import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
@@ -30,6 +34,7 @@ import org.apache.doris.nereids.util.PlanChecker;
import org.apache.doris.nereids.util.PlanConstructor;
import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Sets;
import org.junit.jupiter.api.Test;
class ConvertOuterJoinToAntiJoinTest implements MemoPatternMatchSupported {
@@ -48,7 +53,7 @@ class ConvertOuterJoinToAntiJoinTest implements
MemoPatternMatchSupported {
LogicalPlan plan = new LogicalPlanBuilder(scan1)
.join(scan2, JoinType.LEFT_OUTER_JOIN, Pair.of(0, 0)) //
t1.id = t2.id
.filter(new IsNull(scan2.getOutput().get(0)))
-.project(ImmutableList.of(0, 1))
+.projectExprs(ImmutableList.copyOf(scan1.getOutput()))
.build();
PlanChecker.from(MemoTestUtils.createConnectContext(), plan)
@@ -63,7 +68,7 @@ class ConvertOuterJoinToAntiJoinTest implements
MemoPatternMatchSupported {
LogicalPlan plan = new LogicalPlanBuilder(scan1)
.join(scan2, JoinType.RIGHT_OUTER_J
(doris) branch master updated (c9969587457 -> cb9cfcf9678)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from c9969587457 [feat](log) Add more log about cost time for `dropTable` (#30284) add cb9cfcf9678 [refactor](cmake) Refactor be CMakeLists ENABLE_CLANG_COVERAGE CMAKE_CXX_FLAGS byadd_compile_options (#30333) No new revisions were added by this update. Summary of changes: be/CMakeLists.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [minor](Nereids): enable PushDownTopNDistinctThroughJoin (#30275)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 05331556f31 [minor](Nereids): enable PushDownTopNDistinctThroughJoin
(#30275)
05331556f31 is described below
commit 05331556f318b549a4cee3e46704eddbd26ea3e1
Author: jakevin
AuthorDate: Wed Jan 24 22:19:28 2024 +0800
[minor](Nereids): enable PushDownTopNDistinctThroughJoin (#30275)
---
.../doris/nereids/jobs/executor/Rewriter.java | 3 +-
.../org/apache/doris/nereids/rules/RuleType.java | 3 +-
.../rewrite/PushDownTopNDistinctThroughUnion.java | 8 +-
.../org/apache/doris/nereids/util/PlanUtils.java | 6 +-
.../push_down_top_n_distinct_through_union.out | 138 -
.../shape/query49.out | 162 +++-
.../noStatsRfPrune/query49.out | 168 -
.../no_stats_shape/query49.out | 168 -
.../rf_prune/query49.out | 162 +++-
.../nereids_tpcds_shape_sf100_p0/shape/query49.out | 162 +++-
.../limit_push_down/order_push_down.groovy | 1 +
11 files changed, 573 insertions(+), 408 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
index c4b2945e7cd..4304933f752 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
@@ -109,6 +109,7 @@ import
org.apache.doris.nereids.rules.rewrite.PushDownMinMaxThroughJoin;
import org.apache.doris.nereids.rules.rewrite.PushDownSumThroughJoin;
import org.apache.doris.nereids.rules.rewrite.PushDownSumThroughJoinOneSide;
import org.apache.doris.nereids.rules.rewrite.PushDownTopNDistinctThroughJoin;
+import org.apache.doris.nereids.rules.rewrite.PushDownTopNDistinctThroughUnion;
import org.apache.doris.nereids.rules.rewrite.PushDownTopNThroughJoin;
import org.apache.doris.nereids.rules.rewrite.PushDownTopNThroughUnion;
import org.apache.doris.nereids.rules.rewrite.PushDownTopNThroughWindow;
@@ -334,7 +335,7 @@ public class Rewriter extends AbstractBatchJobExecutor {
new PushDownLimitDistinctThroughJoin(),
new PushDownLimitDistinctThroughUnion(),
new PushDownTopNDistinctThroughJoin(),
-// new PushDownTopNDistinctThroughUnion(),
+new PushDownTopNDistinctThroughUnion(),
new PushDownTopNThroughJoin(),
new PushDownTopNThroughWindow(),
new PushDownTopNThroughUnion()
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
index 248fec7de3f..29c053a8b34 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
@@ -271,8 +271,7 @@ public enum RuleType {
PUSH_DOWN_TOP_N_THROUGH_PROJECT_WINDOW(RuleTypeClass.REWRITE),
PUSH_DOWN_TOP_N_THROUGH_WINDOW(RuleTypeClass.REWRITE),
PUSH_DOWN_TOP_N_THROUGH_UNION(RuleTypeClass.REWRITE),
-PUSH_DOWN_TOP_N_LIMIT_THROUGH_UNION(RuleTypeClass.REWRITE),
-// limit distinct push down
+PUSH_DOWN_TOP_N_DISTINCT_THROUGH_UNION(RuleTypeClass.REWRITE),
PUSH_DOWN_LIMIT_DISTINCT_THROUGH_JOIN(RuleTypeClass.REWRITE),
PUSH_DOWN_LIMIT_DISTINCT_THROUGH_PROJECT_JOIN(RuleTypeClass.REWRITE),
PUSH_DOWN_LIMIT_DISTINCT_THROUGH_UNION(RuleTypeClass.REWRITE),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNDistinctThroughUnion.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNDistinctThroughUnion.java
index 7563ecd5165..f6e944a7a91 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNDistinctThroughUnion.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNDistinctThroughUnion.java
@@ -28,6 +28,7 @@ import
org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
import org.apache.doris.nereids.trees.plans.logical.LogicalUnion;
import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.PlanUtils;
import com.google.common.collect.ImmutableList;
@@ -73,20 +74,19 @@ public class PushDownTopNDistinctThroughUnion implements
RewriteRuleFactory {
NamedExpression output =
union.getOutputs().get(i
(doris) branch master updated (49c35900c74 -> aaa854f8ba8)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 49c35900c74 [fix](ci) add single quote to the value of the session variables when setting it (#30295) add aaa854f8ba8 [fix](Nereids): Except just can merge with left deep shape (#30270) No new revisions were added by this update. Summary of changes: .../doris/nereids/jobs/executor/Rewriter.java | 3 +- .../org/apache/doris/nereids/rules/RuleType.java | 1 + .../nereids/rules/rewrite/MergeSetOperations.java | 22 ++- .../rules/rewrite/MergeSetOperationsExcept.java| 64 + .../data/nereids_p0/set_operations/except.out | 148 + .../suites/nereids_p0/set_operations/except.groovy | 126 ++ 6 files changed, 357 insertions(+), 7 deletions(-) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MergeSetOperationsExcept.java create mode 100644 regression-test/data/nereids_p0/set_operations/except.out create mode 100644 regression-test/suites/nereids_p0/set_operations/except.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [feat](Nereids): eliminate left outer join by unique when comparing mv (#30228)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 7d9300947b3 [feat](Nereids): eliminate left outer join by unique when
comparing mv (#30228)
7d9300947b3 is described below
commit 7d9300947b3f3b9dd6fcbc91641353c89dd81a13
Author: 谢健
AuthorDate: Wed Jan 24 14:05:38 2024 +0800
[feat](Nereids): eliminate left outer join by unique when comparing mv
(#30228)
---
.../joinorder/hypergraph/bitmap/LongBitmap.java| 9 ++
.../joinorder/hypergraph/node/AbstractNode.java| 40 +++--
.../joinorder/hypergraph/node/DPhyperNode.java | 2 +-
.../joinorder/hypergraph/node/StructInfoNode.java | 23 +
.../rules/exploration/mv/HyperGraphComparator.java | 34 ++--
.../rules/rewrite/EliminateJoinByUnique.java | 11 ++-
.../nereids/trees/plans/logical/LogicalJoin.java | 7 +-
.../org/apache/doris/nereids/util/JoinUtils.java | 17
.../rules/exploration/mv/EliminateJoinTest.java| 97 ++
9 files changed, 221 insertions(+), 19 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/bitmap/LongBitmap.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/bitmap/LongBitmap.java
index c50509b2389..fabd8c975af 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/bitmap/LongBitmap.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/bitmap/LongBitmap.java
@@ -20,6 +20,7 @@ package
org.apache.doris.nereids.jobs.joinorder.hypergraph.bitmap;
import org.apache.doris.nereids.trees.plans.RelationId;
import java.util.BitSet;
+import java.util.Collection;
import java.util.Set;
/**
@@ -49,6 +50,14 @@ public class LongBitmap {
return 0;
}
+public static long newBitmap(Collection values) {
+long res = 0;
+for (int v : values) {
+res = LongBitmap.set(res, v);
+}
+return res;
+}
+
public static long clone(long bitmap) {
return bitmap;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/node/AbstractNode.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/node/AbstractNode.java
index 16ee7340876..a4a64e0449d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/node/AbstractNode.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/node/AbstractNode.java
@@ -19,9 +19,14 @@ package
org.apache.doris.nereids.jobs.joinorder.hypergraph.node;
import org.apache.doris.nereids.jobs.joinorder.hypergraph.bitmap.LongBitmap;
import org.apache.doris.nereids.jobs.joinorder.hypergraph.edge.Edge;
+import org.apache.doris.nereids.jobs.joinorder.hypergraph.edge.FilterEdge;
+import org.apache.doris.nereids.jobs.joinorder.hypergraph.edge.JoinEdge;
import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.plans.Plan;
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
import java.util.List;
import java.util.Set;
@@ -30,17 +35,34 @@ import java.util.Set;
*/
public class AbstractNode {
protected final int index;
-protected final List edges;
+protected final List joinEdges;
+protected final List filterEdges;
protected final Plan plan;
protected AbstractNode(Plan plan, int index, List edges) {
this.index = index;
-this.edges = edges;
+this.joinEdges = new ArrayList<>();
+this.filterEdges = new ArrayList<>();
this.plan = plan;
+edges.forEach(e -> {
+if (e instanceof JoinEdge) {
+joinEdges.add((JoinEdge) e);
+} else if (e instanceof FilterEdge) {
+filterEdges.add((FilterEdge) e);
+}
+});
+}
+
+public List getJoinEdges() {
+return ImmutableList.copyOf(joinEdges);
}
public List getEdges() {
-return edges;
+return ImmutableList
+.builder()
+.addAll(joinEdges)
+.addAll(filterEdges)
+.build();
}
public int getIndex() {
@@ -61,7 +83,11 @@ public class AbstractNode {
* @param edge the edge that references this node
*/
public void attachEdge(Edge edge) {
-edges.add(edge);
+if (edge instanceof JoinEdge) {
+joinEdges.add((JoinEdge) edge);
+} else if (edge instanceof FilterEdge) {
+filterEdges.add((FilterEdge) edge);
+}
}
/**
@@ -70,7 +96,11 @@ public class AbstractNode {
* @param edge The edge should be removed
*/
public void removeEdge(Edge edge) {
-edges.remove(edg
(doris) branch master updated: [feature](Planner): Push down TopNDistinct through Join (#30216)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 62b5210b59e [feature](Planner): Push down TopNDistinct through Join
(#30216)
62b5210b59e is described below
commit 62b5210b59ea7cf12b5a2ab2912f8d21b5f182a9
Author: jakevin
AuthorDate: Tue Jan 23 16:46:35 2024 +0800
[feature](Planner): Push down TopNDistinct through Join (#30216)
Push down TopNDistinct through Outer/Cross Join
---
.../doris/nereids/jobs/executor/Rewriter.java | 5 +-
.../org/apache/doris/nereids/rules/RuleType.java | 2 +
.../rewrite/PushDownTopNDistinctThroughJoin.java | 126 +
.../org/apache/doris/nereids/util/PlanUtils.java | 6 +
.../limit_push_down/order_push_down.out| 22 +++-
.../push_down_top_n_distinct_through_join.out | 47
.../push_down_top_n_distinct_through_join.groovy | 68 +++
7 files changed, 271 insertions(+), 5 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
index b2f1569ea1d..1704c629042 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
@@ -107,6 +107,7 @@ import
org.apache.doris.nereids.rules.rewrite.PushDownLimitDistinctThroughUnion;
import org.apache.doris.nereids.rules.rewrite.PushDownMinMaxThroughJoin;
import org.apache.doris.nereids.rules.rewrite.PushDownSumThroughJoin;
import org.apache.doris.nereids.rules.rewrite.PushDownSumThroughJoinOneSide;
+import org.apache.doris.nereids.rules.rewrite.PushDownTopNDistinctThroughJoin;
import org.apache.doris.nereids.rules.rewrite.PushDownTopNThroughJoin;
import org.apache.doris.nereids.rules.rewrite.PushDownTopNThroughUnion;
import org.apache.doris.nereids.rules.rewrite.PushDownTopNThroughWindow;
@@ -329,9 +330,11 @@ public class Rewriter extends AbstractBatchJobExecutor {
topDown(new SplitLimit()),
topDown(
new PushDownLimit(),
-new PushDownTopNThroughJoin(),
new PushDownLimitDistinctThroughJoin(),
new PushDownLimitDistinctThroughUnion(),
+new PushDownTopNDistinctThroughJoin(),
+// new PushDownTopNDistinctThroughUnion(),
+new PushDownTopNThroughJoin(),
new PushDownTopNThroughWindow(),
new PushDownTopNThroughUnion()
),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
index e57934c9202..54ddd4152fa 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
@@ -265,6 +265,8 @@ public enum RuleType {
// topN push down
PUSH_DOWN_TOP_N_THROUGH_JOIN(RuleTypeClass.REWRITE),
PUSH_DOWN_TOP_N_THROUGH_PROJECT_JOIN(RuleTypeClass.REWRITE),
+PUSH_DOWN_TOP_N_DISTINCT_THROUGH_JOIN(RuleTypeClass.REWRITE),
+PUSH_DOWN_TOP_N_DISTINCT_THROUGH_PROJECT_JOIN(RuleTypeClass.REWRITE),
PUSH_DOWN_TOP_N_THROUGH_PROJECT_WINDOW(RuleTypeClass.REWRITE),
PUSH_DOWN_TOP_N_THROUGH_WINDOW(RuleTypeClass.REWRITE),
PUSH_DOWN_TOP_N_THROUGH_UNION(RuleTypeClass.REWRITE),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNDistinctThroughJoin.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNDistinctThroughJoin.java
new file mode 100644
index 000..98d1cdd3478
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNDistinctThroughJoin.java
@@ -0,0 +1,126 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the
(doris) branch master updated: [enhancement](Nereids): add builder for hyper graph (#30061)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 7d840cfa51e [enhancement](Nereids): add builder for hyper graph
(#30061)
7d840cfa51e is described below
commit 7d840cfa51ec3685ceef3985b8c0a329af8163a1
Author: 谢健
AuthorDate: Tue Jan 23 15:31:53 2024 +0800
[enhancement](Nereids): add builder for hyper graph (#30061)
---
.../doris/nereids/jobs/joinorder/JoinOrderJob.java | 7 +-
.../jobs/joinorder/hypergraph/HyperGraph.java | 715 ++---
.../joinorder/hypergraph/node/StructInfoNode.java | 16 -
.../nereids/rules/exploration/mv/StructInfo.java | 2 +-
.../joinorder/hypergraph/CompareOuterJoinTest.java | 24 +-
.../jobs/joinorder/hypergraph/InferJoinTest.java | 16 +-
.../joinorder/hypergraph/InferPredicateTest.java | 4 +-
.../joinorder/hypergraph/PullupExpressionTest.java | 16 +-
.../rules/exploration/mv/BuildStructInfoTest.java | 8 +-
.../rules/exploration/mv/HyperGraphAggTest.java| 6 +-
.../exploration/mv/HyperGraphComparatorTest.java | 16 +-
.../doris/nereids/util/HyperGraphBuilder.java | 4 +-
12 files changed, 398 insertions(+), 436 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/JoinOrderJob.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/JoinOrderJob.java
index b25793090bc..08e40f84a6c 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/JoinOrderJob.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/JoinOrderJob.java
@@ -73,11 +73,12 @@ public class JoinOrderJob extends Job {
}
private Group optimizeJoin(Group group) {
-HyperGraph hyperGraph = HyperGraph.toDPhyperGraph(group);
-for (AbstractNode node : hyperGraph.getNodes()) {
+HyperGraph.Builder builder = HyperGraph.builderForDPhyper(group);
+for (AbstractNode node : builder.getNodes()) {
DPhyperNode dPhyperNode = (DPhyperNode) node;
-hyperGraph.updateNode(node.getIndex(),
optimizePlan(dPhyperNode.getGroup()));
+builder.updateNode(node.getIndex(),
optimizePlan(dPhyperNode.getGroup()));
}
+HyperGraph hyperGraph = builder.build();
// TODO: Right now, we just hardcode the limit with 1, maybe we
need a better way to set it
int limit = 1000;
PlanReceiver planReceiver = new PlanReceiver(this.context, limit,
hyperGraph,
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/HyperGraph.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/HyperGraph.java
index 8a0bd8daaab..8472837d79e 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/HyperGraph.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/HyperGraph.java
@@ -42,6 +42,8 @@ import
org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import org.apache.doris.nereids.util.PlanUtils;
import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Lists;
@@ -58,21 +60,25 @@ import java.util.stream.Collectors;
* It's used for join ordering
*/
public class HyperGraph {
-private final List joinEdges = new ArrayList<>();
-private final List filterEdges = new ArrayList<>();
-private final List nodes = new ArrayList<>();
-private final HashMap slotToNodeMap = new HashMap<>();
// record all edges that can be placed on the subgraph
private final Map treeEdgesCache = new HashMap<>();
+private final List joinEdges;
+private final List filterEdges;
+private final List nodes;
private final Set finalOutputs;
// Record the complex project expression for some subgraph
// e.g. project (a + b)
// |-- join(t1.a = t2.b)
-private final HashMap> complexProject = new
HashMap<>();
+private final Map> complexProject;
-HyperGraph(Set finalOutputs) {
+HyperGraph(Set finalOutputs, List joinEdges,
List nodes, List filterEdges,
+Map> complexProject) {
this.finalOutputs = ImmutableSet.copyOf(finalOutputs);
+this.joinEdges = ImmutableList.copyOf(joinEdges);
+this.nodes = ImmutableList.copyOf(nodes);
+this.complexProject = ImmutableMap.copyOf(complexProject);
+this.filterEdges = ImmutableList.copyOf(filterEdges);
}
public List getJoinEdges() {
@@ -103,191 +109,10 @@ public class HyperGraph {
return nodes.get(index);
}
-/**
- * Store the relation between Alias Slot and Original Sl
(doris) branch master updated (c753b220362 -> ff5216a27a3)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from c753b220362 [regression test](schema change) add boolean type check for agg (#30186) add ff5216a27a3 [refactor](Nereids): Eager Aggregation unify pushdown agg function (#30142) No new revisions were added by this update. Summary of changes: .../rules/rewrite/PushDownMinMaxThroughJoin.java | 17 ++- .../rules/rewrite/PushDownSumThroughJoin.java | 4 +- .../rewrite/PushDownSumThroughJoinOneSide.java | 117 + 3 files changed, 19 insertions(+), 119 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (b2a9c5b7d43 -> 52ae3a92e8c)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from b2a9c5b7d43 [fix](statistics)Reanalyze olapTable if getRowCount is not 0 and last time row count is 0 (#30096) add 52ae3a92e8c [fix](Nereids): fix eliminate join test for pk-fk constraint (#30094) No new revisions were added by this update. Summary of changes: .../data/nereids_rules_p0/pkfk/eliminate_inner.out | 251 - .../nereids_rules_p0/pkfk/eliminate_inner.groovy | 16 +- 2 files changed, 102 insertions(+), 165 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (110b55a2091 -> 9c7d4b518e2)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 110b55a2091 [Improvement](aggregate) optimize for small string aggregate (#29919) add 9c7d4b518e2 [fix](Nereids): fix AssertNumRows StatsCalculator (#30053) No new revisions were added by this update. Summary of changes: .../doris/nereids/stats/StatsCalculator.java | 44 ++-- .../shape/query54.out | 112 +-- .../nereids_tpcds_shape_sf1000_p0/shape/query6.out | 68 ++-- .../noStatsRfPrune/query58.out | 21 ++-- .../no_stats_shape/query58.out | 21 ++-- .../rf_prune/query54.out | 120 ++--- .../rf_prune/query6.out| 7 +- .../nereids_tpcds_shape_sf100_p0/shape/query54.out | 120 ++--- .../nereids_tpcds_shape_sf100_p0/shape/query6.out | 9 +- 9 files changed, 267 insertions(+), 255 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](Nereids): eliminate redundant join condition after inferring condition (#30093)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 0c4e83f5fd3 [fix](Nereids): eliminate redundant join condition after
inferring condition (#30093)
0c4e83f5fd3 is described below
commit 0c4e83f5fd36144aeb4475ea8ffe63b92c40016c
Author: 谢健
AuthorDate: Thu Jan 18 15:55:34 2024 +0800
[fix](Nereids): eliminate redundant join condition after inferring
condition (#30093)
eliminate redundant join when find hashing join condition
such as for plan:
```
T1 join T2 on T1.id = T2.id join T3 on T1.id = T3.id and T2.id = T3.id
```
we infer a new predicate T1.id = T2.id which is redundant. Therefore we
need to eliminate it when find hash condition
---
.../doris/nereids/rules/rewrite/FindHashConditionForJoin.java| 9 +
.../nereids/jobs/joinorder/hypergraph/CompareOuterJoinTest.java | 5 -
2 files changed, 9 insertions(+), 5 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/FindHashConditionForJoin.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/FindHashConditionForJoin.java
index fe51a7d830b..afb593b 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/FindHashConditionForJoin.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/FindHashConditionForJoin.java
@@ -28,6 +28,7 @@ import
org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
import org.apache.doris.nereids.util.JoinUtils;
import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Streams;
import java.util.List;
@@ -61,10 +62,10 @@ public class FindHashConditionForJoin extends
OneRewriteRuleFactory {
return join;
}
-List combinedHashJoinConjuncts = new
ImmutableList.Builder()
-.addAll(join.getHashJoinConjuncts())
-.addAll(extractedHashJoinConjuncts)
-.build();
+List combinedHashJoinConjuncts = Streams
+.concat(join.getHashJoinConjuncts().stream(),
extractedHashJoinConjuncts.stream())
+.distinct()
+.collect(ImmutableList.toImmutableList());
JoinType joinType = join.getJoinType();
if (joinType == JoinType.CROSS_JOIN &&
!combinedHashJoinConjuncts.isEmpty()) {
joinType = JoinType.INNER_JOIN;
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/CompareOuterJoinTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/CompareOuterJoinTest.java
index f73d8a2923e..bbf7746ec6a 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/CompareOuterJoinTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/CompareOuterJoinTest.java
@@ -28,10 +28,12 @@ import
org.apache.doris.nereids.rules.exploration.mv.mapping.RelationMapping;
import org.apache.doris.nereids.rules.exploration.mv.mapping.SlotMapping;
import org.apache.doris.nereids.sqltest.SqlTestBase;
import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.plans.JoinType;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.util.HyperGraphBuilder;
import org.apache.doris.nereids.util.PlanChecker;
+import com.google.common.collect.Sets;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
@@ -69,7 +71,8 @@ class CompareOuterJoinTest extends SqlTestBase {
@Test
void testRandomQuery() {
-Plan p1 = new HyperGraphBuilder().randomBuildPlanWith(3, 3);
+Plan p1 = new HyperGraphBuilder(Sets.newHashSet(JoinType.INNER_JOIN))
+.randomBuildPlanWith(3, 3);
p1 = PlanChecker.from(connectContext, p1)
.analyze()
.rewrite()
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [minor](Nereids): remove useless method isMysqlCompatibleDatabase (#30064)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 5bd03fdc669 [minor](Nereids): remove useless method
isMysqlCompatibleDatabase (#30064)
5bd03fdc669 is described below
commit 5bd03fdc6696fff811b88c094e566ae6ef7dfecb
Author: jakevin
AuthorDate: Wed Jan 17 19:20:19 2024 +0800
[minor](Nereids): remove useless method isMysqlCompatibleDatabase (#30064)
---
.../src/main/java/org/apache/doris/analysis/DropDbStmt.java | 4 ++--
.../src/main/java/org/apache/doris/catalog/Database.java| 13 +++--
.../src/main/java/org/apache/doris/catalog/DatabaseIf.java | 2 +-
.../org/apache/doris/catalog/MysqlCompatibleDatabase.java | 5 -
.../src/main/java/org/apache/doris/clone/TabletChecker.java | 3 ++-
.../main/java/org/apache/doris/datasource/CatalogIf.java| 6 +++---
.../java/org/apache/doris/datasource/InternalCatalog.java | 10 +-
.../main/java/org/apache/doris/httpv2/rest/ShowAction.java | 5 +++--
.../apache/doris/master/PartitionInMemoryInfoCollector.java | 3 ++-
.../doris/transaction/DbUsedDataQuotaInfoCollector.java | 3 ++-
.../java/org/apache/doris/clone/RebalancerTestUtil.java | 3 ++-
11 files changed, 25 insertions(+), 32 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/DropDbStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/DropDbStmt.java
index 9b4f58efbf0..08a1e630eb5 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/DropDbStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/DropDbStmt.java
@@ -17,9 +17,9 @@
package org.apache.doris.analysis;
-import org.apache.doris.catalog.Database;
import org.apache.doris.catalog.DatabaseIf;
import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.MysqlCompatibleDatabase;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
import org.apache.doris.common.UserException;
@@ -62,7 +62,7 @@ public class DropDbStmt extends DdlStmt {
InternalDatabaseUtil.checkDatabase(dbName, ConnectContext.get());
// Don't allow to drop mysql compatible databases
DatabaseIf db = Env.getCurrentInternalCatalog().getDbNullable(dbName);
-if (db != null && (db instanceof Database) && ((Database)
db).isMysqlCompatibleDatabase()) {
+if (db != null && db instanceof MysqlCompatibleDatabase) {
ErrorReport.reportAnalysisException(ErrorCode.ERR_DBACCESS_DENIED_ERROR,
analyzer.getQualifiedUser(), dbName);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java
index 71e61d5791c..96db31af430 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java
@@ -84,14 +84,14 @@ public class Database extends MetaObject implements
Writable, DatabaseIf
private long id;
@SerializedName(value = "fullQualifiedName")
private volatile String fullQualifiedName;
-private ReentrantReadWriteLock rwLock;
+private final ReentrantReadWriteLock rwLock;
// table family group map
-private Map idToTable;
+private final Map idToTable;
@SerializedName(value = "nameToTable")
private Map nameToTable;
// table name lower cast -> table name
-private Map lowerCaseToTableName;
+private final Map lowerCaseToTableName;
// user define function
@SerializedName(value = "name2Function")
@@ -893,11 +893,4 @@ public class Database extends MetaObject implements
Writable, DatabaseIf
public String toString() {
return toJson();
}
-
-// Return ture if database is created for mysql compatibility.
-// Currently, we have two dbs that are created for this purpose,
InformationSchemaDb and MysqlDb,
-public boolean isMysqlCompatibleDatabase() {
-return false;
-}
-
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/DatabaseIf.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/DatabaseIf.java
index 3ba29573374..e80f335be17 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/DatabaseIf.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/DatabaseIf.java
@@ -273,5 +273,5 @@ public interface DatabaseIf {
return -1L;
}
-public Map getIdToTable();
+Map getIdToTable();
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/MysqlCompatibleDatabase.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/MysqlCompatibleDatabase.java
index 1d4aa62dc9d..12d915321ed 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/catalog/MysqlCompatibleDatabase
(doris) branch master updated: [fix](Nereids): fix enable mv rewrite is useless in dphyp (#30042)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a4ad25cabe4 [fix](Nereids): fix enable mv rewrite is useless in dphyp
(#30042)
a4ad25cabe4 is described below
commit a4ad25cabe469db4094dccef0421df6446e3f237
Author: 谢健
AuthorDate: Wed Jan 17 15:11:20 2024 +0800
[fix](Nereids): fix enable mv rewrite is useless in dphyp (#30042)
---
.../jobs/cascades/OptimizeGroupExpressionJob.java | 23 --
.../org/apache/doris/nereids/rules/RuleSet.java| 1 -
2 files changed, 17 insertions(+), 7 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/cascades/OptimizeGroupExpressionJob.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/cascades/OptimizeGroupExpressionJob.java
index 16cf90b786e..8c8e7e43998 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/cascades/OptimizeGroupExpressionJob.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/cascades/OptimizeGroupExpressionJob.java
@@ -24,7 +24,8 @@ import org.apache.doris.nereids.memo.GroupExpression;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.qe.ConnectContext;
-import java.util.ArrayList;
+import com.google.common.collect.ImmutableList;
+
import java.util.Collections;
import java.util.List;
@@ -48,11 +49,6 @@ public class OptimizeGroupExpressionJob extends Job {
countJobExecutionTimesOfGroupExpressions(groupExpression);
List implementationRules = getRuleSet().getImplementationRules();
List explorationRules = getExplorationRules();
-ConnectContext connectContext =
context.getCascadesContext().getConnectContext();
-if
(connectContext.getSessionVariable().isEnableMaterializedViewRewrite()) {
-explorationRules = new ArrayList<>(explorationRules);
-explorationRules.addAll(getRuleSet().getMaterializedViewRules());
-}
for (Rule rule : explorationRules) {
if (rule.isInvalid(disableRules, groupExpression)) {
@@ -70,6 +66,13 @@ public class OptimizeGroupExpressionJob extends Job {
}
private List getExplorationRules() {
+return ImmutableList.builder()
+.addAll(getJoinRules())
+.addAll(getMvRules())
+.build();
+}
+
+private List getJoinRules() {
boolean isDisableJoinReorder =
context.getCascadesContext().getConnectContext().getSessionVariable()
.isDisableJoinReorder()
||
context.getCascadesContext().getMemo().getGroupExpressionsSize() >
context.getCascadesContext()
@@ -96,4 +99,12 @@ public class OptimizeGroupExpressionJob extends Job {
return getRuleSet().getZigZagTreeJoinReorder();
}
}
+
+private List getMvRules() {
+ConnectContext connectContext =
context.getCascadesContext().getConnectContext();
+if
(connectContext.getSessionVariable().isEnableMaterializedViewRewrite()) {
+return getRuleSet().getMaterializedViewRules();
+}
+return ImmutableList.of();
+}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleSet.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleSet.java
index bb5b2f3dcf8..de0f3d518fa 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleSet.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleSet.java
@@ -240,7 +240,6 @@ public class RuleSet {
.build();
public static final List DPHYP_REORDER_RULES =
ImmutableList.builder()
-.addAll(MATERIALIZED_VIEW_RULES)
.add(JoinCommute.BUSHY.build())
.build();
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](Nereids): find hash condition after infer predicate (#30026)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e556cff8240 [fix](Nereids): find hash condition after infer predicate
(#30026)
e556cff8240 is described below
commit e556cff8240f1c3ae6470a5e3a107005df5b9f5a
Author: jakevin
AuthorDate: Wed Jan 17 11:08:18 2024 +0800
[fix](Nereids): find hash condition after infer predicate (#30026)
---
.../org/apache/doris/nereids/jobs/executor/Rewriter.java | 3 ++-
regression-test/data/nereids_p0/hint/fix_leading.out | 4 ++--
regression-test/data/nereids_p0/hint/test_leading.out| 16
3 files changed, 12 insertions(+), 11 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
index cbfe3d28ac8..b2f1569ea1d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
@@ -226,7 +226,7 @@ public class Rewriter extends AbstractBatchJobExecutor {
// but top-down traverse can not cover this case in one
iteration, so bottom-up is more
// efficient because it can find the new plans and apply
transform wherever it is
bottomUp(RuleSet.PUSH_DOWN_FILTERS),
-//after push down, some new filters are generated, which
needs to be optimized. (example: tpch q19)
+// after push down, some new filters are generated, which
needs to be optimized. (example: tpch q19)
topDown(new ExpressionOptimization()),
topDown(
new MergeFilters(),
@@ -263,6 +263,7 @@ public class Rewriter extends AbstractBatchJobExecutor {
// after eliminate outer join, we can move some filters to
join.otherJoinConjuncts,
// this can help to translate plan to backend
topDown(new PushFilterInsideJoin()),
+topDown(new FindHashConditionForJoin()),
topDown(new ExpressionNormalization())
),
diff --git a/regression-test/data/nereids_p0/hint/fix_leading.out
b/regression-test/data/nereids_p0/hint/fix_leading.out
index 898fe5882b9..1410c8e0e6e 100644
--- a/regression-test/data/nereids_p0/hint/fix_leading.out
+++ b/regression-test/data/nereids_p0/hint/fix_leading.out
@@ -9,9 +9,9 @@ PhysicalResultSink
--PhysicalDistribute[DistributionSpecHash]
PhysicalOlapScan[t2]
PhysicalDistribute[DistributionSpecHash]
---NestedLoopJoin[CROSS_JOIN](t3.c3 = t4.c4)
+--hashJoin[INNER_JOIN] hashCondition=((t3.c3 = t4.c4))
otherCondition=()
PhysicalOlapScan[t3]
-PhysicalDistribute[DistributionSpecReplicated]
+PhysicalDistribute[DistributionSpecHash]
--PhysicalOlapScan[t4]
Hint log:
diff --git a/regression-test/data/nereids_p0/hint/test_leading.out
b/regression-test/data/nereids_p0/hint/test_leading.out
index fe3831a9fc4..29811a95161 100644
--- a/regression-test/data/nereids_p0/hint/test_leading.out
+++ b/regression-test/data/nereids_p0/hint/test_leading.out
@@ -2609,7 +2609,7 @@ PhysicalResultSink
PhysicalProject
--PhysicalOlapScan[t2]
PhysicalDistribute[DistributionSpecReplicated]
---NestedLoopJoin[CROSS_JOIN](t1.c1 = t3.c3)
+--hashJoin[INNER_JOIN] hashCondition=((t1.c1 = t3.c3))
otherCondition=()
PhysicalProject
--PhysicalOlapScan[t1]
PhysicalDistribute[DistributionSpecReplicated]
@@ -2631,7 +2631,7 @@ PhysicalResultSink
PhysicalProject
--PhysicalOlapScan[t2]
PhysicalDistribute[DistributionSpecReplicated]
---NestedLoopJoin[CROSS_JOIN](t1.c1 = t3.c3)
+--hashJoin[INNER_JOIN] hashCondition=((t1.c1 = t3.c3))
otherCondition=()
PhysicalProject
--PhysicalOlapScan[t3]
PhysicalDistribute[DistributionSpecReplicated]
@@ -2745,7 +2745,7 @@ PhysicalResultSink
PhysicalProject
--PhysicalOlapScan[t2]
PhysicalDistribute[DistributionSpecReplicated]
---NestedLoopJoin[CROSS_JOIN](t1.c1 = t3.c3)
+--hashJoin[INNER_JOIN] hashCondition=((t1.c1 = t3.c3))
otherCondition=()
PhysicalProject
--PhysicalOlapScan[t1]
PhysicalDistribute[DistributionSpecReplicated]
@@ -2767,7 +2767,7 @@ PhysicalResultSink
PhysicalProject
--PhysicalOlapScan[t2]
PhysicalDistribute[DistributionSpecReplicated]
---NestedLoopJoin[CROSS_JOIN](t1.c1
(doris) branch branch-2.0 updated: [feature](Nereids): InferPredicates support In (#29458) (#30007)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new a24c0356b0e [feature](Nereids): InferPredicates support In (#29458)
(#30007)
a24c0356b0e is described below
commit a24c0356b0e8c4a17b10d82fb125c41778c262d8
Author: jakevin
AuthorDate: Tue Jan 16 16:21:03 2024 +0800
[feature](Nereids): InferPredicates support In (#29458) (#30007)
(cherry picked from commit 7a0734dbd60effa676d87bf5a5b7ca516e134d52)
---
.../nereids/rules/rewrite/InferPredicates.java | 11 +-
.../rules/rewrite/PredicatePropagation.java| 179 +
.../nereids/rules/rewrite/PullUpPredicates.java| 4 +-
.../nereids/trees/expressions/InPredicate.java | 34 ++--
.../nereids/rules/rewrite/InferPredicatesTest.java | 60 ---
.../rules/rewrite/PredicatePropagationTest.java| 51 ++
.../data/nereids_p0/hint/fix_leading.out | 2 +-
.../infer_predicate/infer_predicate.groovy | 2 +-
8 files changed, 220 insertions(+), 123 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/InferPredicates.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/InferPredicates.java
index 3c4593df54c..36236c3db8d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/InferPredicates.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/InferPredicates.java
@@ -27,7 +27,6 @@ import
org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
import org.apache.doris.nereids.util.ExpressionUtils;
import org.apache.doris.nereids.util.PlanUtils;
-import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Sets;
@@ -37,6 +36,7 @@ import java.util.stream.Collectors;
/**
* infer additional predicates for `LogicalFilter` and `LogicalJoin`.
+ *
* The logic is as follows:
* 1. poll up bottom predicate then infer additional predicates
* for example:
@@ -49,9 +49,9 @@ import java.util.stream.Collectors;
* select * from (select * from t1 where t1.id = 1) t join t2 on t.id =
t2.id and t2.id = 1
* 2. put these predicates into `otherJoinConjuncts` , these predicates are
processed in the next
* round of predicate push-down
+ *
*/
public class InferPredicates extends DefaultPlanRewriter
implements CustomRewriter {
-private final PredicatePropagation propagation = new
PredicatePropagation();
private final PullUpPredicates pollUpPredicates = new PullUpPredicates();
@Override
@@ -62,6 +62,9 @@ public class InferPredicates extends
DefaultPlanRewriter implements
@Override
public Plan visitLogicalJoin(LogicalJoin
join, JobContext context) {
join = visitChildren(this, join, context);
+if (join.isMarkJoin()) {
+return join;
+}
Plan left = join.left();
Plan right = join.right();
Set expressions = getAllExpressions(left, right,
join.getOnClauseCondition());
@@ -86,7 +89,7 @@ public class InferPredicates extends
DefaultPlanRewriter implements
break;
}
if (left != join.left() || right != join.right()) {
-return join.withChildren(ImmutableList.of(left, right));
+return join.withChildren(left, right);
} else {
return join;
}
@@ -109,7 +112,7 @@ public class InferPredicates extends
DefaultPlanRewriter implements
Set baseExpressions = pullUpPredicates(left);
baseExpressions.addAll(pullUpPredicates(right));
condition.ifPresent(on ->
baseExpressions.addAll(ExpressionUtils.extractConjunction(on)));
-baseExpressions.addAll(propagation.infer(baseExpressions));
+baseExpressions.addAll(PredicatePropagation.infer(baseExpressions));
return baseExpressions;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java
index 2317da427ea..aa520362a77 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java
@@ -24,6 +24,7 @@ import org.apache.doris.nereids.trees.expressions.Cast;
import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
import org.apache.doris.nereids.trees.expressions.EqualTo;
import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.InPredicate;
import org.apache.doris.nereids.trees.expressions.SlotReference;
import org.apache.doris.nereids.trees.expressions.literal.IntegerLikeLiteral;
imp
(doris) branch master updated: [fix](Nereids): fix NPE InferPredicates (#29978)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 2555e78076e [fix](Nereids): fix NPE InferPredicates (#29978)
2555e78076e is described below
commit 2555e78076e8665593ae0efe22237656ad9cc46d
Author: jakevin
AuthorDate: Tue Jan 16 10:09:25 2024 +0800
[fix](Nereids): fix NPE InferPredicates (#29978)
PredicatePropagation shouldn't add null into List.
---
.../java/org/apache/doris/nereids/rules/rewrite/InferPredicates.java | 4 ++--
.../org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java | 5 -
2 files changed, 6 insertions(+), 3 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/InferPredicates.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/InferPredicates.java
index 36236c3db8d..bafbc45cae3 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/InferPredicates.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/InferPredicates.java
@@ -122,8 +122,8 @@ public class InferPredicates extends
DefaultPlanRewriter implements
private Plan inferNewPredicate(Plan plan, Set expressions) {
Set predicates = expressions.stream()
-.filter(c -> !c.getInputSlots().isEmpty() &&
plan.getOutputSet().containsAll(
-c.getInputSlots())).collect(Collectors.toSet());
+.filter(c -> !c.getInputSlots().isEmpty() &&
plan.getOutputSet().containsAll(c.getInputSlots()))
+.collect(Collectors.toSet());
predicates.removeAll(plan.accept(pollUpPredicates, null));
return PlanUtils.filterOrSelf(predicates, plan);
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java
index 5d11a1fa542..d1eba6cce36 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java
@@ -95,7 +95,10 @@ public class PredicatePropagation {
slotPredicates.forEach((left, exprs) -> {
for (Slot right : equalSet.calEqualSet(left)) {
for (Expression expr : exprs) {
-inferred.add(doInferPredicate(left, right, expr));
+Expression inferPredicate = doInferPredicate(left, right,
expr);
+if (inferPredicate != null) {
+inferred.add(inferPredicate);
+}
}
}
});
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](Nereids): fix offset in PlanTranslator (#29789)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 0882223dcb6 [fix](Nereids): fix offset in PlanTranslator (#29789)
0882223dcb6 is described below
commit 0882223dcb6054975e02e76223db94cc533947df
Author: jakevin
AuthorDate: Mon Jan 15 17:34:08 2024 +0800
[fix](Nereids): fix offset in PlanTranslator (#29789)
Current BE operator don't support `offset`, we need add offset into
`ExchangeNode`
---
.../glue/translator/PhysicalPlanTranslator.java| 23 +++-
.../processor/post/AddOffsetIntoDistribute.java| 41 ++
.../nereids/processor/post/PlanPostProcessors.java | 1 +
.../trees/plans/physical/PhysicalLimit.java| 5 +++
4 files changed, 61 insertions(+), 9 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
index 6270c5795b3..b20d0738140 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
@@ -268,13 +268,14 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor distribute,
PlanTranslatorContext context) {
-PlanFragment inputFragment = distribute.child().accept(this, context);
-List> distributeExprLists =
getDistributeExprs(distribute.child());
+Plan child = distribute.child();
+PlanFragment inputFragment = child.accept(this, context);
+List> distributeExprLists = getDistributeExprs(child);
// TODO: why need set streaming here? should remove this.
if (inputFragment.getPlanRoot() instanceof AggregationNode
-&& distribute.child() instanceof PhysicalHashAggregate
-&& context.getFirstAggregateInFragment(inputFragment) ==
distribute.child()) {
-PhysicalHashAggregate hashAggregate =
(PhysicalHashAggregate) distribute.child();
+&& child instanceof PhysicalHashAggregate
+&& context.getFirstAggregateInFragment(inputFragment) ==
child) {
+PhysicalHashAggregate hashAggregate =
(PhysicalHashAggregate) child;
if (hashAggregate.getAggPhase() == AggPhase.LOCAL
&& hashAggregate.getAggMode() == AggMode.INPUT_TO_BUFFER) {
AggregationNode aggregationNode = (AggregationNode)
inputFragment.getPlanRoot();
@@ -285,23 +286,26 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor validOutputIds = distribute.getOutputExprIds();
-if (distribute.child() instanceof PhysicalHashAggregate) {
+if (child instanceof PhysicalHashAggregate) {
// we must add group by keys to output list,
// otherwise we could not process aggregate's output without group
by keys
-List keys = ((PhysicalHashAggregate)
distribute.child()).getGroupByExpressions().stream()
+List keys = ((PhysicalHashAggregate)
child).getGroupByExpressions().stream()
.filter(SlotReference.class::isInstance)
.map(SlotReference.class::cast)
.map(SlotReference::getExprId)
.collect(Collectors.toList());
keys.addAll(validOutputIds);
validOutputIds = keys;
+} else if (child instanceof PhysicalLimit && ((PhysicalLimit)
child).getPhase().isGlobal()) {
+// because sort already contains Offset, we don't need to handle
PhysicalTopN
+exchangeNode.setOffset(((PhysicalLimit) child).getOffset());
}
if (inputFragment instanceof MultiCastPlanFragment) {
// TODO: remove this logic when we split to multi-window in
logical window to physical window conversion
MultiCastDataSink multiCastDataSink = (MultiCastDataSink)
inputFragment.getSink();
DataStreamSink dataStreamSink =
multiCastDataSink.getDataStreamSinks().get(
multiCastDataSink.getDataStreamSinks().size() - 1);
-if (!(distribute.child() instanceof PhysicalProject)) {
+if (!(child instanceof PhysicalProject)) {
List projectionExprs = new ArrayList<>();
PhysicalCTEConsumer consumer = getCTEConsumerChild(distribute);
Preconditions.checkState(consumer != null, "consumer not
found");
@@ -1591,7 +1595,8 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitorhttp://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distri
(doris) branch master updated: extract agg in struct info node (#29853)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 245efbe4786 extract agg in struct info node (#29853)
245efbe4786 is described below
commit 245efbe4786f0f4409d0c4398ffc533b675e898c
Author: 谢健
AuthorDate: Mon Jan 15 13:32:21 2024 +0800
extract agg in struct info node (#29853)
---
.../joinorder/hypergraph/node/StructInfoNode.java | 62 ++
.../rules/exploration/mv/HyperGraphComparator.java | 35 +++-
.../rules/exploration/mv/HyperGraphAggTest.java| 98 ++
3 files changed, 191 insertions(+), 4 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/node/StructInfoNode.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/node/StructInfoNode.java
index 424ca6a5f10..9e3886bfdca 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/node/StructInfoNode.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/node/StructInfoNode.java
@@ -19,14 +19,26 @@ package
org.apache.doris.nereids.jobs.joinorder.hypergraph.node;
import org.apache.doris.nereids.jobs.joinorder.hypergraph.HyperGraph;
import org.apache.doris.nereids.jobs.joinorder.hypergraph.edge.Edge;
+import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.LeafPlan;
import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.algebra.CatalogRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCatalogRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import org.apache.doris.nereids.util.Utils;
import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
import java.util.ArrayList;
+import java.util.Collection;
import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
/**
* HyperGraph Node.
@@ -34,9 +46,13 @@ import java.util.List;
public class StructInfoNode extends AbstractNode {
private List graphs = new ArrayList<>();
+private final List> expressions;
+private final Set relationSet;
public StructInfoNode(int index, Plan plan, List edges) {
super(extractPlan(plan), index, edges);
+relationSet = plan.collect(CatalogRelation.class::isInstance);
+expressions = collectExpressions(plan);
}
public StructInfoNode(int index, Plan plan) {
@@ -48,6 +64,52 @@ public class StructInfoNode extends AbstractNode {
this.graphs = graphs;
}
+private @Nullable List> collectExpressions(Plan plan) {
+if (plan instanceof LeafPlan) {
+return ImmutableList.of();
+}
+List> childExpressions =
collectExpressions(plan.child(0));
+if (!isValidNodePlan(plan) || childExpressions == null) {
+return null;
+}
+if (plan instanceof LogicalAggregate) {
+return ImmutableList.>builder()
+.add(ImmutableSet.copyOf(plan.getExpressions()))
+.add(ImmutableSet.copyOf(((LogicalAggregate)
plan).getGroupByExpressions()))
+.addAll(childExpressions)
+.build();
+}
+return ImmutableList.>builder()
+.add(ImmutableSet.copyOf(plan.getExpressions()))
+.addAll(childExpressions)
+.build();
+}
+
+private boolean isValidNodePlan(Plan plan) {
+return plan instanceof LogicalProject || plan instanceof
LogicalAggregate
+|| plan instanceof LogicalFilter || plan instanceof
LogicalCatalogRelation;
+}
+
+/**
+ * get all expressions of nodes
+ */
+public @Nullable List getExpressions() {
+return expressions.stream()
+.flatMap(Collection::stream)
+.collect(Collectors.toList());
+}
+
+public @Nullable List> getExprSetList() {
+return expressions;
+}
+
+/**
+ * return catalog relation
+ */
+public Set getCatalogRelation() {
+return relationSet;
+}
+
private static Plan extractPlan(Plan plan) {
if (plan instanceof GroupPlan) {
//TODO: Note mv can be in logicalExpression, how can we choose it
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/HyperGraphComparator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/HyperGraphComparator.java
index 7817475c7b8..b
(doris) branch master updated: [refactor](Nereids): avoid ConnectContext.get() ASAP to improve proformance (#29952)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ce3c60d110a [refactor](Nereids): avoid ConnectContext.get() ASAP to
improve proformance (#29952)
ce3c60d110a is described below
commit ce3c60d110af82c72c64a6ec9e9b902bdafbf5e6
Author: jakevin
AuthorDate: Mon Jan 15 10:00:47 2024 +0800
[refactor](Nereids): avoid ConnectContext.get() ASAP to improve proformance
(#29952)
---
.../org/apache/doris/nereids/cost/CostModelV1.java | 2 +-
.../glue/translator/PhysicalPlanTranslator.java| 7 +++--
.../glue/translator/PlanTranslatorContext.java | 4 +++
.../java/org/apache/doris/nereids/jobs/Job.java| 5 +++
.../nereids/jobs/cascades/CostAndEnforcerJob.java | 5 ---
.../org/apache/doris/nereids/metrics/Event.java| 7 +++--
.../doris/nereids/minidump/MinidumpUtils.java | 36 --
.../expression/rules/FoldConstantRuleOnFE.java | 2 +-
.../doris/nereids/rules/rewrite/CTEInline.java | 7 +++--
.../org/apache/doris/nereids/trees/plans/Plan.java | 2 +-
.../plans/commands/InsertIntoTableCommand.java | 4 +--
.../trees/plans/commands/info/AlterMTMVInfo.java | 4 +--
.../trees/plans/commands/info/CreateMTMVInfo.java | 4 +--
.../trees/plans/commands/info/DropMTMVInfo.java| 4 +--
.../trees/plans/commands/info/RefreshMTMVInfo.java | 4 +--
15 files changed, 53 insertions(+), 44 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java
index 011afb46fc2..d56a9d8be2a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java
@@ -75,7 +75,7 @@ class CostModelV1 extends PlanVisitor {
beNumber = sessionVariable.getBeNumberForTest();
parallelInstance = 8;
} else {
-beNumber = Math.max(1,
ConnectContext.get().getEnv().getClusterInfo().getBackendsNumber(true));
+beNumber = Math.max(1,
connectContext.getEnv().getClusterInfo().getBackendsNumber(true));
parallelInstance = Math.max(1,
connectContext.getSessionVariable().getParallelExecInstanceNum());
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
index 9f8f82ad679..6270c5795b3 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
@@ -179,12 +179,12 @@ import
org.apache.doris.planner.external.iceberg.IcebergScanNode;
import org.apache.doris.planner.external.jdbc.JdbcScanNode;
import org.apache.doris.planner.external.odbc.OdbcScanNode;
import org.apache.doris.planner.external.paimon.PaimonScanNode;
-import org.apache.doris.qe.ConnectContext;
import org.apache.doris.statistics.StatisticConstants;
import org.apache.doris.tablefunction.TableValuedFunctionIf;
import org.apache.doris.thrift.TFetchOption;
import org.apache.doris.thrift.TPartitionType;
import org.apache.doris.thrift.TPushAggOp;
+import org.apache.doris.thrift.TResultSinkType;
import org.apache.doris.thrift.TRuntimeFilterType;
import com.google.common.base.Preconditions;
@@ -363,8 +363,9 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor physicalResultSink,
PlanTranslatorContext context) {
PlanFragment planFragment = physicalResultSink.child().accept(this,
context);
-planFragment.setSink(new ResultSink(planFragment.getPlanRoot().getId(),
-ConnectContext.get().getResultSinkType()));
+TResultSinkType resultSinkType = context.getConnectContext() != null ?
context.getConnectContext()
+.getResultSinkType() : null;
+planFragment.setSink(new
ResultSink(planFragment.getPlanRoot().getId(), resultSinkType));
return planFragment;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PlanTranslatorContext.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PlanTranslatorContext.java
index 8f273d4ec08..77b386c79e3 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PlanTranslatorContext.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PlanTranslatorContext.java
@@ -151,6 +151,10 @@ public class PlanTranslatorContext {
return connectContext == null ? null :
connectContext.getSessionVariable();
}
+public ConnectContext getConnectContext() {
+return connectContext;
+}
+
public Set
(doris) branch master updated: [fix](Nereids): add def in test (#29796)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new defd4fbd07a [fix](Nereids): add def in test (#29796)
defd4fbd07a is described below
commit defd4fbd07aad8d14be9be2a508a0da79588dd4a
Author: jakevin
AuthorDate: Thu Jan 11 00:04:24 2024 +0800
[fix](Nereids): add def in test (#29796)
missing `def` will cause variable is global, it's easily to cause
concurrent problem
---
.../suites/nereids_p0/create_table/test_ctas.groovy| 4 ++--
.../suites/nereids_p0/expression/case_when_to_if.groovy| 10 +-
.../suites/nereids_p0/expression/topn_to_max.groovy| 4 ++--
3 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/regression-test/suites/nereids_p0/create_table/test_ctas.groovy
b/regression-test/suites/nereids_p0/create_table/test_ctas.groovy
index dcbf41139b8..507bff03ea0 100644
--- a/regression-test/suites/nereids_p0/create_table/test_ctas.groovy
+++ b/regression-test/suites/nereids_p0/create_table/test_ctas.groovy
@@ -72,8 +72,8 @@ suite("nereids_test_ctas") {
) as select test_varchar, lpad(test_text,10,'0') as test_text,
test_datetime, test_default_timestamp from test_ctas;
"""
-res = sql """SHOW CREATE TABLE `test_ctas2`"""
-assertTrue(res.size() != 0)
+def res1 = sql """SHOW CREATE TABLE `test_ctas2`"""
+assertTrue(res1.size() != 0)
qt_select """select count(*) from test_ctas2"""
diff --git
a/regression-test/suites/nereids_p0/expression/case_when_to_if.groovy
b/regression-test/suites/nereids_p0/expression/case_when_to_if.groovy
index 8c41b8d1c67..c83ce27c017 100644
--- a/regression-test/suites/nereids_p0/expression/case_when_to_if.groovy
+++ b/regression-test/suites/nereids_p0/expression/case_when_to_if.groovy
@@ -30,7 +30,7 @@ suite("test_case_when_to_if") {
from test_case_when_to_if
group by k2;
'''
-res = sql '''
+def res = sql '''
explain rewritten plan select k2,
sum(case when (k1=1) then 1 end) sum1
from test_case_when_to_if
@@ -45,13 +45,13 @@ suite("test_case_when_to_if") {
from test_case_when_to_if
group by k2;
'''
-res = sql '''
+def res1 = sql '''
explain rewritten plan select k2,
sum(case when (k1=1) then 1 else null end) sum1
from test_case_when_to_if
group by k2;
'''
-assertTrue(res.toString().contains("if"))
+assertTrue(res1.toString().contains("if"))
sql '''
select k2,
@@ -59,11 +59,11 @@ suite("test_case_when_to_if") {
from test_case_when_to_if
group by k2;
'''
-res = sql '''
+def res2 = sql '''
explain rewritten plan select k2,
sum(case when (k1>0) then k1 else abs(k1) end) sum1
from test_case_when_to_if
group by k2;
'''
-assertTrue(res.toString().contains("if"))
+assertTrue(res2.toString().contains("if"))
}
diff --git a/regression-test/suites/nereids_p0/expression/topn_to_max.groovy
b/regression-test/suites/nereids_p0/expression/topn_to_max.groovy
index 4ed378144fa..83fb9cc8492 100644
--- a/regression-test/suites/nereids_p0/expression/topn_to_max.groovy
+++ b/regression-test/suites/nereids_p0/expression/topn_to_max.groovy
@@ -41,9 +41,9 @@ suite("test_topn_to_max") {
select topn(k2, 1)
from test_topn_to_max;
'''
-res = sql '''
+def res1 = sql '''
explain rewritten plan select topn(k2, 1)
from test_topn_to_max;
'''
-assertTrue(res.toString().contains("max"), res.toString() + " should
contain max")
+assertTrue(res1.toString().contains("max"), res1.toString() + " should
contain max")
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Cleanup](Nereids): delete useless `ddlSql` to avoid wrong usage (#29788)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 8102aefc606 [Cleanup](Nereids): delete useless `ddlSql` to avoid wrong
usage (#29788)
8102aefc606 is described below
commit 8102aefc606cf0d0548477abb7c20edfa6779446
Author: jakevin
AuthorDate: Wed Jan 10 21:15:24 2024 +0800
[Cleanup](Nereids): delete useless `ddlSql` to avoid wrong usage (#29788)
ddlSql is useless and some code use getDdlSql() wrong, so delete those code
---
.../src/main/java/org/apache/doris/catalog/MetaObject.java | 4 ++--
.../src/main/java/org/apache/doris/catalog/OlapTable.java| 12 ++--
fe/fe-core/src/main/java/org/apache/doris/catalog/Table.java | 11 ---
fe/fe-core/src/main/java/org/apache/doris/catalog/View.java | 5 -
.../java/org/apache/doris/datasource/InternalCatalog.java| 2 +-
.../apache/doris/nereids/rules/analysis/BindRelation.java| 2 +-
.../test/java/org/apache/doris/catalog/CreateViewTest.java | 5 -
7 files changed, 6 insertions(+), 35 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/MetaObject.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/MetaObject.java
index 09930b09594..2de2234ea16 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/MetaObject.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/MetaObject.java
@@ -26,14 +26,14 @@ import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
-public class MetaObject implements Writable {
+public abstract class MetaObject implements Writable {
@SerializedName(value = "signature")
protected long signature;
@SerializedName(value = "lastCheckTime")
protected long lastCheckTime; // last check consistency time
-public MetaObject() {
+protected MetaObject() {
signature = -1L;
lastCheckTime = -1L;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
index a8d7fc085bd..3236fc09ec9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
@@ -21,7 +21,6 @@ import org.apache.doris.alter.MaterializedViewHandler;
import org.apache.doris.analysis.AggregateInfo;
import org.apache.doris.analysis.Analyzer;
import org.apache.doris.analysis.ColumnDef;
-import org.apache.doris.analysis.CreateTableStmt;
import org.apache.doris.analysis.DataSortInfo;
import org.apache.doris.analysis.Expr;
import org.apache.doris.analysis.SlotDescriptor;
@@ -1264,11 +1263,6 @@ public class OlapTable extends Table {
return dataSize;
}
-@Override
-public CreateTableStmt toCreateTableStmt(String dbName) {
-throw new RuntimeException("Don't support anymore");
-}
-
// Get the md5 of signature string of this table with specified partitions.
// This method is used to determine whether the tables have the same
schema.
// Contains:
@@ -2396,7 +2390,7 @@ public class OlapTable extends Table {
&& getEnableUniqueKeyMergeOnWrite());
}
-public void initAutoIncrentGenerator(long dbId) {
+public void initAutoIncrementGenerator(long dbId) {
for (Column column : fullSchema) {
if (column.isAutoInc()) {
autoIncrementGenerator = new AutoIncrementGenerator(dbId, id,
column.getUniqueId());
@@ -2498,9 +2492,7 @@ public class OlapTable extends Table {
public List getAllTablets() throws AnalysisException {
List tablets = Lists.newArrayList();
for (Partition partition : getPartitions()) {
-for (Tablet tablet : partition.getBaseIndex().getTablets()) {
-tablets.add(tablet);
-}
+tablets.addAll(partition.getBaseIndex().getTablets());
}
return tablets;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Table.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Table.java
index d0df5bad714..3395da6b58e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Table.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Table.java
@@ -18,7 +18,6 @@
package org.apache.doris.catalog;
import org.apache.doris.alter.AlterCancelException;
-import org.apache.doris.analysis.CreateTableStmt;
import org.apache.doris.catalog.constraint.Constraint;
import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
@@ -116,8 +115,6 @@ public abstract class Table extends MetaObject implements
Writable, TableIf {
// table(view)'s comment
@SerializedName(value = "comment")
protected String comment = "";
-// sql
(doris) branch master updated: [feature](Nereids): merge topNs (#28246)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e04a8eea8f2 [feature](Nereids): merge topNs (#28246)
e04a8eea8f2 is described below
commit e04a8eea8f2f0265f558153f3aaff33673c02f24
Author: jakevin
AuthorDate: Wed Jan 10 15:33:45 2024 +0800
[feature](Nereids): merge topNs (#28246)
merge topNs like
```
TopN
|
TopN
merge ->
TopN
```
---
.../doris/nereids/jobs/executor/Rewriter.java | 2 +
.../org/apache/doris/nereids/rules/RuleType.java | 1 +
.../doris/nereids/rules/rewrite/MergeTopNs.java| 56
.../nereids/rules/rewrite/MergeTopNsTest.java | 77 ++
4 files changed, 136 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
index e02c299e71f..cbfe3d28ac8 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
@@ -82,6 +82,7 @@ import org.apache.doris.nereids.rules.rewrite.MergeFilters;
import org.apache.doris.nereids.rules.rewrite.MergeOneRowRelationIntoUnion;
import org.apache.doris.nereids.rules.rewrite.MergeProjects;
import org.apache.doris.nereids.rules.rewrite.MergeSetOperations;
+import org.apache.doris.nereids.rules.rewrite.MergeTopNs;
import org.apache.doris.nereids.rules.rewrite.NormalizeSort;
import org.apache.doris.nereids.rules.rewrite.OrExpansion;
import org.apache.doris.nereids.rules.rewrite.PruneEmptyPartition;
@@ -323,6 +324,7 @@ public class Rewriter extends AbstractBatchJobExecutor {
// generate one PhysicalLimit if current
distribution is gather or two
// PhysicalLimits with gather exchange
topDown(new LimitSortToTopN()),
+topDown(new MergeTopNs()),
topDown(new SplitLimit()),
topDown(
new PushDownLimit(),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
index 61efe9e32f0..e57934c9202 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
@@ -245,6 +245,7 @@ public enum RuleType {
PUSH_PROJECT_INTO_ONE_ROW_RELATION(RuleTypeClass.REWRITE),
PUSH_PROJECT_INTO_UNION(RuleTypeClass.REWRITE),
MERGE_SET_OPERATION(RuleTypeClass.REWRITE),
+MERGE_TOP_N(RuleTypeClass.REWRITE),
BUILD_AGG_FOR_UNION(RuleTypeClass.REWRITE),
COUNT_DISTINCT_REWRITE(RuleTypeClass.REWRITE),
INNER_TO_CROSS_JOIN(RuleTypeClass.REWRITE),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MergeTopNs.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MergeTopNs.java
new file mode 100644
index 000..39ea7a384df
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MergeTopNs.java
@@ -0,0 +1,56 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
+
+import java.util.List;
+
+/**
+ * this rule aims to merge consecutive topN
+ *
+ * topN - child topN
+ * If child topN orderby list is prefix subList of topN =>
+ * topN with limit = min(topN.limit, childTopN.limit), offset = topN.offset +
childTopN.offset
+ */
+public class MergeTopNs extends OneRewriteRuleFactory {
+@Override
+public Rule build() {
+return logicalTopN(logicalTopN())
+.then(topN -> {
+
(doris) branch master updated: [refactor](Nereids): unify all `replaceNamedExpressions` (#28228)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new cedd492dec4 [refactor](Nereids): unify all `replaceNamedExpressions`
(#28228)
cedd492dec4 is described below
commit cedd492dec4c3e29edf694233d671511d2d6d3ca
Author: jakevin
AuthorDate: Tue Jan 9 13:32:55 2024 +0800
[refactor](Nereids): unify all `replaceNamedExpressions` (#28228)
Use a unified function `replaceNamedExpressions ` instead of implementing
it yourself repeatedly.
---
.../nereids/rules/rewrite/EliminateFilter.java | 10 +--
.../rewrite/PushProjectIntoOneRowRelation.java | 36 ++---
.../doris/nereids/trees/plans/algebra/Project.java | 13 +---
.../apache/doris/nereids/util/ExpressionUtils.java | 33 +
.../doris/nereids/util/ImmutableEqualSet.java | 11 +--
.../org/apache/doris/nereids/util/PlanUtils.java | 23 +-
.../shape/query51.out | 85 +++---
.../shape/query53.out | 61
.../noStatsRfPrune/query51.out | 85 +++---
.../noStatsRfPrune/query53.out | 57 +++
.../no_stats_shape/query51.out | 85 +++---
.../no_stats_shape/query53.out | 57 +++
.../rf_prune/query51.out | 85 +++---
.../rf_prune/query53.out | 61
.../nereids_tpcds_shape_sf100_p0/shape/query51.out | 85 +++---
.../nereids_tpcds_shape_sf100_p0/shape/query53.out | 61
16 files changed, 405 insertions(+), 443 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateFilter.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateFilter.java
index 2f46c43ecff..a3de71a770e 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateFilter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateFilter.java
@@ -21,9 +21,8 @@ import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
-import org.apache.doris.nereids.trees.expressions.Alias;
import org.apache.doris.nereids.trees.expressions.Expression;
-import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.logical.LogicalEmptyRelation;
@@ -36,7 +35,6 @@ import com.google.common.collect.ImmutableSet;
import java.util.List;
import java.util.Map;
-import java.util.stream.Collectors;
/**
* Eliminate filter which is FALSE or TRUE.
@@ -68,11 +66,9 @@ public class EliminateFilter implements RewriteRuleFactory {
.toRule(RuleType.ELIMINATE_FILTER),
logicalFilter(logicalOneRowRelation()).thenApply(ctx -> {
LogicalFilter filter = ctx.root;
-Map replaceMap =
-filter.child().getOutputs().stream().filter(e -> e
instanceof Alias)
-
.collect(Collectors.toMap(NamedExpression::toSlot, e -> ((Alias) e).child()));
+Map replaceMap =
ExpressionUtils.generateReplaceMap(filter.child().getOutputs());
-ImmutableSet.Builder newConjuncts = ImmutableSet.builder();
+ImmutableSet.Builder newConjuncts =
ImmutableSet.builder();
ExpressionRewriteContext context =
new ExpressionRewriteContext(ctx.cascadesContext);
for (Expression expression : filter.getConjuncts()) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushProjectIntoOneRowRelation.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushProjectIntoOneRowRelation.java
index 04faffd9afc..21fad1904c1 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushProjectIntoOneRowRelation.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushProjectIntoOneRowRelation.java
@@ -19,16 +19,11 @@ package org.apache.doris.nereids.rules.rewrite;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
-import org.apache.doris.nereids.trees.expressions.Alias;
-import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.NamedE
(doris) branch master updated: [refactor](Nereids): refactor PredicatePropagation & support to infer Equal Condition (#29644)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 86fc19397b8 [refactor](Nereids): refactor PredicatePropagation &
support to infer Equal Condition (#29644)
86fc19397b8 is described below
commit 86fc19397b80613f179b412999cde0e4219bf7b0
Author: jakevin
AuthorDate: Mon Jan 8 19:42:12 2024 +0800
[refactor](Nereids): refactor PredicatePropagation & support to infer Equal
Condition (#29644)
---
.../nereids/rules/rewrite/EliminateJoinByFK.java | 6 +-
.../rules/rewrite/PredicatePropagation.java| 176 -
.../nereids/trees/plans/logical/LogicalJoin.java | 8 +-
...eEquivalenceSet.java => ImmutableEqualSet.java} | 64 +---
.../rules/rewrite/PredicatePropagationTest.java| 16 ++
.../data/nereids_p0/hint/fix_leading.out | 2 +-
6 files changed, 134 insertions(+), 138 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateJoinByFK.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateJoinByFK.java
index 594dee50853..b4a6eac207b 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateJoinByFK.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateJoinByFK.java
@@ -36,7 +36,7 @@ import
org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import org.apache.doris.nereids.trees.plans.logical.LogicalRelation;
import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
-import org.apache.doris.nereids.util.ImmutableEquivalenceSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
@@ -145,7 +145,7 @@ public class EliminateJoinByFK extends
DefaultPlanRewriter implement
return project;
}
-private @Nullable Map
mapPrimaryToForeign(ImmutableEquivalenceSet equivalenceSet,
+private @Nullable Map
mapPrimaryToForeign(ImmutableEqualSet equivalenceSet,
Set foreignKeys) {
ImmutableMap.Builder builder = new
ImmutableMap.Builder<>();
for (Slot foreignSlot : foreignKeys) {
@@ -164,7 +164,7 @@ public class EliminateJoinByFK extends
DefaultPlanRewriter implement
// 4. if foreign key is null, add a isNotNull predicate for
null-reject join condition
private Plan eliminateJoin(LogicalProject> project,
ForeignKeyContext context) {
LogicalJoin join = project.child();
-ImmutableEquivalenceSet equalSet = join.getEqualSlots();
+ImmutableEqualSet equalSet = join.getEqualSlots();
Set leftSlots =
Sets.intersection(join.left().getOutputSet(), equalSet.getAllItemSet());
Set rightSlots =
Sets.intersection(join.right().getOutputSet(), equalSet.getAllItemSet());
if (context.isForeignKey(leftSlots) &&
context.isPrimaryKey(rightSlots)) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java
index 7788bbb7f06..5d11a1fa542 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PredicatePropagation.java
@@ -17,16 +17,14 @@
package org.apache.doris.nereids.rules.rewrite;
-import org.apache.doris.nereids.parser.NereidsParser;
-import org.apache.doris.nereids.rules.expression.rules.DateFunctionRewrite;
-import
org.apache.doris.nereids.rules.expression.rules.SimplifyComparisonPredicate;
+import org.apache.doris.common.Pair;
import org.apache.doris.nereids.trees.expressions.Cast;
import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
import org.apache.doris.nereids.trees.expressions.EqualTo;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.InPredicate;
+import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.expressions.SlotReference;
-import org.apache.doris.nereids.trees.expressions.literal.IntegerLikeLiteral;
import org.apache.doris.nereids.types.DataType;
import org.apache.doris.nereids.types.DateTimeType;
import org.apache.doris.nereids.types.DateTimeV2Type;
@@ -35,11 +33,15 @@ import org.apache.doris.nereids.types.DateV2Type;
import org.apache.doris.nereids.types.coercion.CharacterType;
import org.apache.doris.nereids.types.coercion.DateLikeType;
import org.apache.doris.nereids.types.coercion.IntegralType;
+import org.apache.doris.nereids.uti
(doris) branch master updated: [feature](Nereids): refresh view hypergraph after inferring join (#29469)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 59bc048ddb6 [feature](Nereids): refresh view hypergraph after
inferring join (#29469)
59bc048ddb6 is described below
commit 59bc048ddb6424d4319c1e23a35d78f1567c2958
Author: 谢健
AuthorDate: Mon Jan 8 15:30:14 2024 +0800
[feature](Nereids): refresh view hypergraph after inferring join (#29469)
---
.../joinorder/hypergraph/ConflictRulesMaker.java | 108 +
.../jobs/joinorder/hypergraph/HyperGraph.java | 73 +
.../jobs/joinorder/hypergraph/edge/FilterEdge.java | 4 +
.../jobs/joinorder/hypergraph/edge/JoinEdge.java | 5 +
.../rules/exploration/mv/HyperGraphComparator.java | 140 +
.../exploration/mv/HyperGraphComparatorTest.java | 172 +
6 files changed, 399 insertions(+), 103 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/ConflictRulesMaker.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/ConflictRulesMaker.java
new file mode 100644
index 000..11db05843fe
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/ConflictRulesMaker.java
@@ -0,0 +1,108 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.jobs.joinorder.hypergraph;
+
+import org.apache.doris.nereids.jobs.joinorder.hypergraph.bitmap.LongBitmap;
+import org.apache.doris.nereids.jobs.joinorder.hypergraph.edge.Edge;
+import org.apache.doris.nereids.jobs.joinorder.hypergraph.edge.FilterEdge;
+import org.apache.doris.nereids.jobs.joinorder.hypergraph.edge.JoinEdge;
+import org.apache.doris.nereids.rules.rewrite.PushDownFilterThroughJoin;
+import org.apache.doris.nereids.trees.plans.JoinType;
+
+import java.util.BitSet;
+import java.util.List;
+
+/**
+ * This is a conflict rule maker to
+ */
+public class ConflictRulesMaker {
+private ConflictRulesMaker() {}
+
+/**
+ * make conflict rules for filter edges
+ */
+public static void makeFilterConflictRules(
+JoinEdge joinEdge, List joinEdges, List
filterEdges) {
+long leftSubNodes = joinEdge.getLeftSubNodes(joinEdges);
+long rightSubNodes = joinEdge.getRightSubNodes(joinEdges);
+filterEdges.forEach(e -> {
+if (LongBitmap.isSubset(e.getReferenceNodes(), leftSubNodes)
+&&
!PushDownFilterThroughJoin.COULD_PUSH_THROUGH_LEFT.contains(joinEdge.getJoinType()))
{
+e.addLeftRejectEdge(joinEdge);
+}
+if (LongBitmap.isSubset(e.getReferenceNodes(), rightSubNodes)
+&&
!PushDownFilterThroughJoin.COULD_PUSH_THROUGH_RIGHT.contains(joinEdge.getJoinType()))
{
+e.addRightRejectEdge(joinEdge);
+}
+});
+}
+
+/**
+ * Make edge with CD-C algorithm in
+ * On the correct and complete enumeration of the core search
+ */
+public static void makeJoinConflictRules(JoinEdge edgeB, List
joinEdges) {
+BitSet leftSubTreeEdges = subTreeEdges(edgeB.getLeftChildEdges(),
joinEdges);
+BitSet rightSubTreeEdges = subTreeEdges(edgeB.getRightChildEdges(),
joinEdges);
+long leftRequired = edgeB.getLeftRequiredNodes();
+long rightRequired = edgeB.getRightRequiredNodes();
+
+for (int i = leftSubTreeEdges.nextSetBit(0); i >= 0; i =
leftSubTreeEdges.nextSetBit(i + 1)) {
+JoinEdge childA = joinEdges.get(i);
+if (!JoinType.isAssoc(childA.getJoinType(), edgeB.getJoinType())) {
+leftRequired = LongBitmap.newBitmapUnion(leftRequired,
childA.getLeftSubNodes(joinEdges));
+childA.addLeftRejectEdge(edgeB);
+}
+if (!JoinType.isLAssoc(childA.getJoinType(), edgeB.getJoinType()))
{
+leftRequired = LongBitmap.newBitmapUnion(leftRequired,
childA.getRightSubNodes(joinEdges));
+childA.addLeftRejectEdge(edgeB)
(doris) branch master updated: [fix](Nereids): fix shape of eliminate_inner regression test (#29613)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 787a234e051 [fix](Nereids): fix shape of eliminate_inner regression
test (#29613)
787a234e051 is described below
commit 787a234e051136f09f0d178a3bd6484ec2a7e58c
Author: 谢健
AuthorDate: Mon Jan 8 15:27:53 2024 +0800
[fix](Nereids): fix shape of eliminate_inner regression test (#29613)
---
regression-test/suites/nereids_rules_p0/pkfk/eliminate_inner.groovy | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/regression-test/suites/nereids_rules_p0/pkfk/eliminate_inner.groovy
b/regression-test/suites/nereids_rules_p0/pkfk/eliminate_inner.groovy
index 30a96f5405c..64a530cc225 100644
--- a/regression-test/suites/nereids_rules_p0/pkfk/eliminate_inner.groovy
+++ b/regression-test/suites/nereids_rules_p0/pkfk/eliminate_inner.groovy
@@ -18,7 +18,7 @@
suite("eliminate_inner") {
sql "SET enable_nereids_planner=true"
sql "SET enable_fallback_to_original_planner=false"
-sql "SET ignore_shape_nodes='PhysicalDistribute,PhysicalProject'"
+sql "SET ignore_shape_nodes='PhysicalDistribute[DistributionSpecGather],
PhysicalDistribute[DistributionSpecHash],PhysicalDistribute[DistributionSpecExecutionAny],PhysicalProject'"
sql "SET disable_join_reorder=true"
sql """
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [test](Nereids): add more test for eliminate inner join by fk (#29390)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 77fbbf63ed4 [test](Nereids): add more test for eliminate inner join by fk (#29390) 77fbbf63ed4 is described below commit 77fbbf63ed4b5f0ed5f2bda88d228222d6925851 Author: 谢健 AuthorDate: Fri Jan 5 16:21:24 2024 +0800 [test](Nereids): add more test for eliminate inner join by fk (#29390) --- .../data/nereids_rules_p0/pkfk/eliminate_inner.out | 309 + .../nereids_rules_p0/pkfk/eliminate_inner.groovy | 113 2 files changed, 422 insertions(+) diff --git a/regression-test/data/nereids_rules_p0/pkfk/eliminate_inner.out b/regression-test/data/nereids_rules_p0/pkfk/eliminate_inner.out new file mode 100644 index 000..bc465978ccb --- /dev/null +++ b/regression-test/data/nereids_rules_p0/pkfk/eliminate_inner.out @@ -0,0 +1,309 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !name -- +simple_case + +-- !shape -- +PhysicalResultSink +--hashJoin[INNER_JOIN] hashCondition=((pkt.pk = fkt_not_null.fk)) otherCondition=() +PhysicalOlapScan[pkt] +PhysicalOlapScan[fkt_not_null] + +-- !res -- + +-- !name -- +with_pk_col + +-- !shape -- +PhysicalResultSink +--hashJoin[INNER_JOIN] hashCondition=((pkt.pk = fkt_not_null.fk)) otherCondition=() +PhysicalOlapScan[pkt] +PhysicalOlapScan[fkt_not_null] + +-- !res -- + +-- !name -- +with_pk_col + +-- !shape -- +PhysicalResultSink +--hashJoin[INNER_JOIN] hashCondition=((pkt.pk = fkt_not_null.fk)) otherCondition=() +PhysicalOlapScan[pkt] +hashJoin[INNER_JOIN] hashCondition=((fkt_not_null1.fk = fkt_not_null2.fk)) otherCondition=() +--PhysicalOlapScan[fkt_not_null] +--PhysicalOlapScan[fkt_not_null] + +-- !res -- + +-- !name -- +with_pk_col + +-- !shape -- +PhysicalResultSink +--hashJoin[INNER_JOIN] hashCondition=((pkt.pk = fkt_not_null.fk)) otherCondition=() build RFs:RF1 fk->[pk] +filter((pkt.pk > 1)) +--PhysicalOlapScan[pkt] apply RFs: RF1 +hashJoin[INNER_JOIN] hashCondition=((fkt_not_null1.fk = fkt_not_null2.fk)) otherCondition=() build RFs:RF0 fk->[fk] +--filter((fkt_not_null1.fk > 1)) +PhysicalOlapScan[fkt_not_null] apply RFs: RF0 +--filter((fkt_not_null2.fk > 1)) +PhysicalOlapScan[fkt_not_null] + +-- !res -- + +-- !name -- +with_pk_col + +-- !shape -- +PhysicalResultSink +--hashJoin[INNER_JOIN] hashCondition=((pkt.pk = fkt_not_null.fk)) otherCondition=() +PhysicalOlapScan[pkt] +hashAgg[LOCAL] +--PhysicalOlapScan[fkt_not_null] + +-- !res -- + +-- !name -- +with_pk_col + +-- !shape -- +PhysicalResultSink +--hashJoin[INNER_JOIN] hashCondition=((pkt.pk = fkt_not_null.fk)) otherCondition=() +PhysicalOlapScan[pkt] +hashAgg[GLOBAL] +--hashAgg[LOCAL] +PhysicalUnion +--PhysicalOlapScan[fkt_not_null] +--PhysicalOlapScan[fkt_not_null] + +-- !res -- + +-- !name -- +fk with window + +-- !shape -- +PhysicalResultSink +--hashJoin[INNER_JOIN] hashCondition=((pkt.pk = fkt_not_null.fk)) otherCondition=() +PhysicalOlapScan[pkt] +PhysicalWindow +--PhysicalQuickSort[LOCAL_SORT] +PhysicalOlapScan[fkt_not_null] + +-- !res -- + +-- !name -- +fk with limit + +-- !shape -- +PhysicalResultSink +--hashJoin[INNER_JOIN] hashCondition=((pkt.pk = fkt_not_null.fk)) otherCondition=() build RFs:RF0 fk->[pk] +PhysicalOlapScan[pkt] apply RFs: RF0 +PhysicalLimit[GLOBAL] +--PhysicalLimit[LOCAL] +PhysicalOlapScan[fkt_not_null] + +-- !res -- + +-- !name -- +pk with filter that same as fk + +-- !shape -- +PhysicalResultSink +--hashJoin[INNER_JOIN] hashCondition=((pkt.pk = fkt_not_null.fk)) otherCondition=() build RFs:RF0 fk->[pk] +filter((pkt.pk = 1)) +--PhysicalOlapScan[pkt] apply RFs: RF0 +filter((fkt_not_null.fk = 1)) +--PhysicalOlapScan[fkt_not_null] + +-- !res -- + +-- !name -- +pk with filter that included same as fk + +-- !shape -- +PhysicalResultSink +--hashJoin[INNER_JOIN] hashCondition=((pkt.pk = fkt_not_null.fk)) otherCondition=() build RFs:RF0 fk->[pk] +filter((pkt.pk = 1)) +--PhysicalOlapScan[pkt] apply RFs: RF0 +filter((cast(f as DOUBLE) = 1) and (fkt_not_null.fk = 1)) +--PhysicalOlapScan[fkt_not_null] + +-- !res -- + +-- !name -- +pk with filter that not same as fk + +-- !shape -- +PhysicalResultSink +--hashJoin[INNER_JOIN] hashCondition=((pkt.pk = fkt_not_null.fk)) otherCondition=() build RFs:RF0 fk->[pk] +filter((cast(p as DOUBLE) = 1) and (pkt.pk = 1)) +--PhysicalOlapScan[pkt] apply RFs: RF0 +filter((cast(f as DOUBLE) = 1) and (fkt_not_null.fk = 1)) +--PhysicalOlapScan[fkt_not_null] + +-- !res -- + +-- !name -- +simple_case + +-- !shape -- +PhysicalResultSink +--filter(( not fk IS NULL)) +PhysicalOlapScan[fkt] +
(doris) branch master updated (d86277c2c22 -> 0eee560f945)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from d86277c2c22 [optimize](cooldown)Improved filtering conditions for cooldown tablet (#27338) add 0eee560f945 [enhancement](Nereids): add test for push filter through operator (#27294) No new revisions were added by this update. Summary of changes: .../filter_push_down/push_filter_through.out | 406 + .../filter_push_down/push_filter_through.groovy| 279 ++ 2 files changed, 685 insertions(+) create mode 100644 regression-test/data/nereids_rules_p0/filter_push_down/push_filter_through.out create mode 100644 regression-test/suites/nereids_rules_p0/filter_push_down/push_filter_through.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [minor](Nereids): cleanup code (#29413)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f2fa62f5242 [minor](Nereids): cleanup code (#29413)
f2fa62f5242 is described below
commit f2fa62f5242ba45de9b22cfb0b271af79609e4df
Author: jakevin
AuthorDate: Tue Jan 2 20:13:59 2024 +0800
[minor](Nereids): cleanup code (#29413)
---
.../glue/translator/PhysicalPlanTranslator.java| 3 ++-
.../doris/nereids/properties/DistributionSpec.java | 1 -
.../trees/plans/physical/PhysicalDistribute.java | 4 ++--
.../org/apache/doris/planner/EmptySetNode.java | 1 -
.../java/org/apache/doris/planner/PlanNode.java| 22 --
.../org/apache/doris/nereids/util/PlanChecker.java | 1 -
6 files changed, 4 insertions(+), 28 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
index 6ad21d34890..f8b23935318 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
@@ -313,10 +313,11 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor distribution = new PhysicalDistribute<>(
this,
-child.getLogicalProperties(),
new GroupPlan(child));
return new GroupExpression(distribution, Lists.newArrayList(child));
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalDistribute.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalDistribute.java
index e3b547dd884..d62225cdcd0 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalDistribute.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalDistribute.java
@@ -52,8 +52,8 @@ public class PhysicalDistribute
extends PhysicalUnary
groupExpression,
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/EmptySetNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/EmptySetNode.java
index e4b3cfd2264..867c220d9fe 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/EmptySetNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/EmptySetNode.java
@@ -43,7 +43,6 @@ public class EmptySetNode extends PlanNode {
public EmptySetNode(PlanNodeId id, ArrayList tupleIds) {
super(id, tupleIds, "EMPTYSET", StatisticalType.EMPTY_SET_NODE);
cardinality = 0L;
-offset = 0;
Preconditions.checkArgument(tupleIds.size() > 0);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java
index a61e214f05f..3f32ee59060 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java
@@ -28,15 +28,12 @@ import org.apache.doris.analysis.Expr;
import org.apache.doris.analysis.ExprId;
import org.apache.doris.analysis.ExprSubstitutionMap;
import org.apache.doris.analysis.FunctionCallExpr;
-import org.apache.doris.analysis.FunctionName;
import org.apache.doris.analysis.SlotId;
import org.apache.doris.analysis.SlotRef;
import org.apache.doris.analysis.TupleDescriptor;
import org.apache.doris.analysis.TupleId;
import org.apache.doris.catalog.Column;
-import org.apache.doris.catalog.Function;
import org.apache.doris.catalog.OlapTable;
-import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.NotImplementedException;
import org.apache.doris.common.TreeNode;
@@ -45,7 +42,6 @@ import org.apache.doris.statistics.PlanStats;
import org.apache.doris.statistics.StatisticalType;
import org.apache.doris.statistics.StatsDeriveResult;
import org.apache.doris.thrift.TExplainLevel;
-import org.apache.doris.thrift.TFunctionBinaryType;
import org.apache.doris.thrift.TPlan;
import org.apache.doris.thrift.TPlanNode;
import org.apache.doris.thrift.TPushAggOp;
@@ -405,24 +401,6 @@ public abstract class PlanNode extends TreeNode
implements PlanStats {
return statsDeriveResultList;
}
-protected void initCompoundPredicate(Expr expr) {
-if (expr instanceof CompoundPredicate) {
-CompoundPredicate compoundPredicate = (CompoundPredicate) expr;
-compoundPredicate.setType(Type.BOOLEAN);
-List args = new ArrayList<>();
-args.add(Type.BOOLEAN);
-args.add(Type.BOOLEAN);
-Function function = new Function(new FunctionName("",
compoundPredicate.getOp().toS
(doris) branch master updated: [feature](Nereids): eliminate left outer join by unique (#28853)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c07e3b7b47d [feature](Nereids): eliminate left outer join by unique
(#28853)
c07e3b7b47d is described below
commit c07e3b7b47d98ab77569258d6bea01e3fa26ba3b
Author: 谢健
AuthorDate: Tue Jan 2 15:50:49 2024 +0800
[feature](Nereids): eliminate left outer join by unique (#28853)
---
.../doris/nereids/jobs/executor/Rewriter.java | 7 +-
.../org/apache/doris/nereids/rules/RuleType.java | 1 +
.../rules/rewrite/EliminateJoinByUnique.java | 48 +++
.../rules/rewrite/EliminateJoinByUniqueTest.java | 97 ++
4 files changed, 152 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
index 4fe9df664c3..a97bd61e5c6 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
@@ -57,6 +57,7 @@ import
org.apache.doris.nereids.rules.rewrite.EliminateEmptyRelation;
import org.apache.doris.nereids.rules.rewrite.EliminateFilter;
import org.apache.doris.nereids.rules.rewrite.EliminateGroupBy;
import org.apache.doris.nereids.rules.rewrite.EliminateJoinByFK;
+import org.apache.doris.nereids.rules.rewrite.EliminateJoinByUnique;
import org.apache.doris.nereids.rules.rewrite.EliminateJoinCondition;
import org.apache.doris.nereids.rules.rewrite.EliminateLimit;
import org.apache.doris.nereids.rules.rewrite.EliminateNotNull;
@@ -295,7 +296,11 @@ public class Rewriter extends AbstractBatchJobExecutor {
),
// this rule should invoke after infer predicate and push down
distinct, and before push down limit
-custom(RuleType.ELIMINATE_JOIN_BY_FOREIGN_KEY,
EliminateJoinByFK::new),
+topic("eliminate join according unique or foreign key",
+custom(RuleType.ELIMINATE_JOIN_BY_FOREIGN_KEY,
EliminateJoinByFK::new),
+topDown(new EliminateJoinByUnique())
+),
+
// this rule should be after topic "Column pruning and infer
predicate"
topic("Join pull up",
topDown(
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
index 32794946dda..57e62380137 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
@@ -208,6 +208,7 @@ public enum RuleType {
ELIMINATE_UNNECESSARY_PROJECT(RuleTypeClass.REWRITE),
ELIMINATE_OUTER_JOIN(RuleTypeClass.REWRITE),
ELIMINATE_GROUP_BY(RuleTypeClass.REWRITE),
+ELIMINATE_JOIN_BY_UK(RuleTypeClass.REWRITE),
ELIMINATE_DEDUP_JOIN_CONDITION(RuleTypeClass.REWRITE),
ELIMINATE_NULL_AWARE_LEFT_ANTI_JOIN(RuleTypeClass.REWRITE),
ELIMINATE_ASSERT_NUM_ROWS(RuleTypeClass.REWRITE),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateJoinByUnique.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateJoinByUnique.java
new file mode 100644
index 000..6dba90572d5
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateJoinByUnique.java
@@ -0,0 +1,48 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.NullSafeEqual;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+
+/**
+ * Eliminate outer join.
+ */
+public class EliminateJoinByUnique extends OneRewriteRuleFactory {
+@Override
+public Rule build() {
+return logicalProject(
+logicalJoin(
(doris) branch master updated: [enhancement](Nereids): refactor eliminating inner join by foreign key (#28816)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 2794427e7f3 [enhancement](Nereids): refactor eliminating inner join by
foreign key (#28816)
2794427e7f3 is described below
commit 2794427e7f3a020e1131a433739c3e638bd61381
Author: 谢健
AuthorDate: Fri Dec 29 13:41:54 2023 +0800
[enhancement](Nereids): refactor eliminating inner join by foreign key
(#28816)
---
.../nereids/rules/rewrite/EliminateJoinByFK.java | 246 ++---
.../nereids/util/ImmutableEquivalenceSet.java | 20 +-
.../rules/rewrite/EliminateJoinByFkTest.java | 102 +++--
.../shape/query23.out | 41 ++--
.../noStatsRfPrune/query23.out | 48 ++--
.../no_stats_shape/query23.out | 48 ++--
.../rf_prune/query23.out | 41 ++--
.../nereids_tpcds_shape_sf100_p0/shape/query23.out | 41 ++--
8 files changed, 300 insertions(+), 287 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateJoinByFK.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateJoinByFK.java
index 17fcf7842ef..594dee50853 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateJoinByFK.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateJoinByFK.java
@@ -29,24 +29,17 @@ import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.expressions.SlotReference;
import org.apache.doris.nereids.trees.expressions.functions.ExpressionTrait;
import org.apache.doris.nereids.trees.plans.Plan;
-import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
import org.apache.doris.nereids.trees.plans.logical.LogicalCatalogRelation;
import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
-import org.apache.doris.nereids.trees.plans.logical.LogicalLimit;
import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import org.apache.doris.nereids.trees.plans.logical.LogicalRelation;
-import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
-import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
-import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
import org.apache.doris.nereids.util.ImmutableEquivalenceSet;
-import com.google.common.collect.BiMap;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
-import com.google.common.collect.ImmutableMap.Builder;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Sets;
@@ -54,7 +47,6 @@ import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
-import java.util.Map.Entry;
import java.util.Set;
import java.util.stream.Collectors;
import javax.annotation.Nullable;
@@ -80,7 +72,9 @@ public class EliminateJoinByFK extends
DefaultPlanRewriter implement
@Override
public Plan visit(Plan plan, ForeignKeyContext context) {
Plan newPlan = visitChildren(this, plan, context);
-context.passThroughPlan(plan);
+// always expire primary key except filter, project and join.
+// always keep foreign key alive
+context.expirePrimaryKey(plan);
return newPlan;
}
@@ -97,35 +91,29 @@ public class EliminateJoinByFK extends
DefaultPlanRewriter implement
return relation;
}
-private boolean canEliminate(LogicalJoin join, BiMap
equalSlots, ForeignKeyContext context) {
+private boolean canEliminate(LogicalJoin join, Map
primaryToForeign,
+ForeignKeyContext context) {
if (!join.getOtherJoinConjuncts().isEmpty()) {
return false;
}
if (!join.getJoinType().isInnerJoin() &&
!join.getJoinType().isSemiJoin()) {
return false;
}
-return context.satisfyConstraint(equalSlots, join);
+return context.satisfyConstraint(primaryToForeign, join);
}
-private boolean isForeignKeyAndUnique(Plan plan,
-Set keySet, ForeignKeyContext context) {
-boolean unique = keySet.stream()
-.allMatch(s ->
plan.getLogicalProperties().getFunctionalDependencies().isUnique(s));
-return unique && context.isForeignKey(keySet);
-}
-
-private @Nullable Map
tryGetOutputToChildMap(Plan child,
-Set output, BiMap equalSlots) {
-
(doris) branch master updated: [feature](Nereids): support comparing mv with inferred predicate (#29132)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 48d41a8c8b7 [feature](Nereids): support comparing mv with inferred
predicate (#29132)
48d41a8c8b7 is described below
commit 48d41a8c8b76ce22a8a9235e92862b06a5171aa6
Author: 谢健
AuthorDate: Fri Dec 29 10:38:53 2023 +0800
[feature](Nereids): support comparing mv with inferred predicate (#29132)
---
.../rules/exploration/mv/ComparisonResult.java | 11
.../rules/exploration/mv/HyperGraphComparator.java | 16 +++--
.../joinorder/hypergraph/CompareOuterJoinTest.java | 2 -
.../joinorder/hypergraph/InferPredicateTest.java | 72 ++
.../joinorder/hypergraph/PullupExpressionTest.java | 3 +-
.../agg_with_roll_up/aggregate_with_roll_up.groovy | 2 +-
.../aggregate_without_roll_up.groovy | 2 +-
.../mv/join/inner/inner_join.groovy| 2 +-
.../mv/join/left_outer/outer_join.groovy | 2 +-
.../mv/partition_mv_rewrite.groovy | 2 +-
10 files changed, 101 insertions(+), 13 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/ComparisonResult.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/ComparisonResult.java
index 8836745465e..1eb49cbfc0e 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/ComparisonResult.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/ComparisonResult.java
@@ -116,4 +116,15 @@ public class ComparisonResult {
viewNoNullableSlotBuilder.build(), valid);
}
}
+
+@Override
+public String toString() {
+if (isInvalid()) {
+return "INVALID";
+}
+return String.format("viewExpressions: %s \n "
++ "queryExpressions :%s \n "
++ "viewNoNullableSlot :%s \n",
+viewExpressions, queryExpressions, viewNoNullableSlot);
+}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/HyperGraphComparator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/HyperGraphComparator.java
index a869fe729a9..04efecc9c25 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/HyperGraphComparator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/HyperGraphComparator.java
@@ -27,6 +27,7 @@ import
org.apache.doris.nereids.rules.rewrite.PushDownFilterThroughJoin;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.util.ExpressionUtils;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
@@ -41,6 +42,7 @@ import java.util.Map;
import java.util.Map.Entry;
import java.util.Optional;
import java.util.Set;
+import java.util.stream.Collectors;
/**
* HyperGraphComparator
@@ -108,16 +110,22 @@ public class HyperGraphComparator {
private ComparisonResult buildComparisonRes() {
ComparisonResult.Builder builder = new ComparisonResult.Builder();
for (Entry> e :
pullUpQueryExprWithEdge.entrySet()) {
-if (!e.getValue().isEmpty() && !canPullUp(e.getKey())) {
+List rawFilter = e.getValue().stream()
+.filter(expr -> !ExpressionUtils.isInferred(expr))
+.collect(Collectors.toList());
+if (!rawFilter.isEmpty() && !canPullUp(e.getKey())) {
return ComparisonResult.INVALID;
}
-builder.addQueryExpressions(e.getValue());
+builder.addQueryExpressions(rawFilter);
}
for (Entry> e :
pullUpViewExprWithEdge.entrySet()) {
-if (!e.getValue().isEmpty() && !canPullUp(e.getKey())) {
+List rawFilter = e.getValue().stream()
+.filter(expr -> !ExpressionUtils.isInferred(expr))
+.collect(Collectors.toList());
+if (!rawFilter.isEmpty() && !canPullUp(e.getKey())) {
return ComparisonResult.INVALID;
}
-builder.addViewExpressions(e.getValue());
+builder.addViewExpressions(rawFilter);
}
for (Pair> inferredCond :
inferredViewEdgeMap.values()) {
builder.addViewNoNullableSlot(inferredCond.second);
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/CompareOuterJoinTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/CompareOuterJoinTest.java
index 105a5d
(doris) branch master updated (14c902b5041 -> 5171a77f9e2)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 14c902b5041 [fix](regression test) fix test_alter_colocate_table (#29009) add 5171a77f9e2 [fix](Nereids): merge Offset in Limit Translator (#29100) No new revisions were added by this update. Summary of changes: .../glue/translator/PhysicalPlanTranslator.java| 5 +- .../doris/nereids/rules/rewrite/MergeLimits.java | 16 ++- .../eliminate_not_null/eliminate_not_null.out | 63 - .../filter_push_down/push_filter_inside_join.out | 156 +++-- .../eliminate_not_null/eliminate_not_null.groovy | 2 + .../filter_push_through_aggregate.groovy | 4 +- .../push_filter_inside_join.groovy | 3 +- 7 files changed, 98 insertions(+), 151 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (bc08535285a -> f816d13c56a)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from bc08535285a [fix](Nereids) throw readable exception when meet unsupport sup-query (#29147) add f816d13c56a [feature](Nereids): eliminate groupby (#28615) No new revisions were added by this update. Summary of changes: .../doris/nereids/jobs/executor/Rewriter.java | 5 ++ .../nereids/properties/FunctionalDependencies.java | 3 +- .../org/apache/doris/nereids/rules/RuleType.java | 1 + .../nereids/rules/rewrite/EliminateGroupBy.java| 83 + .../rules/rewrite/EliminateGroupByTest.java| 100 + 5 files changed, 191 insertions(+), 1 deletion(-) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateGroupBy.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/EliminateGroupByTest.java - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (1c0bbcf4165 -> 42690415635)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 1c0bbcf4165 [fix](doc) typo fix in sql-interception docs (#28445) add 42690415635 [enhancement](Nereids): catch exception when calling getConstraintMap for external map (#29112) No new revisions were added by this update. Summary of changes: .../java/org/apache/doris/catalog/TableIf.java | 11 +- .../nereids/rules/rewrite/EliminateJoinByFK.java | 4 --- .../rules/rewrite/PullUpJoinFromUnionAll.java | 40 +- 3 files changed, 19 insertions(+), 36 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [feature](Nereids): support infer join when comapring mv (#28988)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 58e7ad82b50 [feature](Nereids): support infer join when comapring mv
(#28988)
58e7ad82b50 is described below
commit 58e7ad82b504897882b90d21cbe0aeef94186c7f
Author: 谢健
AuthorDate: Wed Dec 27 10:40:44 2023 +0800
[feature](Nereids): support infer join when comapring mv (#28988)
---
.../jobs/joinorder/hypergraph/HyperGraph.java | 171 +---
.../jobs/joinorder/hypergraph/edge/Edge.java | 40 ++-
.../jobs/joinorder/hypergraph/edge/JoinEdge.java | 12 +
.../rules/exploration/mv/ComparisonResult.java | 41 ++-
.../rules/exploration/mv/HyperGraphComparator.java | 299 +
.../nereids/rules/exploration/mv/StructInfo.java | 3 +-
.../nereids/trees/plans/logical/LogicalJoin.java | 5 +
.../joinorder/hypergraph/CompareOuterJoinTest.java | 25 +-
...ompareOuterJoinTest.java => InferJoinTest.java} | 105 +++-
.../joinorder/hypergraph/PullupExpressionTest.java | 9 +-
.../rules/exploration/mv/BuildStructInfoTest.java | 4 +-
11 files changed, 444 insertions(+), 270 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/HyperGraph.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/HyperGraph.java
index 7bd33c64b3c..f094efe7180 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/HyperGraph.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/HyperGraph.java
@@ -27,8 +27,6 @@ import
org.apache.doris.nereids.jobs.joinorder.hypergraph.node.DPhyperNode;
import org.apache.doris.nereids.jobs.joinorder.hypergraph.node.StructInfoNode;
import org.apache.doris.nereids.memo.Group;
import org.apache.doris.nereids.memo.GroupExpression;
-import org.apache.doris.nereids.rules.exploration.mv.ComparisonResult;
-import
org.apache.doris.nereids.rules.exploration.mv.LogicalCompatibilityContext;
import org.apache.doris.nereids.rules.rewrite.PushDownFilterThroughJoin;
import org.apache.doris.nereids.trees.expressions.Alias;
import org.apache.doris.nereids.trees.expressions.Expression;
@@ -44,20 +42,14 @@ import
org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import org.apache.doris.nereids.util.PlanUtils;
import com.google.common.base.Preconditions;
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Lists;
-import com.google.common.collect.Sets;
import java.util.ArrayList;
import java.util.BitSet;
import java.util.HashMap;
-import java.util.HashSet;
import java.util.List;
import java.util.Map;
-import java.util.Map.Entry;
-import java.util.Optional;
import java.util.Set;
import java.util.stream.Collectors;
@@ -272,11 +264,11 @@ public class HyperGraph {
filterEdges.forEach(e -> {
if (LongBitmap.isSubset(e.getReferenceNodes(), leftSubNodes)
&&
!PushDownFilterThroughJoin.COULD_PUSH_THROUGH_LEFT.contains(joinEdge.getJoinType()))
{
-e.addRejectEdge(joinEdge);
+e.addLeftRejectEdge(joinEdge);
}
if (LongBitmap.isSubset(e.getReferenceNodes(), rightSubNodes)
&&
!PushDownFilterThroughJoin.COULD_PUSH_THROUGH_RIGHT.contains(joinEdge.getJoinType()))
{
-e.addRejectEdge(joinEdge);
+e.addRightRejectEdge(joinEdge);
}
});
}
@@ -293,11 +285,11 @@ public class HyperGraph {
JoinEdge childA = joinEdges.get(i);
if (!JoinType.isAssoc(childA.getJoinType(), edgeB.getJoinType())) {
leftRequired = LongBitmap.newBitmapUnion(leftRequired,
childA.getLeftSubNodes(joinEdges));
-childA.addRejectEdge(edgeB);
+childA.addLeftRejectEdge(edgeB);
}
if (!JoinType.isLAssoc(childA.getJoinType(), edgeB.getJoinType()))
{
leftRequired = LongBitmap.newBitmapUnion(leftRequired,
childA.getRightSubNodes(joinEdges));
-childA.addRejectEdge(edgeB);
+childA.addLeftRejectEdge(edgeB);
}
}
@@ -305,11 +297,11 @@ public class HyperGraph {
JoinEdge childA = joinEdges.get(i);
if (!JoinType.isAssoc(edgeB.getJoinType(), childA.getJoinType())) {
rightRequired = LongBitmap.newBitmapUnion(rightRequired,
childA.getRightSubNodes(joinEdges));
-childA.addRejectEdge(edgeB);
+childA.addRightRejectEdge(edgeB);
}
if (!JoinType.isRAssoc(edgeB.getJoinType(), childA.getJoinType()))
{
(doris) branch master updated (e9848066c95 -> 425bbaa4fde)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from e9848066c95 [FIX](type) fix matchExactType for complex type (#28233) add 425bbaa4fde [test](Nereids): add assert debug log for TopnToMaxTest (#28755) No new revisions were added by this update. Summary of changes: regression-test/suites/nereids_p0/expression/topn_to_max.groovy | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (34fd376f33b -> 62833006d62)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 34fd376f33b [fix](publish version) fix publish fail but return ok (#28425) add 62833006d62 [test](Nereids): remove shape plan project and distribute in eager test (#28724) No new revisions were added by this update. Summary of changes: .../eliminate_join_condition.out | 72 +- .../limit_push_down/limit_push_down.out| 1218 +++- .../limit_push_down/order_push_down.out| 1028 ++--- .../push_down_limit_distinct.out | 26 +- .../push_down_top_n_distinct_through_union.out | 204 ++-- .../push_down_top_n_through_union.out | 267 ++--- .../eliminate_join_condition.groovy|2 + .../limit_push_down/limit_push_down.groovy |4 +- .../limit_push_down/order_push_down.groovy |3 +- .../push_down_limit_distinct.groovy|2 + .../push_down_top_n_distinct_through_union.groovy |2 + .../push_down_top_n_through_union.groovy |2 + 12 files changed, 973 insertions(+), 1857 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [test](Nereids): add test for scalar agg (#28712)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 2683b1e8c32 [test](Nereids): add test for scalar agg (#28712)
2683b1e8c32 is described below
commit 2683b1e8c32a3c0cd0d6dea61edf778c716aacd2
Author: jakevin
AuthorDate: Wed Dec 20 17:35:04 2023 +0800
[test](Nereids): add test for scalar agg (#28712)
---
.../PushDownFilterThroughAggregationTest.java | 33 +++---
1 file changed, 29 insertions(+), 4 deletions(-)
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushDownFilterThroughAggregationTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushDownFilterThroughAggregationTest.java
index 830921b9a2b..36cb8cee8d4 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushDownFilterThroughAggregationTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushDownFilterThroughAggregationTest.java
@@ -27,6 +27,8 @@ import
org.apache.doris.nereids.trees.expressions.NamedExpression;
import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.expressions.StatementScopeIdGenerator;
import org.apache.doris.nereids.trees.expressions.functions.agg.Max;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Sum;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.If;
import org.apache.doris.nereids.trees.expressions.literal.Literal;
import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
@@ -91,6 +93,28 @@ public class PushDownFilterThroughAggregationTest implements
MemoPatternMatchSup
);
}
+@Test
+void scalarAgg() {
+LogicalPlan plan = new LogicalPlanBuilder(scan)
+.agg(ImmutableList.of(), ImmutableList.of((new
Sum(scan.getOutput().get(0))).alias("sum")))
+.filter(new If(Literal.of(false), Literal.of(false),
Literal.of(false)))
+.project(ImmutableList.of(0))
+.build();
+
+PlanChecker.from(MemoTestUtils.createConnectContext(), plan)
+.applyTopDown(new PushDownFilterThroughAggregation())
+.printlnTree()
+.matches(
+logicalProject(
+logicalFilter(
+logicalAggregate(
+logicalOlapScan()
+)
+)
+)
+);
+}
+
/*-
* origin plan:
*project
@@ -174,7 +198,8 @@ public class PushDownFilterThroughAggregationTest
implements MemoPatternMatchSup
logicalAggregate(
logicalFilter(
logicalRepeat()
-).when(filter ->
filter.getConjuncts().equals(ImmutableSet.of(filterPredicateId)))
+).when(filter -> filter.getConjuncts()
+
.equals(ImmutableSet.of(filterPredicateId)))
)
)
);
@@ -195,9 +220,9 @@ public class PushDownFilterThroughAggregationTest
implements MemoPatternMatchSup
.matches(
logicalProject(
logicalFilter(
-logicalAggregate(
-logicalRepeat()
-)
+logicalAggregate(
+logicalRepeat()
+)
).when(filter ->
filter.getConjuncts().equals(ImmutableSet.of(filterPredicateId)))
)
);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [performance](Nereids): avoid use getStringValue() in getTimeFormatter (#28582)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 57a045c4052 [performance](Nereids): avoid use getStringValue() in
getTimeFormatter (#28582)
57a045c4052 is described below
commit 57a045c4052c2347698438b29500ea00f10c7d5b
Author: jakevin
AuthorDate: Tue Dec 19 17:17:19 2023 +0800
[performance](Nereids): avoid use getStringValue() in getTimeFormatter
(#28582)
Original `getTimeFormatter()` will convert `long` to `string`, and then
parse `string` to `int`.
(cherry picked from commit 646f1ea087acf5153596e86ef49e84246a68af6c)
---
.../org/apache/doris/analysis/DateLiteral.java | 22 +++---
1 file changed, 7 insertions(+), 15 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
index 811b8d257b7..29db8d13e90 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
@@ -1066,25 +1066,17 @@ public class DateLiteral extends LiteralExpr {
}
public LocalDateTime getTimeFormatter() {
-TemporalAccessor accessor;
if (type.equals(Type.DATE) || type.equals(Type.DATEV2)) {
-accessor = DATE_FORMATTER.parse(getStringValue());
+return LocalDateTime.of((int) this.year, (int) this.month, (int)
this.day, 0, 0, 0);
} else if (type.isDatetimeV2()) {
-accessor =
DATE_TIME_FORMATTER_TO_MICRO_SECOND.parse(getStringValue());
+return LocalDateTime.of((int) this.year, (int) this.month, (int)
this.day, (int) this.hour,
+(int) this.minute,
+(int) this.second, (int) this.microsecond * 1000);
} else {
-accessor = DATE_TIME_FORMATTER.parse(getStringValue());
+return LocalDateTime.of((int) this.year, (int) this.month, (int)
this.day, (int) this.hour,
+(int) this.minute,
+(int) this.second);
}
-final int year = accessor.get(ChronoField.YEAR);
-final int month = accessor.get(ChronoField.MONTH_OF_YEAR);
-final int dayOfMonth = accessor.get(ChronoField.DAY_OF_MONTH);
-final int hour = getOrDefault(accessor, ChronoField.HOUR_OF_DAY, 0);
-final int minute = getOrDefault(accessor, ChronoField.MINUTE_OF_HOUR,
0);
-final int second = getOrDefault(accessor,
ChronoField.SECOND_OF_MINUTE, 0);
-final int microSeconds = getOrDefault(accessor,
ChronoField.MICRO_OF_SECOND, 0);
-
-// LocalDateTime of(int year, int month, int dayOfMonth, int hour, int
minute,
-// int second, int nanoOfSecond)
-return LocalDateTime.of(year, month, dayOfMonth, hour, minute, second,
microSeconds * 1000);
}
public DateLiteral plusYears(int year) throws AnalysisException {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [chore](Nereids): remove SemiJoinCommute in rewriter (#28563)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ddcfba0368c [chore](Nereids): remove SemiJoinCommute in rewriter
(#28563)
ddcfba0368c is described below
commit ddcfba0368c9273cc202622318c0db851251f08c
Author: jakevin
AuthorDate: Mon Dec 18 20:47:17 2023 +0800
[chore](Nereids): remove SemiJoinCommute in rewriter (#28563)
---
.../main/java/org/apache/doris/nereids/jobs/executor/Analyzer.java| 4 ++--
.../main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java| 2 --
.../nereids/rules/rewrite/{JoinCommute.java => SemiJoinCommute.java} | 2 +-
.../java/org/apache/doris/nereids/rules/rewrite/ReorderJoinTest.java | 1 +
4 files changed, 4 insertions(+), 5 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Analyzer.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Analyzer.java
index b78cb093600..7f579f35336 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Analyzer.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Analyzer.java
@@ -43,7 +43,7 @@ import
org.apache.doris.nereids.rules.analysis.ReplaceExpressionByChildOutput;
import
org.apache.doris.nereids.rules.analysis.ResolveOrdinalInOrderByAndGroupBy;
import org.apache.doris.nereids.rules.analysis.SubqueryToApply;
import org.apache.doris.nereids.rules.analysis.UserAuthentication;
-import org.apache.doris.nereids.rules.rewrite.JoinCommute;
+import org.apache.doris.nereids.rules.rewrite.SemiJoinCommute;
import java.util.List;
import java.util.Objects;
@@ -124,7 +124,7 @@ public class Analyzer extends AbstractBatchJobExecutor {
bottomUp(new CheckAnalysis()),
topDown(new EliminateGroupByConstant()),
topDown(new NormalizeAggregate()),
-bottomUp(new JoinCommute()),
+bottomUp(new SemiJoinCommute()),
bottomUp(
new CollectSubQueryAlias(),
new CollectJoinConstraint()
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
index 10da21d7244..9104869387e 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
@@ -73,7 +73,6 @@ import
org.apache.doris.nereids.rules.rewrite.InferFilterNotNull;
import org.apache.doris.nereids.rules.rewrite.InferJoinNotNull;
import org.apache.doris.nereids.rules.rewrite.InferPredicates;
import org.apache.doris.nereids.rules.rewrite.InferSetOperatorDistinct;
-import org.apache.doris.nereids.rules.rewrite.JoinCommute;
import org.apache.doris.nereids.rules.rewrite.LimitSortToTopN;
import org.apache.doris.nereids.rules.rewrite.MergeFilters;
import org.apache.doris.nereids.rules.rewrite.MergeOneRowRelationIntoUnion;
@@ -229,7 +228,6 @@ public class Rewriter extends AbstractBatchJobExecutor {
),
// push down SEMI Join
bottomUp(
-new JoinCommute(),
new TransposeSemiJoinLogicalJoin(),
new TransposeSemiJoinLogicalJoinProject(),
new TransposeSemiJoinAgg(),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/JoinCommute.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SemiJoinCommute.java
similarity index 96%
rename from
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/JoinCommute.java
rename to
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SemiJoinCommute.java
index 4611851787c..66557535721 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/JoinCommute.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SemiJoinCommute.java
@@ -27,7 +27,7 @@ import org.apache.doris.qe.ConnectContext;
* RightAntiJoin -> LeftAntiJoin
* RightOuterJoin -> LeftOuterJoin
*/
-public class JoinCommute extends OneRewriteRuleFactory {
+public class SemiJoinCommute extends OneRewriteRuleFactory {
@Override
public Rule build() {
return logicalJoin()
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/ReorderJoinTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/ReorderJoinTest.java
index 9e7a1ed9ff7..770181c7ac3 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/ReorderJoinTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/ReorderJoinTest.java
@@ -124,6 +124,7 @@ class ReorderJoinTest implements MemoPatternMatch
(doris) branch master updated (868884145cd -> 9cf9d568c72)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 868884145cd [Opt](expr) Opt decimal comparsion expr. (#28551)
add 9cf9d568c72 [feature](Nereids): add is null in predicate and put or
expansion rule in rewriter (#28348)
No new revisions were added by this update.
Summary of changes:
.../org/apache/doris/nereids/jobs/executor/Rewriter.java | 3 +++
.../main/java/org/apache/doris/nereids/rules/RuleSet.java | 2 --
.../rules/{exploration => rewrite}/OrExpansion.java | 15 ++-
.../org/apache/doris/nereids/stats/FilterEstimation.java | 2 +-
regression-test/data/nereids_p0/union/or_expansion.out| 15 ++-
.../suites/nereids_p0/union/or_expansion.groovy | 6 ++
6 files changed, 34 insertions(+), 9 deletions(-)
rename fe/fe-core/src/main/java/org/apache/doris/nereids/rules/{exploration =>
rewrite}/OrExpansion.java (95%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (8986bb6bb4d -> 0f93ee87936)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 8986bb6bb4d [fix](Planner): parse more Punctuation Date/DateTime (#28432) add 0f93ee87936 [fix](Nereids): TransposeSemiJoinAgg can't apply in Scalar Agg (#28434) No new revisions were added by this update. Summary of changes: .../doris/nereids/rules/rewrite/TransposeSemiJoinAgg.java| 4 .../nereids/rules/rewrite/TransposeSemiJoinAggTest.java | 12 2 files changed, 16 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
