(doris) branch branch-2.0 updated (f6beec3a414 -> eca955577a8)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git from f6beec3a414 [fix](env) state listener avoid endless waiting #27881 (#41462) add eca955577a8 [fix](block_rule) SQL block rule not working after FE restart (#41228) (#41252) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/doris/common/Config.java | 4 .../org/apache/doris/blockrule/SqlBlockRule.java | 16 -- .../apache/doris/blockrule/SqlBlockRuleMgr.java| 5 + .../doris/blockrule/SqlBlockRuleMgrTest.java | 25 ++ 4 files changed, 44 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (ce982a959e0 -> b41927833f0)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from ce982a959e0 [Chore](materialized-view) mow table's mv need have all key column (#41496) add b41927833f0 [fix](Nereids) could not parse date/datetime with blank + zone (#41394) No new revisions were added by this update. Summary of changes: .../trees/expressions/literal/DateLiteral.java | 12 +++--- .../trees/expressions/literal/DateLiteralTest.java | 6 + .../nereids/util/DateTimeFormatterUtilsTest.java | 26 ++ 3 files changed, 41 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (bfccd0ebafc -> e8fb88f9167)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from bfccd0ebafc [fix](mtmv)Fix show mtmv time should wrapped in double quotation marks (#41419) add e8fb88f9167 [opt](compile) ensure fe-core compile with language level 8 (#41417) No new revisions were added by this update. Summary of changes: fe/fe-core/pom.xml | 3 +++ .../src/main/java/org/apache/doris/cloud/rpc/MetaServiceClient.java| 2 +- 2 files changed, 4 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch update-mysql-compatibility created (now 0ccf0eb3de7)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch update-mysql-compatibility in repository https://gitbox.apache.org/repos/asf/doris-website.git at 0ccf0eb3de7 Update mysql-compatibility.md This branch includes the following new commits: new 0ccf0eb3de7 Update mysql-compatibility.md The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (26bbfa13827 -> bfccd0ebafc)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 26bbfa13827 [fix](test)resolve table name conflict between test cases (#41492) add bfccd0ebafc [fix](mtmv)Fix show mtmv time should wrapped in double quotation marks (#41419) No new revisions were added by this update. Summary of changes: .../org/apache/doris/mtmv/MTMVRefreshSchedule.java | 3 ++- .../test/java/org/apache/doris/mtmv/MTMVTest.java | 2 +- .../suites/mtmv_p0/test_show_create_mtmv.groovy| 23 ++ 3 files changed, 26 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) 01/01: Update mysql-compatibility.md
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a commit to branch update-mysql-compatibility in repository https://gitbox.apache.org/repos/asf/doris-website.git commit 0ccf0eb3de760ba2dabc93c53801894040fa6f29 Author: morrySnow <101034200+morrys...@users.noreply.github.com> AuthorDate: Mon Sep 30 14:28:58 2024 +0800 Update mysql-compatibility.md --- .../version-2.1/query/query-data/mysql-compatibility.md| 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/versioned_docs/version-2.1/query/query-data/mysql-compatibility.md b/versioned_docs/version-2.1/query/query-data/mysql-compatibility.md index 77b7bf92d4d..d16b1c42287 100644 --- a/versioned_docs/version-2.1/query/query-data/mysql-compatibility.md +++ b/versioned_docs/version-2.1/query/query-data/mysql-compatibility.md @@ -117,13 +117,13 @@ Doris has several unique data types. Here are the details: The supported types for fields are `BOOLEAN`, `TINYINT`, `SMALLINT`, `INT`, `BIGINT`, `LARGEINT`, `FLOAT`, `DOUBLE`, `DECIMAL`, `DATE`, `DATETIME`, `CHAR`, `VARCHAR`, and `STRING`. -- Agg_State +- **Agg_State** AGG_STATE is a data type in Doris that cannot be used as a key column. During table creation, the signature of the aggregation function needs to be declared. The length and default value do not need to be specified, and the actual storage size depends on the implementation of the function. - AGG_STATE can only be used in combination with [state](../../sql-manual/sql-functions/combinators/state) /[merge](../../sql-manual/sql-functions/combinators/merge)/[union](../../sql-manual/sql-functions/combinators/union) functions from the SQL manual for aggregators. + AGG_STATE can only be used in combination with [state](../../sql-manual/sql-functions/combinators/state)/[merge](../../sql-manual/sql-functions/combinators/merge)/[union](../../sql-manual/sql-functions/combinators/union) functions from the SQL manual for aggregators. ## Syntax @@ -270,11 +270,11 @@ CREATE MATERIALIZED VIEW (IF NOT EXISTS)? mvName=multipartIdentifier The syntax of Doris ALTER is basically the same as that of MySQL. -### **Drop-Table/Drop-Index** +### Drop-Table/Drop-Index The syntax of Doris DROP is basically the same as MySQL. -### **DML** +### DML **Insert** @@ -327,7 +327,7 @@ DELETE FROM table_name [table_alias] This syntax can only be used on the UNIQUE KEY model table. -The Doris DELTE syntax is basically the same as MySQL. Due to Doris is an analytical database, deletions can't be too frequent. +The Doris DELETE syntax is basically the same as MySQL. Due to Doris is an analytical database, deletions can't be too frequent. **Select** - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](nereids)should prune logicalSink's child node in ColumnPruning (#41289)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 3c3a028913b [fix](nereids)should prune logicalSink's child node in
ColumnPruning (#41289)
3c3a028913b is described below
commit 3c3a028913b7ee1662cab23f20461ed7254c06f2
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Fri Sep 27 19:12:29 2024 +0800
[fix](nereids)should prune logicalSink's child node in ColumnPruning
(#41289)
---
.../main/java/org/apache/doris/nereids/rules/rewrite/ColumnPruning.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ColumnPruning.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ColumnPruning.java
index 20a91ca5657..d480c203f29 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ColumnPruning.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ColumnPruning.java
@@ -217,7 +217,7 @@ public class ColumnPruning extends
DefaultPlanRewriter implements
@Override
public Plan visitLogicalSink(LogicalSink logicalSink,
PruneContext context) {
-return skipPruneThisAndFirstLevelChildren(logicalSink);
+return pruneChildren(logicalSink, logicalSink.getOutputSet());
}
// the backend not support filter(project(agg)), so we can not prune the
key set in the agg,
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (0ef9ecdfffa -> c87c90421c3)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 0ef9ecdfffa [Fix](Nereids) fix fold const be return type (#41164) add c87c90421c3 [fix](Nereids) where of copy into should be optional (#41418) No new revisions were added by this update. Summary of changes: fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4 | 2 +- fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [Fix](nereids) fix create view with nullable column (#41234) (#41393)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 82228358b94 [Fix](nereids) fix create view with nullable column
(#41234) (#41393)
82228358b94 is described below
commit 82228358b943d42b7775de00c5ead7566393e6b5
Author: feiniaofeiafei <53502832+feiniaofeia...@users.noreply.github.com>
AuthorDate: Fri Sep 27 19:13:54 2024 +0800
[Fix](nereids) fix create view with nullable column (#41234) (#41393)
cherry-pick from master #41234
---
.../trees/plans/commands/info/BaseViewInfo.java| 5 ++-
.../data/correctness/test_view_varchar_length.out | 2 +-
.../data/ddl_p0/test_create_view_nereids.out | 10 +
.../suites/ddl_p0/test_create_view_nereids.groovy | 47 ++
4 files changed, 61 insertions(+), 3 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/BaseViewInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/BaseViewInfo.java
index 0cbaa167fdf..7d24e99b890 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/BaseViewInfo.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/BaseViewInfo.java
@@ -152,7 +152,8 @@ public class BaseViewInfo {
protected void createFinalCols(List outputs) throws
org.apache.doris.common.AnalysisException {
if (simpleColumnDefinitions.isEmpty()) {
for (Slot output : outputs) {
-Column column = new Column(output.getName(),
output.getDataType().toCatalogDataType());
+Column column = new Column(output.getName(),
output.getDataType().toCatalogDataType(),
+output.nullable());
finalCols.add(column);
}
} else {
@@ -161,7 +162,7 @@ public class BaseViewInfo {
}
for (int i = 0; i < simpleColumnDefinitions.size(); ++i) {
Column column = new
Column(simpleColumnDefinitions.get(i).getName(),
-outputs.get(i).getDataType().toCatalogDataType());
+outputs.get(i).getDataType().toCatalogDataType(),
outputs.get(i).nullable());
column.setComment(simpleColumnDefinitions.get(i).getComment());
finalCols.add(column);
}
diff --git a/regression-test/data/correctness/test_view_varchar_length.out
b/regression-test/data/correctness/test_view_varchar_length.out
index e53fe9ff97f..dd5128a3118 100644
--- a/regression-test/data/correctness/test_view_varchar_length.out
+++ b/regression-test/data/correctness/test_view_varchar_length.out
@@ -1,4 +1,4 @@
-- This file is automatically generated. You should know what you did if you
want to edit this
-- !sql --
-name varchar(32) No false \N
+name varchar(32) Yes false \N
diff --git a/regression-test/data/ddl_p0/test_create_view_nereids.out
b/regression-test/data/ddl_p0/test_create_view_nereids.out
index 8ee98fa7021..4cac00352a9 100644
--- a/regression-test/data/ddl_p0/test_create_view_nereids.out
+++ b/regression-test/data/ddl_p0/test_create_view_nereids.out
@@ -363,3 +363,13 @@ test_havingCREATE VIEW `test_having` AS select
sum(`internal`.`regression_test_
-- !complicated_view1_sql --
test_view_complicated CREATE VIEW `test_view_complicated` AS SELECT
`internal`.`regression_test_ddl_p0`.`t`.`id`,
`internal`.`regression_test_ddl_p0`.`t`.`value3`, `t`.`row_num` FROM (\n
SELECT `internal`.`regression_test_ddl_p0`.`t1`.`id`,
`internal`.`regression_test_ddl_p0`.`tt`.`value3`, ROW_NUMBER() OVER (PARTITION
BY `internal`.`regression_test_ddl_p0`.`t1`.`id` ORDER BY
`internal`.`regression_test_ddl_p0`.`tt`.`value3` DESC) as `row_num`\nFROM
(SELECT `internal`.`regress [...]
+-- !nullable --
+1 小区A 10 1 2024-09-01T09:002024-09-01T10:00
+2 小区B 11 1 2024-09-01T09:002024-09-01T10:00
+3 小区C \N 1 2024-09-01T09:002024-09-01T10:00
+
+-- !nullable_view_with_cols --
+1 小区A 10 1 2024-09-01T09:002024-09-01T10:00
+2 小区B 11 1 2024-09-01T09:002024-09-01T10:00
+3 小区C \N 1 2024-09-01T09:002024-09-01T10:00
+
diff --git a/regression-test/suites/ddl_p0/test_create_view_nereids.groovy
b/regression-test/suites/ddl_p0/test_create_view_nereids.groovy
index ad3c84f25ec..38347deffc6 100644
--- a/regression-test/suites/ddl_p0/test_create_view_nereids.groovy
+++ b/regression-test/suites/ddl_p0/test_create_view_nereids.groovy
@@ -390,4 +390,51 @@ suite("test_create_view_nereids") {
WHERE value3 < 280 AND (id < 3 or id >8);"""
qt_complicated_view1 &q
(doris) branch branch-2.0 updated: [opt](privilege) Grant check name (#39597) (#39858)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 40f021ee1a8 [opt](privilege) Grant check name (#39597) (#39858)
40f021ee1a8 is described below
commit 40f021ee1a804468b7efafb69113193ff4281a3e
Author: zhangdong <493738...@qq.com>
AuthorDate: Fri Sep 27 19:32:21 2024 +0800
[opt](privilege) Grant check name (#39597) (#39858)
pick from master #39597
---
.../org/apache/doris/mysql/privilege/Auth.java | 31 +++
.../apache/doris/datasource/CatalogMgrTest.java| 12 ++
.../auth_p0/test_grant_nonexist_table.groovy | 45 ++
.../ccr_mow_syncer_p0/test_get_binlog.groovy | 1 -
4 files changed, 79 insertions(+), 10 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Auth.java
b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Auth.java
index 7b130c1700a..2d2a84c3be2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Auth.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Auth.java
@@ -34,8 +34,10 @@ import org.apache.doris.analysis.SetUserPropertyStmt;
import org.apache.doris.analysis.TablePattern;
import org.apache.doris.analysis.UserIdentity;
import org.apache.doris.analysis.WorkloadGroupPattern;
+import org.apache.doris.catalog.DatabaseIf;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.InfoSchemaDb;
+import org.apache.doris.catalog.TableIf;
import org.apache.doris.cluster.ClusterNamespace;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.AuthenticationException;
@@ -51,6 +53,7 @@ import org.apache.doris.common.Pair;
import org.apache.doris.common.PatternMatcherException;
import org.apache.doris.common.UserException;
import org.apache.doris.common.io.Writable;
+import org.apache.doris.datasource.CatalogIf;
import org.apache.doris.datasource.InternalCatalog;
import org.apache.doris.ldap.LdapManager;
import org.apache.doris.ldap.LdapUserInfo;
@@ -81,6 +84,7 @@ import java.io.IOException;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
+import java.util.Objects;
import java.util.Set;
import java.util.concurrent.locks.ReentrantReadWriteLock;
import java.util.stream.Collectors;
@@ -578,6 +582,7 @@ public class Auth implements Writable {
throws DdlException {
writeLock();
try {
+checkTablePatternExist(tblPattern);
if (role == null) {
if (!doesUserExist(userIdent)) {
throw new DdlException("user " + userIdent + " does not
exist");
@@ -596,6 +601,32 @@ public class Auth implements Writable {
}
}
+private void checkTablePatternExist(TablePattern tablePattern) throws
DdlException {
+Objects.requireNonNull(tablePattern, "tablePattern can not be null");
+PrivLevel privLevel = tablePattern.getPrivLevel();
+if (privLevel == PrivLevel.GLOBAL) {
+return;
+}
+CatalogIf catalog =
Env.getCurrentEnv().getCatalogMgr().getCatalog(tablePattern.getQualifiedCtl());
+if (catalog == null) {
+throw new DdlException("catalog:" + tablePattern.getQualifiedCtl()
+ " does not exist");
+}
+if (privLevel == PrivLevel.CATALOG) {
+return;
+}
+DatabaseIf db = catalog.getDbNullable(tablePattern.getQualifiedDb());
+if (db == null) {
+throw new DdlException("database:" + tablePattern.getQualifiedDb()
+ " does not exist");
+}
+if (privLevel == PrivLevel.DATABASE) {
+return;
+}
+TableIf table = db.getTableNullable(tablePattern.getTbl());
+if (table == null) {
+throw new DdlException("table:" + tablePattern.getTbl() + " does
not exist");
+}
+}
+
// grant for ResourcePattern
private void grantInternal(UserIdentity userIdent, String role,
ResourcePattern resourcePattern, PrivBitSet privs,
boolean errOnNonExist, boolean isReplay) throws DdlException {
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/CatalogMgrTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/CatalogMgrTest.java
index 95d1826017d..4a3dd818790 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/CatalogMgrTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/CatalogMgrTest.java
@@ -101,13 +101,6 @@ public class CatalogMgrTest extends TestWithFeService {
// grant with no catalog is switched, internal catalog works.
CreateRoleStmt createRole1 = (CreateRoleStmt)
parseAndAnalyzeStmt("create role role1;", rootCt
(doris) branch master updated: [fix](test)resolve table name conflict in test case (#41373)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 43dd4322a51 [fix](test)resolve table name conflict in test case
(#41373)
43dd4322a51 is described below
commit 43dd4322a51935d04d8d9240d35cbc03e35c621c
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Fri Sep 27 19:11:11 2024 +0800
[fix](test)resolve table name conflict in test case (#41373)
---
.../suites/datatype_p0/decimalv3/test_decimalv3_where.groovy | 12 ++--
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git
a/regression-test/suites/datatype_p0/decimalv3/test_decimalv3_where.groovy
b/regression-test/suites/datatype_p0/decimalv3/test_decimalv3_where.groovy
index d998b7fddc3..c67fd3d42cc 100644
--- a/regression-test/suites/datatype_p0/decimalv3/test_decimalv3_where.groovy
+++ b/regression-test/suites/datatype_p0/decimalv3/test_decimalv3_where.groovy
@@ -24,21 +24,21 @@ suite("test_decimalv3_where") {
sql """insert into ${tableName} values
(1,'doris',20,324.10),(2,'spark',10,95.5),(3,'flink',9,20)"""
qt_decimalv3 "select * from ${tableName} where dr != 1 order by age;"
-sql """drop table if exists test_sys_update_basic_test_update_decimal_tb"""
-sql """CREATE TABLE test_sys_update_basic_test_update_decimal_tb (
+sql """drop table if exists test_sys_update_basic_test__decimal_tb"""
+sql """CREATE TABLE test_sys_update_basic_test__decimal_tb (
k1 DECIMAL(10, 5) NULL,
v1 DECIMAL(10, 5) NULL
) UNIQUE KEY(k1) DISTRIBUTED BY HASH(k1) BUCKETS 5 PROPERTIES (
"replication_allocation" = "tag.location.default: 1"
);"""
-sql """insert into test_sys_update_basic_test_update_decimal_tb values
+sql """insert into test_sys_update_basic_test__decimal_tb values
(1.001, 2.002), (1.002, 0.0002), (1.003, 0.10001),
(1.004, 0.100044001), (1.005, 0.100045001);"""
-qt_select """select * from test_sys_update_basic_test_update_decimal_tb
where k1 = 1.001;"""
+qt_select """select * from test_sys_update_basic_test__decimal_tb where k1
= 1.001;"""
sql """
-UPDATE test_sys_update_basic_test_update_decimal_tb SET v1=0.0001
WHERE k1 = 1.001;
+UPDATE test_sys_update_basic_test__decimal_tb SET v1=0.0001 WHERE k1 =
1.001;
"""
qt_select_after_update """
-select * from test_sys_update_basic_test_update_decimal_tb where k1 =
1.001;
+select * from test_sys_update_basic_test__decimal_tb where k1 = 1.001;
"""
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [fix](meta) fix Unknown column 'mva_SUM__CAST` (#41284)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 2df6c8418f5 [fix](meta) fix Unknown column 'mva_SUM__CAST` (#41284)
2df6c8418f5 is described below
commit 2df6c8418f5092a6a4d7f1da93c77fdd34a449d3
Author: 924060929 <924060...@qq.com>
AuthorDate: Fri Sep 27 18:01:21 2024 +0800
[fix](meta) fix Unknown column 'mva_SUM__CAST` (#41284)
cherry pick from master #41283
---
.../apache/doris/catalog/MaterializedIndexMeta.java | 19 +++
1 file changed, 19 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/MaterializedIndexMeta.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/MaterializedIndexMeta.java
index 23b0a353366..4ab0c536aec 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/catalog/MaterializedIndexMeta.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/catalog/MaterializedIndexMeta.java
@@ -18,6 +18,7 @@
package org.apache.doris.catalog;
import org.apache.doris.analysis.Analyzer;
+import org.apache.doris.analysis.CastExpr;
import org.apache.doris.analysis.CreateMaterializedViewStmt;
import org.apache.doris.analysis.Expr;
import org.apache.doris.analysis.SlotRef;
@@ -202,6 +203,24 @@ public class MaterializedIndexMeta implements Writable,
GsonPostProcessable {
}
}
+boolean isCastSlot =
+entry.getValue() instanceof CastExpr &&
entry.getValue().getChild(0) instanceof SlotRef;
+
+// Compatibility code for older versions of mv
+// old version:
+// goods_number -> mva_SUM__CAST(`goods_number` AS BIGINT)
+// new version:
+// goods_number -> mva_SUM__CAST(`goods_number` AS bigint)
+if (isCastSlot && !match) {
+for (Column column : schema) {
+if (column.getName().equalsIgnoreCase(entry.getKey())) {
+column.setDefineExpr(entry.getValue());
+match = true;
+break;
+}
+}
+}
+
if (!match) {
// Compatibility code for older versions of mv
// store_id -> mv_store_id
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [chore](planner) remove useless legacy planner parse in ConnectProcessor (#41248)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 46d3e90dc9e [chore](planner) remove useless legacy planner parse in
ConnectProcessor (#41248)
46d3e90dc9e is described below
commit 46d3e90dc9e459b2467afc5c61296b069e2dff86
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Thu Sep 26 14:58:15 2024 +0800
[chore](planner) remove useless legacy planner parse in ConnectProcessor
(#41248)
---
.../java/org/apache/doris/qe/ConnectProcessor.java | 35 --
1 file changed, 35 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
index 3a1e95ad6d3..42724eebbe1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
@@ -22,8 +22,6 @@ import org.apache.doris.analysis.InsertStmt;
import org.apache.doris.analysis.KillStmt;
import org.apache.doris.analysis.LiteralExpr;
import org.apache.doris.analysis.QueryStmt;
-import org.apache.doris.analysis.SqlParser;
-import org.apache.doris.analysis.SqlScanner;
import org.apache.doris.analysis.StatementBase;
import org.apache.doris.analysis.UserIdentity;
import org.apache.doris.catalog.Column;
@@ -43,7 +41,6 @@ import org.apache.doris.common.NotImplementedException;
import org.apache.doris.common.Pair;
import org.apache.doris.common.UserException;
import org.apache.doris.common.util.DebugUtil;
-import org.apache.doris.common.util.SqlParserUtils;
import org.apache.doris.common.util.SqlUtils;
import org.apache.doris.common.util.Util;
import org.apache.doris.datasource.CatalogIf;
@@ -87,7 +84,6 @@ import org.apache.logging.log4j.Logger;
import org.apache.thrift.TException;
import java.io.IOException;
-import java.io.StringReader;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.List;
@@ -492,37 +488,6 @@ public abstract class ConnectProcessor {
auditAfterExec(origStmt, parsedStmt, statistics, true);
}
-// analyze the origin stmt and return multi-statements
-protected List parse(String originStmt) throws
AnalysisException, DdlException {
-if (LOG.isDebugEnabled()) {
-LOG.debug("the originStmts are: {}", originStmt);
-}
-// Parse statement with parser generated by CUP&FLEX
-SqlScanner input = new SqlScanner(new StringReader(originStmt),
ctx.getSessionVariable().getSqlMode());
-SqlParser parser = new SqlParser(input);
-try {
-return SqlParserUtils.getMultiStmts(parser);
-} catch (Error e) {
-throw new AnalysisException("Please check your sql, we meet an
error when parsing.", e);
-} catch (AnalysisException | DdlException e) {
-String errorMessage = parser.getErrorMsg(originStmt);
-if (LOG.isDebugEnabled()) {
-LOG.debug("origin stmt: {}; Analyze error message: {}",
originStmt, parser.getErrorMsg(originStmt), e);
-}
-if (errorMessage == null) {
-throw e;
-} else {
-throw new AnalysisException(errorMessage, e);
-}
-} catch (ArrayStoreException e) {
-throw new AnalysisException("Sql parser can't convert the result
to array, please check your sql.", e);
-} catch (Exception e) {
-// TODO(lingbin): we catch 'Exception' to prevent unexpected error,
-// should be removed this try-catch clause future.
-throw new AnalysisException("Internal Error, maybe syntax error or
this is a bug: " + e.getMessage(), e);
-}
-}
-
// Get the column definitions of a table
@SuppressWarnings("rawtypes")
protected void handleFieldList(String tableName) throws
ConnectionException {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [improvement](nereids) Simplify ScanNode projection handling by removing redundant conditions (#40801) (#41315)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 4deda2fce73 [improvement](nereids) Simplify ScanNode projection
handling by removing redundant conditions (#40801) (#41315)
4deda2fce73 is described below
commit 4deda2fce738db0a6c0db0a0f9b430e61a267d41
Author: zy-kkk
AuthorDate: Thu Sep 26 10:35:01 2024 +0800
[improvement](nereids) Simplify ScanNode projection handling by removing
redundant conditions (#40801) (#41315)
pick from master #40801
This PR simplifies the handling of `ScanNode` projection logic.
Previously, the code included multiple conditional checks to determine
whether a `projectionTuple` should be generated. These conditions have
been removed, and now `projectionTuple `is always generated for
`ScanNode`, ensuring a consistent projection setup. Additionally,
redundant handling of `SlotId` and `SlotRef` has been eliminated, making
the code cleaner and easier to maintain. The behavior for `OlapScanNode`
remains unchanged.
---
.../glue/translator/PhysicalPlanTranslator.java| 49 +-
.../jdbc/test_doris_jdbc_catalog.out | 8
.../jdbc/test_doris_jdbc_catalog.groovy| 4 ++
3 files changed, 23 insertions(+), 38 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
index 7cfeb3dbaff..a3d3a1885f3 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
@@ -1951,21 +1951,10 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor slotIdsByOrder = Lists.newArrayList();
-if (requiredByProjectSlotIdSet.size() != requiredSlotIdSet.size()
-|| new HashSet<>(projectionExprs).size() !=
projectionExprs.size()
-|| projectionExprs.stream().anyMatch(expr -> !(expr
instanceof SlotRef))) {
-projectionTuple = generateTupleDesc(slots,
-((ScanNode) inputPlanNode).getTupleDesc().getTable(),
context);
-inputPlanNode.setProjectList(projectionExprs);
-inputPlanNode.setOutputTupleDesc(projectionTuple);
-} else {
-for (int i = 0; i < slots.size(); ++i) {
-context.addExprIdSlotRefPair(slots.get(i).getExprId(),
-(SlotRef) projectionExprs.get(i));
-slotIdsByOrder.add(((SlotRef)
projectionExprs.get(i)).getSlotId());
-}
-}
+projectionTuple = generateTupleDesc(slots,
+((ScanNode) inputPlanNode).getTupleDesc().getTable(),
context);
+inputPlanNode.setProjectList(projectionExprs);
+inputPlanNode.setOutputTupleDesc(projectionTuple);
// TODO: this is a temporary scheme to support two phase read when
has project.
// we need to refactor all topn opt into rbo stage.
@@ -1975,20 +1964,16 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor requiredSlotIdSet, Set
requiredByProjectSlotIdSet,
-List slotIdsByOrder, PlanTranslatorContext context) {
+PlanTranslatorContext context) {
// TODO: use smallest slot if do not need any slot in upper node
SlotDescriptor smallest = scanNode.getTupleDesc().getSlots().get(0);
-if (CollectionUtils.isNotEmpty(slotIdsByOrder)) {
-// if we eliminate project above scan, we should ensure the slot
order of scan's output is same with
-// the projection's output. So, we need to reorder the output slot
in scan's tuple.
-Map idToSlotDescMap =
scanNode.getTupleDesc().getSlots().stream()
-.filter(s -> requiredSlotIdSet.contains(s.getId()))
-.collect(Collectors.toMap(SlotDescriptor::getId, s -> s));
-scanNode.getTupleDesc().getSlots().clear();
-for (SlotId slotId : slotIdsByOrder) {
-
scanNode.getTupleDesc().getSlots().add(idToSlotDescMap.get(slotId));
-}
-} else {
-scanNode.getTupleDesc().getSlots().removeIf(s ->
!requiredSlotIdSet.contains(s.getId()));
-}
+scanNode.getTupleDesc().getSlots().removeIf(s ->
!requiredSlotIdSet.contains(s.getId()));
if (scanNode.getTupleDesc().getSlots().isEmpty()) {
scanNode.getTupleDesc().getSlots().add(smallest);
}
diff --git
a/regression-test/data/external_table_p0/jdbc/test_doris_jdbc_catalog.out
b/regr
(doris) branch master updated: [opt](tools) update tools schema (#41304)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 2919634ec95 [opt](tools) update tools schema (#41304) 2919634ec95 is described below commit 2919634ec95638fb6f92219fc39bc78b49cf0614 Author: xzj7019 <13794+xzj7...@users.noreply.github.com> AuthorDate: Wed Sep 25 22:55:56 2024 +0800 [opt](tools) update tools schema (#41304) update tpcds tools schema, mainly for data type refinement for different sf. --- tools/tpcds-tools/ddl/create-tpcds-tables-sf1.sql | 242 ++--- .../tpcds-tools/ddl/create-tpcds-tables-sf100.sql | 242 ++--- .../tpcds-tools/ddl/create-tpcds-tables-sf1000.sql | 242 ++--- .../ddl/create-tpcds-tables-sf1.sql| 234 ++-- 4 files changed, 480 insertions(+), 480 deletions(-) diff --git a/tools/tpcds-tools/ddl/create-tpcds-tables-sf1.sql b/tools/tpcds-tools/ddl/create-tpcds-tables-sf1.sql index 084ae989a3a..e163086fc52 100644 --- a/tools/tpcds-tools/ddl/create-tpcds-tables-sf1.sql +++ b/tools/tpcds-tools/ddl/create-tpcds-tables-sf1.sql @@ -16,7 +16,7 @@ -- under the License. CREATE TABLE IF NOT EXISTS customer_demographics ( -cd_demo_sk bigint not null, +cd_demo_sk integer not null, cd_gender char(1), cd_marital_status char(1), cd_education_status char(20), @@ -32,7 +32,7 @@ PROPERTIES ( "replication_num" = "1" ); CREATE TABLE IF NOT EXISTS reason ( -r_reason_sk bigint not null, +r_reason_sk integer not null, r_reason_id char(16) not null, r_reason_desc char(100) ) @@ -42,7 +42,7 @@ PROPERTIES ( "replication_num" = "1" ); CREATE TABLE IF NOT EXISTS date_dim ( -d_date_sk bigint not null, +d_date_sk integer not null, d_date_id char(16) not null, d_date date, d_month_seq integer, @@ -77,7 +77,7 @@ PROPERTIES ( "replication_num" = "1" ); CREATE TABLE IF NOT EXISTS warehouse ( -w_warehouse_sk bigint not null, +w_warehouse_sk integer not null, w_warehouse_id char(16) not null, w_warehouse_name varchar(20), w_warehouse_sq_ft integer, @@ -98,24 +98,24 @@ PROPERTIES ( "replication_num" = "1" ); CREATE TABLE IF NOT EXISTS catalog_sales ( -cs_item_sk bigint not null, -cs_order_number bigint not null, -cs_sold_date_sk bigint, -cs_sold_time_sk bigint, -cs_ship_date_sk bigint, -cs_bill_customer_sk bigint, -cs_bill_cdemo_sk bigint, -cs_bill_hdemo_sk bigint, -cs_bill_addr_sk bigint, -cs_ship_customer_sk bigint, -cs_ship_cdemo_sk bigint, -cs_ship_hdemo_sk bigint, -cs_ship_addr_sk bigint, -cs_call_center_sk bigint, -cs_catalog_page_sk bigint, -cs_ship_mode_sk bigint, -cs_warehouse_sk bigint, -cs_promo_sk bigint, +cs_item_sk integer not null, +cs_order_number integer not null, +cs_sold_date_sk integer, +cs_sold_time_sk integer, +cs_ship_date_sk integer, +cs_bill_customer_sk integer, +cs_bill_cdemo_sk integer, +cs_bill_hdemo_sk integer, +cs_bill_addr_sk integer, +cs_ship_customer_sk integer, +cs_ship_cdemo_sk integer, +cs_ship_hdemo_sk integer, +cs_ship_addr_sk integer, +cs_call_center_sk integer, +cs_catalog_page_sk integer, +cs_ship_mode_sk integer, +cs_warehouse_sk integer, +cs_promo_sk integer, cs_quantity integer, cs_wholesale_cost decimal(7,2), cs_list_price decimal(7,2), @@ -140,7 +140,7 @@ PROPERTIES ( "colocate_with" = "catalog" ); CREATE TABLE IF NOT EXISTS call_center ( - cc_call_center_sk bigint not null, + cc_call_center_sk integer not null, cc_call_center_id char(16) not null, cc_rec_start_date date, cc_rec_end_date date, @@ -179,9 +179,9 @@ PROPERTIES ( ); CREATE TABLE IF NOT EXISTS inventory ( -inv_date_sk bigint not null, -inv_item_sk bigint not null, -inv_warehouse_sk bigint, +inv_date_sk integer not null, +inv_item_sk integer not null, +inv_warehouse_sk integer, inv_quantity_on_hand integer ) DUPLICATE KEY(inv_date_sk, inv_item_sk, inv_warehouse_sk) @@ -190,23 +190,23 @@ PROPERTIES ( "replication_num" = "1" ); CREATE TABLE IF NOT EXISTS catalog_returns ( - cr_item_sk bigint not null, - cr_order_number bigint not null, - cr_returned_date_sk bigint, - cr_returned_time_sk bigint, - cr_refunded_customer_sk bigint, - cr_refunded_cdemo_sk bigint, - cr_refunded_hdemo_sk bigint, - cr_refunded_addr_sk bigint, - cr_returning_customer_sk bigint, - cr_returning_cdemo_sk bigint, - cr_returning_hdemo_sk bigint, - cr_returning_addr_sk bigint, - cr_call_center_sk bigint, - cr_catalog_page_sk bigint, - cr_ship_mode_sk bigint, - cr_warehouse_
(doris) branch master updated (f8a40ad95cc -> 330ba26445d)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from f8a40ad95cc [Fix](Nereids) fix date and date time arithmatic (#40745) add 330ba26445d [Fix](nereids) fix create view with nullable column (#41234) No new revisions were added by this update. Summary of changes: .../trees/plans/commands/info/BaseViewInfo.java| 5 ++- .../data/correctness/test_view_varchar_length.out | 2 +- .../data/ddl_p0/test_create_view_nereids.out | 10 + .../suites/ddl_p0/test_create_view_nereids.groovy | 47 ++ 4 files changed, 61 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (3d4c22d1194 -> e165fe5ef8a)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 3d4c22d1194 [fix](pipelinex) fix fragment instance progress reports (part 2) (#40694) add e165fe5ef8a [fix](parser) should not use selectHint in any place (#41260) No new revisions were added by this update. Summary of changes: fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (58399d8c2ce -> b52e59aa0d3)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 58399d8c2ce [fix](mysql)Support COM_CHANGE_USER and other mysql command. (#40932) add b52e59aa0d3 [fix](test) mv_p0/unique/unique_rewrite.groovy should set delete without partition (#40705) No new revisions were added by this update. Summary of changes: regression-test/suites/mv_p0/unique/unique_rewrite.groovy | 1 + 1 file changed, 1 insertion(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [enhance](mtmv)Optimize the logic of mtmv lock (#41010)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 185eda32219 [enhance](mtmv)Optimize the logic of mtmv lock (#41010)
185eda32219 is described below
commit 185eda322190abdff49ef11058e13326c29520c7
Author: zhangdong <493738...@qq.com>
AuthorDate: Wed Sep 25 10:38:31 2024 +0800
[enhance](mtmv)Optimize the logic of mtmv lock (#41010)
- When deleting a job, do not query and delete it first. Instead, call
the method to delete the job directly. If the job does not exist(When
the materialized view only creates the table and the job is not yet
created, the materialized view is concurrently deleted), throw an
exception
- When changing mtmv, narrow down the scope of the lock and place it in
each sub method
---
.../src/main/java/org/apache/doris/alter/Alter.java | 8
.../src/main/java/org/apache/doris/catalog/MTMV.java | 2 ++
.../java/org/apache/doris/mtmv/MTMVJobManager.java | 20 ++--
3 files changed, 12 insertions(+), 18 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
b/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
index 72dc8141e27..e6a2fe0229f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
@@ -1003,8 +1003,6 @@ public class Alter {
try {
Database db =
Env.getCurrentInternalCatalog().getDbOrDdlException(tbl.getDb());
mtmv = (MTMV) db.getTableOrMetaException(tbl.getTbl(),
TableType.MATERIALIZED_VIEW);
-
-mtmv.writeMvLock();
switch (alterMTMV.getOpType()) {
case ALTER_REFRESH_INFO:
mtmv.alterRefreshInfo(alterMTMV.getRefreshInfo());
@@ -1017,8 +1015,6 @@ public class Alter {
break;
case ADD_TASK:
mtmv.addTaskResult(alterMTMV.getTask(),
alterMTMV.getRelation(), alterMTMV.getPartitionSnapshots());
-Env.getCurrentEnv().getMtmvService()
-.refreshComplete(mtmv, alterMTMV.getRelation(),
alterMTMV.getTask());
break;
default:
throw new RuntimeException("Unknown type value: " +
alterMTMV.getOpType());
@@ -1031,10 +1027,6 @@ public class Alter {
} catch (UserException e) {
// if MTMV has been dropped, ignore this exception
LOG.warn(e);
-} finally {
-if (mtmv != null) {
-mtmv.writeMvUnlock();
-}
}
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
index bbfed9cf82e..7716dbfe686 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
@@ -205,6 +205,8 @@ public class MTMV extends OlapTable {
}
this.jobInfo.addHistoryTask(task);
this.refreshSnapshot.updateSnapshots(partitionSnapshots,
getPartitionNames());
+Env.getCurrentEnv().getMtmvService()
+.refreshComplete(this, relation, task);
} finally {
writeMvUnlock();
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVJobManager.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVJobManager.java
index 1ace738f1d0..8ffcea423d7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVJobManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVJobManager.java
@@ -48,6 +48,8 @@ import org.apache.doris.qe.ConnectContext;
import com.google.common.collect.Lists;
import org.apache.commons.collections.CollectionUtils;
import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
import java.util.List;
@@ -55,6 +57,8 @@ import java.util.List;
* when do some operation, do something about job
*/
public class MTMVJobManager implements MTMVHookService {
+private static final Logger LOG =
LogManager.getLogger(MTMVJobManager.class);
+
public static final String MTMV_JOB_PREFIX = "inner_mtmv_";
/**
@@ -124,16 +128,12 @@ public class MTMVJobManager implements MTMVHookService {
*/
@Override
public void dropMTMV(MTMV mtmv) throws DdlException {
-List jobs = Env.getCurrentEnv().getJobManager()
-.queryJobs(JobType.MV, mtmv.getJobInfo().getJobName());
-if (!CollectionUtils.isEmpty(jobs)) {
-try {
-Env.getCurrentEnv().getJobManager()
-.unregisterJob(jobs.get(0).getJobId());
-
(doris) branch master updated (4b9894132bc -> 7354b530fed)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 4b9894132bc [fix](analysis) Fix ColumnDef to sql result (#41205) add 7354b530fed [fix](block_rule) SQL block rule not working after FE restart (#41228) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/doris/common/Config.java | 4 .../org/apache/doris/blockrule/SqlBlockRule.java | 16 -- .../apache/doris/blockrule/SqlBlockRuleMgr.java| 5 + .../doris/blockrule/SqlBlockRuleMgrTest.java | 25 ++ 4 files changed, 44 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch fix_column_def_sql_result deleted (was c16a139acdf)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch fix_column_def_sql_result in repository https://gitbox.apache.org/repos/asf/doris.git was c16a139acdf fixup The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (37f79089873 -> 4b9894132bc)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 37f79089873 [chore](Nereids) remove built-in deprecated sql dialect convertor (#41204) add 4b9894132bc [fix](analysis) Fix ColumnDef to sql result (#41205) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/doris/analysis/ColumnDef.java | 13 - .../test/java/org/apache/doris/analysis/ColumnDefTest.java | 8 +++- 2 files changed, 19 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [chore](Nereids) remove built-in deprecated sql dialect convertor (#41204)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 37f79089873 [chore](Nereids) remove built-in deprecated sql dialect
convertor (#41204)
37f79089873 is described below
commit 37f790898730a054a9d63efe1bd346b3d4777a86
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Wed Sep 25 10:27:20 2024 +0800
[chore](Nereids) remove built-in deprecated sql dialect convertor (#41204)
---
.../nereids/analyzer/PlaceholderExpression.java| 101 ---
.../nereids/parser/AbstractFnCallTransformer.java | 45 ---
.../nereids/parser/AbstractFnCallTransformers.java | 108
.../nereids/parser/CommonFnCallTransformer.java| 138 -
.../nereids/parser/ComplexFnCallTransformer.java | 26
.../doris/nereids/parser/LogicalPlanBuilder.java | 1 +
.../nereids/trees/expressions/Expression.java | 2 -
.../functions/ExplicitlyCastableSignature.java | 2 +-
.../expressions/functions/IdenticalSignature.java | 2 +-
.../functions/ImplicitlyCastableSignature.java | 2 +-
.../functions/NullOrIdenticalSignature.java| 2 +-
.../expressions/visitor/ExpressionVisitor.java | 9 --
.../trees/plans/commands/info/CreateTableInfo.java | 1 -
.../plans/commands/info}/PartitionTableInfo.java | 9 +-
.../org/apache/doris/nereids/types/ArrayType.java | 2 +-
.../org/apache/doris/nereids/types/MapType.java| 2 +-
.../org/apache/doris/nereids/types/StructType.java | 2 +-
.../coercion}/ComplexDataType.java | 2 +-
.../doris/nereids/util/TypeCoercionUtils.java | 2 +-
19 files changed, 11 insertions(+), 447 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/PlaceholderExpression.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/PlaceholderExpression.java
deleted file mode 100644
index 9b2dcde49bd..000
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/PlaceholderExpression.java
+++ /dev/null
@@ -1,101 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.nereids.analyzer;
-
-import
org.apache.doris.nereids.parser.CommonFnCallTransformer.PlaceholderCollector;
-import org.apache.doris.nereids.trees.expressions.Expression;
-import org.apache.doris.nereids.trees.expressions.functions.AlwaysNotNullable;
-import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
-
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
-
-import java.util.List;
-import java.util.Objects;
-import java.util.Set;
-
-/**
- * Expression placeHolder, the expression in PlaceHolderExpression will be
collected by
- *
- * @see PlaceholderCollector
- */
-public class PlaceholderExpression extends Expression implements
AlwaysNotNullable {
-
-private final ImmutableSet> delegateClazzSet;
-/**
- * start from 1, set the index of this placeholderExpression in
sourceFnTransformedArguments
- * this placeholderExpression will be replaced later
- */
-private final int position;
-
-public PlaceholderExpression(List children, Class delegateClazz, int position) {
-super(children);
-this.delegateClazzSet = ImmutableSet.of(
-Objects.requireNonNull(delegateClazz, "delegateClazz should
not be null"));
-this.position = position;
-}
-
-public PlaceholderExpression(List children,
- Set>
delegateClazzSet, int position) {
-super(children);
-this.delegateClazzSet = ImmutableSet.copyOf(delegateClazzSet);
-this.position = position;
-}
-
-public static PlaceholderExpression of(Class
delegateClazz, int position) {
-return new PlaceholderExpression(ImmutableList.of(), delegateClazz,
position);
-}
-
-public static PlaceholderExpression of(Set>
delegateClazzSet, int position) {
-return new PlaceholderEx
(doris) branch master updated (370ddd8fb2e -> 47ce9f905e8)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 370ddd8fb2e [feat](metatable) support table$partitions for hive table
(#40774)
add 47ce9f905e8 [fix](Nereids) simplify decimal comparison wrong when cast
to smaller scale (#41151)
No new revisions were added by this update.
Summary of changes:
.../rules/expression/rules/SimplifyCastRule.java | 11 +-
.../rules/SimplifyComparisonPredicate.java | 24 +-
.../rules/SimplifyDecimalV3Comparison.java | 25 +-
.../expressions/literal/DecimalV3Literal.java |8 +-
.../org/apache/doris/nereids/types/DataType.java |2 +-
.../expression/rules/SimplifyCastRuleTest.java | 46 +-
.../rules/SimplifyComparisonPredicateTest.java | 130 ++
.../rules/SimplifyDecimalV3ComparisonTest.java | 50 +-
.../data/export_p0/test_export_data_types.out | 48 +-
.../http_stream/test_http_stream_properties.out| 1452 +++---
.../stream_load/test_stream_load_properties.out| 2100 +---
.../test_array_aggregation_functions.out | 20 +-
.../math_functions/test_least_greatest.out | 37 +-
.../test_simplify_decimal_comparison.groovy} | 12 +-
14 files changed, 1931 insertions(+), 2034 deletions(-)
copy regression-test/suites/{correctness/test_removed_session_var.groovy =>
nereids_rules_p0/expression/test_simplify_decimal_comparison.groovy} (75%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (9ef26181478 -> b0449d2ff93)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 9ef26181478 [Bug](materialized-view) only throw 'MV same with base table is useless' when addMVClause's Ke… (#41144) add b0449d2ff93 [revert](storage) storage medium of partition should not inherit from table (#41192) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/doris/datasource/InternalCatalog.java | 7 --- regression-test/suites/mtmv_p0/test_storage_medium_mtmv.groovy | 2 ++ 2 files changed, 2 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [opt](nereids) refine expression estimation (#40698)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 7a7bc6a7f73 [opt](nereids) refine expression estimation (#40698)
7a7bc6a7f73 is described below
commit 7a7bc6a7f73d81847f1416dca3485ed18589c274
Author: xzj7019 <13794+xzj7...@users.noreply.github.com>
AuthorDate: Tue Sep 24 14:15:22 2024 +0800
[opt](nereids) refine expression estimation (#40698)
Stats deriving refinement step 2: refine expression estimation(part I)
a. refine casewhen/if/literal's avgDataSize/numNull/ndv info.
b. search column statistics cache at first during expression visiting.
c. fix StringType's width() returning -1.
---
.../doris/nereids/stats/ExpressionEstimation.java | 27 ++
.../doris/nereids/stats/FilterEstimation.java | 1 +
.../org/apache/doris/nereids/types/StringType.java | 5 --
.../nereids/types/coercion/CharacterType.java | 2 +-
.../doris/statistics/ColumnStatisticBuilder.java | 3 +-
.../org/apache/doris/statistics/Statistics.java| 2 +-
.../nereids/stats/ExpressionEstimationTest.java| 63 ++
7 files changed, 85 insertions(+), 18 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/ExpressionEstimation.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/ExpressionEstimation.java
index 126e9041721..b26b0315047 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/ExpressionEstimation.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/ExpressionEstimation.java
@@ -128,6 +128,10 @@ public class ExpressionEstimation extends
ExpressionVisitor childrenExpr = expr.children();
if (CollectionUtils.isEmpty(childrenExpr)) {
return ColumnStatistic.UNKNOWN;
@@ -135,26 +139,28 @@ public class ExpressionEstimation extends
ExpressionVisitor statsForLeft.maxValue || val <
statsForLeft.minValue) {
+// TODO: will fix this in the next pr by adding
RangeScalable protection
selectivity = 0.0;
} else if (ndv >= 1.0) {
selectivity = StatsMathUtil.minNonNaN(1.0, 1.0 / ndv);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/StringType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/StringType.java
index 935716e42bf..8e92f83274e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/StringType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/StringType.java
@@ -31,11 +31,6 @@ public class StringType extends CharacterType {
super(-1);
}
-@Override
-public int width() {
-return len;
-}
-
@Override
public Type toCatalogDataType() {
return Type.STRING;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/CharacterType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/CharacterType.java
index 446ccc7fd00..781b1257028 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/CharacterType.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/CharacterType.java
@@ -26,8 +26,8 @@ import org.apache.doris.nereids.types.StringType;
*/
public abstract class CharacterType extends PrimitiveType {
-public static final int DEFAULT_SLOT_SIZE = 20;
private static final int WIDTH = 16;
+public static final int DEFAULT_WIDTH = WIDTH;
protected final int len;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatisticBuilder.java
b/fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatisticBuilder.java
index 47002355de9..4e190ce388e 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatisticBuilder.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatisticBuilder.java
@@ -199,8 +199,7 @@ public class ColumnStatisticBuilder {
// When defining SQL schemas, users often tend to set the length of
string \
// fields much longer than actually needed for storage.
if (slot.getDataType() instanceof CharacterType) {
-avgSizeByte = Math.min(avgSizeByte,
-CharacterType.DEFAULT_SLOT_SIZE);
+avgSizeByte = Math.min(avgSizeByte, CharacterType.DEFAULT_WIDTH);
}
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/statistics/Statistics.java
b/fe/fe-core/src/main/java/org/apache/doris/statistics/Statistics.java
index e18dc097920..7e539ef68f2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/Statistics.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/Statistics.java
@@ -151,7 +151,7 @@ public class Statistics {
for (Slot slot : slots) {
(doris) branch master updated (19424d59b3a -> c38938d2067)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 19424d59b3a [case](mtmv) Failed to fix the case of concurrent insert overwrite execution (#41125) add c38938d2067 [fix](nereids)modify split_part function to fold to null literal with correct data type (#41109) No new revisions were added by this update. Summary of changes: .../trees/expressions/functions/executable/StringArithmetic.java | 2 +- .../nereids_p0/sql_functions/string_functions/test_split_part.groovy | 5 + 2 files changed, 6 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [case](mtmv) Failed to fix the case of concurrent insert overwrite execution (#41125)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 19424d59b3a [case](mtmv) Failed to fix the case of concurrent insert
overwrite execution (#41125)
19424d59b3a is described below
commit 19424d59b3af02e02a9328545adb74ead56f30e4
Author: zhangdong <493738...@qq.com>
AuthorDate: Mon Sep 23 19:09:26 2024 +0800
[case](mtmv) Failed to fix the case of concurrent insert overwrite
execution (#41125)
after PR #40558
insert overwrite is not allowed in the same table
---
.../test_iot_auto_detect_concurrent.groovy | 14 ++
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git
a/regression-test/suites/insert_overwrite_p1/test_iot_auto_detect_concurrent.groovy
b/regression-test/suites/insert_overwrite_p1/test_iot_auto_detect_concurrent.groovy
index e796edfe5bb..0ce026fb99b 100644
---
a/regression-test/suites/insert_overwrite_p1/test_iot_auto_detect_concurrent.groovy
+++
b/regression-test/suites/insert_overwrite_p1/test_iot_auto_detect_concurrent.groovy
@@ -26,6 +26,7 @@ suite("test_iot_auto_detect_concurrent") {
sql new
File("""${context.file.parent}/ddl/test_iot_auto_detect_concurrent.sql""").text
def success_status = true
+def err_msg = ""
def load_data = { range, offset, expect_success ->
try {
sql " use test_iot_auto_detect_concurrent; "
@@ -37,6 +38,7 @@ suite("test_iot_auto_detect_concurrent") {
success_status = false
log.info("fails one")
}
+err_msg = e.getMessage()
log.info("successfully catch the failed insert")
return
}
@@ -98,10 +100,14 @@ suite("test_iot_auto_detect_concurrent") {
thread6.join()
thread7.join()
// suppose result: Success to overwrite with a multiple of ten values
-assertTrue(success_status)
-qt_sql3 " select count(k0) from test_concurrent_write; "
-qt_sql4 " select count(distinct k0) from test_concurrent_write; "
-
+if (!success_status) {
+// Not allowed running Insert Overwrite on same table
+assertTrue(err_msg.contains('same table'))
+} else {
+// The execution was fast, resulting in no concurrent execution
+qt_sql3 " select count(k0) from test_concurrent_write; "
+qt_sql4 " select count(distinct k0) from test_concurrent_write; "
+}
/// with drop partition concurrently
success_status = true
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](nereids)adjust conjunct's nullable info in LogicalExternalRelation (#41014)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 8c34bfea870 [fix](nereids)adjust conjunct's nullable info in
LogicalExternalRelation (#41014)
8c34bfea870 is described below
commit 8c34bfea87010ac0e2debcd4375729c84b701785
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Mon Sep 23 11:04:46 2024 +0800
[fix](nereids)adjust conjunct's nullable info in LogicalExternalRelation
(#41014)
---
.../nereids/rules/rewrite/AdjustNullable.java | 12
.../nereids/rules/rewrite/AdjustNullableTest.java | 74 ++
2 files changed, 86 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AdjustNullable.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AdjustNullable.java
index 808288b8fe3..198b6363d9a 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AdjustNullable.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AdjustNullable.java
@@ -31,6 +31,7 @@ import
org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewri
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
import org.apache.doris.nereids.trees.plans.logical.LogicalCTEConsumer;
+import org.apache.doris.nereids.trees.plans.logical.LogicalExternalRelation;
import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
import org.apache.doris.nereids.trees.plans.logical.LogicalGenerate;
import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
@@ -276,6 +277,17 @@ public class AdjustNullable extends
DefaultPlanRewriter> imple
return cteConsumer.withTwoMaps(consumerToProducerOutputMap,
producerToConsumerOutputMap);
}
+@Override
+public Plan visitLogicalExternalRelation(LogicalExternalRelation relation,
Map replaceMap) {
+if (!relation.getConjuncts().isEmpty()) {
+relation.getOutputSet().forEach(s -> replaceMap.put(s.getExprId(),
s));
+Set conjuncts =
updateExpressions(relation.getConjuncts(), replaceMap);
+return
relation.withConjuncts(conjuncts).recomputeLogicalProperties();
+} else {
+return relation;
+}
+}
+
private T updateExpression(T input, Map replaceMap) {
return (T) input.rewriteDownShortCircuit(e ->
e.accept(SlotReferenceReplacer.INSTANCE, replaceMap));
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/AdjustNullableTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/AdjustNullableTest.java
new file mode 100644
index 000..023f9c4f7ff
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/AdjustNullableTest.java
@@ -0,0 +1,74 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.GreaterThan;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.RelationId;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJdbcScan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
+import org.apache.doris.nereids.types.IntegerType;
+import org.apache.doris.nereids.util.MemoPatternMatchSupported;
+import org.apache.doris.nereids.util.PlanConstructor;
+
+import mockit.Mock;
+import mockit.MockUp;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Tests for {@link AdjustNullableTest}.
+ */
+class AdjustNull
(doris) branch branch-2.1 updated: [opt](nereids)show user friendly error msg when window function contains order by expression (#40937) (#41036)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new dbd72deaa5c [opt](nereids)show user friendly error msg when window
function contains order by expression (#40937) (#41036)
dbd72deaa5c is described below
commit dbd72deaa5c901cba0d0a8f37a48c88f1cdb3d07
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Fri Sep 20 18:50:55 2024 +0800
[opt](nereids)show user friendly error msg when window function contains
order by expression (#40937) (#41036)
pick from master #40937
---
.../rules/rewrite/ExtractAndNormalizeWindowExpression.java| 4
.../normalize_window/normalize_window_nullable_agg_test.groovy| 8
2 files changed, 12 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ExtractAndNormalizeWindowExpression.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ExtractAndNormalizeWindowExpression.java
index e82c3f7b416..2cfe4523003 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ExtractAndNormalizeWindowExpression.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ExtractAndNormalizeWindowExpression.java
@@ -17,6 +17,7 @@
package org.apache.doris.nereids.rules.rewrite;
+import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.trees.expressions.Alias;
@@ -60,6 +61,9 @@ public class ExtractAndNormalizeWindowExpression extends
OneRewriteRuleFactory i
if (output instanceof WindowExpression) {
WindowExpression windowExpression = (WindowExpression)
output;
Expression expression = ((WindowExpression)
output).getFunction();
+if (expression.containsType(OrderExpression.class)) {
+throw new AnalysisException("order by is not
supported in " + expression);
+}
if (expression instanceof NullableAggregateFunction) {
// NullableAggregateFunction in window function
should be always nullable
// Because there may be no data in the window
frame, null values will be generated.
diff --git
a/regression-test/suites/nereids_rules_p0/normalize_window/normalize_window_nullable_agg_test.groovy
b/regression-test/suites/nereids_rules_p0/normalize_window/normalize_window_nullable_agg_test.groovy
index 915a4d025a8..7087797e4e4 100644
---
a/regression-test/suites/nereids_rules_p0/normalize_window/normalize_window_nullable_agg_test.groovy
+++
b/regression-test/suites/nereids_rules_p0/normalize_window/normalize_window_nullable_agg_test.groovy
@@ -68,4 +68,12 @@ suite("normalize_window_nullable_agg") {
//qt_window_funnel """select window_funnel(3600 * 3, 'default', t.xwhen,
t.xwhat = 1, t.xwhat = 2 ) over (order by xwhat rows
//between 2 preceding and 1 preceding) AS level from
windowfunnel_test_normalize_window t;"""
qt_sequence_match "SELECT sequence_match('(?1)(?2)', xwhen, xwhat = 1,
xwhat = 3) over (order by xwhat rows between 2 preceding and 1 preceding) FROM
windowfunnel_test_normalize_window;"
+
+
+sql "set enable_nereids_planner=true;"
+sql "set enable_fallback_to_original_planner=false;"
+test {
+sql "select group_concat(xwho order by xwhat) over(partition by xwhen)
from windowfunnel_test_normalize_window;"
+exception "order by is not supported"
+}
}
\ No newline at end of file
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [test](mtmv) Fix regression test unstable (#40871)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new da6ac0c5f47 [test](mtmv) Fix regression test unstable (#40871)
da6ac0c5f47 is described below
commit da6ac0c5f472b707b1dba30f71d5a6d85b158545
Author: seawinde <149132972+seawi...@users.noreply.github.com>
AuthorDate: Fri Sep 20 18:41:22 2024 +0800
[test](mtmv) Fix regression test unstable (#40871)
1. Optimized `waitForRollUpJob` method in `Suite.groovy` to make sure
roll up is build succesfully before rewirte.
2. Modify expect rewritten result in` agg_sync_mv.groovy`
---
.../org/apache/doris/regression/suite/Suite.groovy | 21 +
.../ddl_p0/test_create_table_like_nereids.groovy| 7 ---
.../suites/nereids_p0/hint/test_use_mv.groovy | 4 ++--
.../mv/aggregate/agg_sync_mv.groovy | 4 +++-
4 files changed, 18 insertions(+), 18 deletions(-)
diff --git
a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy
b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy
index ad14e23f94a..73b2cf9b5bc 100644
---
a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy
+++
b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy
@@ -1397,29 +1397,26 @@ class Suite implements GroovyInterceptable {
}
}
-def getMVJobState = { tableName, limit ->
-def jobStateResult = sql """ SHOW ALTER TABLE ROLLUP WHERE
TableName='${tableName}' ORDER BY CreateTime DESC limit ${limit}"""
-if (jobStateResult.size() != limit) {
+def getMVJobState = { tableName, rollUpName ->
+def jobStateResult = sql """ SHOW ALTER TABLE ROLLUP WHERE
TableName='${tableName}' and IndexName = '${rollUpName}' ORDER BY CreateTime
DESC limit 1"""
+if (jobStateResult == null || jobStateResult.isEmpty()) {
logger.info("show alter table roll is empty" + jobStateResult)
return "NOT_READY"
}
-for (int i = 0; i < jobStateResult.size(); i++) {
-logger.info("getMVJobState is " + jobStateResult[i][8])
-if (!jobStateResult[i][8].equals("FINISHED")) {
-return "NOT_READY"
-}
+logger.info("getMVJobState jobStateResult is " +
jobStateResult.toString())
+if (!jobStateResult[0][8].equals("FINISHED")) {
+return "NOT_READY"
}
return "FINISHED";
}
-def waitForRollUpJob = (tbName, timeoutMillisecond, limit) -> {
+def waitForRollUpJob = (tbName, rollUpName, timeoutMillisecond) -> {
long startTime = System.currentTimeMillis()
long timeoutTimestamp = startTime + timeoutMillisecond
String result
-// time out or has run exceed 10 minute, then break
-while (timeoutTimestamp > System.currentTimeMillis() &&
System.currentTimeMillis() - startTime < 60){
-result = getMVJobState(tbName, limit)
+while (timeoutTimestamp > System.currentTimeMillis()){
+result = getMVJobState(tbName, rollUpName)
if (result == "FINISHED") {
sleep(200)
return
diff --git
a/regression-test/suites/ddl_p0/test_create_table_like_nereids.groovy
b/regression-test/suites/ddl_p0/test_create_table_like_nereids.groovy
index e6ca0b696ff..a371f5ac051 100644
--- a/regression-test/suites/ddl_p0/test_create_table_like_nereids.groovy
+++ b/regression-test/suites/ddl_p0/test_create_table_like_nereids.groovy
@@ -46,7 +46,8 @@ suite("test_create_table_like_nereids") {
// with all rollup
sql "drop table if exists table_like_with_roll_up"
sql "CREATE TABLE table_like_with_roll_up LIKE mal_test_create_table_like
with rollup;"
-waitForRollUpJob("mal_test_create_table_like", 5000, 2)
+waitForRollUpJob("mal_test_create_table_like", "r1", 6)
+waitForRollUpJob("mal_test_create_table_like", "r2", 6)
explain {
sql ("select sum(a) from table_like_with_roll_up group by a")
contains "ru1"
@@ -59,7 +60,7 @@ suite("test_create_table_like_nereids") {
// with partial rollup
sql "drop table if exists table_like_with_partial_roll_up;"
sql "CREATE TABLE table_like_with_partial_roll_up LIKE
mal_test_create_table_like with rollup (ru1);"
-waitForRollUpJob("mal_test_create_table_like", 5000, 2)
+waitForRo
(doris) branch master updated: [chore](test) remove enable_nereids_planner=false in cases (#41035)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 04223de2eb4 [chore](test) remove enable_nereids_planner=false in cases
(#41035)
04223de2eb4 is described below
commit 04223de2eb4b9bcc8b3ec9d52ca158a3e6f870a2
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Fri Sep 20 14:24:14 2024 +0800
[chore](test) remove enable_nereids_planner=false in cases (#41035)
- test_cloud_mow_insert_with_retry
---
.../cloud/test_cloud_mow_insert_with_retry.out | 7 ---
.../cloud/test_cloud_mow_insert_with_retry.groovy | 70 ++
2 files changed, 30 insertions(+), 47 deletions(-)
diff --git
a/regression-test/data/fault_injection_p0/cloud/test_cloud_mow_insert_with_retry.out
b/regression-test/data/fault_injection_p0/cloud/test_cloud_mow_insert_with_retry.out
index 979483692d3..f3a322f2b9f 100644
---
a/regression-test/data/fault_injection_p0/cloud/test_cloud_mow_insert_with_retry.out
+++
b/regression-test/data/fault_injection_p0/cloud/test_cloud_mow_insert_with_retry.out
@@ -6,10 +6,3 @@
-- !sql --
1 1 1
--- !sql --
-1 1 1
-2 2 2
-
--- !sql --
-1 1 1
-
diff --git
a/regression-test/suites/fault_injection_p0/cloud/test_cloud_mow_insert_with_retry.groovy
b/regression-test/suites/fault_injection_p0/cloud/test_cloud_mow_insert_with_retry.groovy
index f7038b80e42..99eb18a4917 100644
---
a/regression-test/suites/fault_injection_p0/cloud/test_cloud_mow_insert_with_retry.groovy
+++
b/regression-test/suites/fault_injection_p0/cloud/test_cloud_mow_insert_with_retry.groovy
@@ -29,11 +29,10 @@ suite("test_cloud_mow_insert_with_retry", "nonConcurrent") {
def dbName = "regression_test_fault_injection_p0_cloud"
def table1 = dbName + ".test_cloud_mow_insert_with_retry"
setFeConfigTemporary(customFeConfig) {
-for (item in ["legacy", "nereids"]) {
-try {
-
GetDebugPoint().enableDebugPointForAllBEs("CloudEngineCalcDeleteBitmapTask.execute.enable_wait")
-sql "DROP TABLE IF EXISTS ${table1} FORCE;"
-sql """ CREATE TABLE IF NOT EXISTS ${table1} (
+try {
+
GetDebugPoint().enableDebugPointForAllBEs("CloudEngineCalcDeleteBitmapTask.execute.enable_wait")
+sql "DROP TABLE IF EXISTS ${table1} FORCE;"
+sql """ CREATE TABLE IF NOT EXISTS ${table1} (
`k1` int NOT NULL,
`c1` int,
`c2` int
@@ -43,44 +42,35 @@ suite("test_cloud_mow_insert_with_retry", "nonConcurrent") {
"enable_unique_key_merge_on_write" = "true",
"disable_auto_compaction" = "true",
"replication_num" = "1"); """
-connect(user = context.config.jdbcUser, password =
context.config.jdbcPassword, url = context.config.jdbcUrl +
"&useLocalSessionState=true") {
-if (item == "nereids") {
-sql """ set enable_nereids_planner=true; """
-sql """ set enable_fallback_to_original_planner=false;
"""
-} else {
-sql """ set enable_nereids_planner = false; """
-}
-def timeout = 2000
-def now = System.currentTimeMillis()
-sql "insert into ${table1} values(1,1,1);"
-def time_diff = System.currentTimeMillis() - now
-logger.info("time_diff:" + time_diff)
-assertTrue(time_diff > timeout, "insert or delete should
take over " + timeout + " ms")
+connect(user = context.config.jdbcUser, password =
context.config.jdbcPassword, url = context.config.jdbcUrl +
"&useLocalSessionState=true") {
+def timeout = 2000
+def now = System.currentTimeMillis()
+sql "insert into ${table1} values(1,1,1);"
+def time_diff = System.currentTimeMillis() - now
+logger.info("time_diff:" + time_diff)
+assertTrue(time_diff > timeout, "insert or delete should take
over " + timeout + " ms")
-now = System.currentTimeMillis()
-sql "insert into ${table1} values(2,2,2);"
-
(doris) branch master updated: [opt](nereids) show user friendly error msg when window function contains order by expression (#40937)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ed2ec7b460a [opt](nereids) show user friendly error msg when window
function contains order by expression (#40937)
ed2ec7b460a is described below
commit ed2ec7b460a79eac66b240a3415e189c76901f22
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Fri Sep 20 11:11:51 2024 +0800
[opt](nereids) show user friendly error msg when window function contains
order by expression (#40937)
---
.../nereids/rules/rewrite/ExtractAndNormalizeWindowExpression.java| 4
.../normalize_window/normalize_window_nullable_agg_test.groovy| 4
2 files changed, 8 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ExtractAndNormalizeWindowExpression.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ExtractAndNormalizeWindowExpression.java
index e82c3f7b416..2cfe4523003 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ExtractAndNormalizeWindowExpression.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ExtractAndNormalizeWindowExpression.java
@@ -17,6 +17,7 @@
package org.apache.doris.nereids.rules.rewrite;
+import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.trees.expressions.Alias;
@@ -60,6 +61,9 @@ public class ExtractAndNormalizeWindowExpression extends
OneRewriteRuleFactory i
if (output instanceof WindowExpression) {
WindowExpression windowExpression = (WindowExpression)
output;
Expression expression = ((WindowExpression)
output).getFunction();
+if (expression.containsType(OrderExpression.class)) {
+throw new AnalysisException("order by is not
supported in " + expression);
+}
if (expression instanceof NullableAggregateFunction) {
// NullableAggregateFunction in window function
should be always nullable
// Because there may be no data in the window
frame, null values will be generated.
diff --git
a/regression-test/suites/nereids_rules_p0/normalize_window/normalize_window_nullable_agg_test.groovy
b/regression-test/suites/nereids_rules_p0/normalize_window/normalize_window_nullable_agg_test.groovy
index 915a4d025a8..9617e8bb19c 100644
---
a/regression-test/suites/nereids_rules_p0/normalize_window/normalize_window_nullable_agg_test.groovy
+++
b/regression-test/suites/nereids_rules_p0/normalize_window/normalize_window_nullable_agg_test.groovy
@@ -68,4 +68,8 @@ suite("normalize_window_nullable_agg") {
//qt_window_funnel """select window_funnel(3600 * 3, 'default', t.xwhen,
t.xwhat = 1, t.xwhat = 2 ) over (order by xwhat rows
//between 2 preceding and 1 preceding) AS level from
windowfunnel_test_normalize_window t;"""
qt_sequence_match "SELECT sequence_match('(?1)(?2)', xwhen, xwhat = 1,
xwhat = 3) over (order by xwhat rows between 2 preceding and 1 preceding) FROM
windowfunnel_test_normalize_window;"
+test {
+sql "select group_concat(xwho order by xwhat) over(partition by xwhen)
from windowfunnel_test_normalize_window;"
+exception "order by is not supported"
+}
}
\ No newline at end of file
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (107856e83a6 -> fc7f8f3c09a)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 107856e83a6 [fix](restore) Ignore base tablet if it is migrated (#41017) add fc7f8f3c09a [fix](Nereids) merge request properties map of group should compare cost (#40819) No new revisions were added by this update. Summary of changes: .../apache/doris/nereids/memo/GroupExpression.java | 22 +++- .../doris/nereids/memo/GroupExpressionTest.java| 62 ++ 2 files changed, 83 insertions(+), 1 deletion(-) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/memo/GroupExpressionTest.java - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [chore](test) remove enable_nereids_planner=false in cases (#40984)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d8277526697 [chore](test) remove enable_nereids_planner=false in cases
(#40984)
d8277526697 is described below
commit d827752669772530cf6470294acec177d4770378
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Fri Sep 20 10:24:20 2024 +0800
[chore](test) remove enable_nereids_planner=false in cases (#40984)
---
.../test_partial_update_seq_map_col.out| 94 --
.../test_partial_update_mow_with_sync_mv.groovy| 19 +--
.../test_partial_update_seq_map_col.groovy | 144 ++---
3 files changed, 71 insertions(+), 186 deletions(-)
diff --git
a/regression-test/data/unique_with_mow_p0/partial_update/test_partial_update_seq_map_col.out
b/regression-test/data/unique_with_mow_p0/partial_update/test_partial_update_seq_map_col.out
index 5bdfef02182..4fab559a433 100644
---
a/regression-test/data/unique_with_mow_p0/partial_update/test_partial_update_seq_map_col.out
+++
b/regression-test/data/unique_with_mow_p0/partial_update/test_partial_update_seq_map_col.out
@@ -93,97 +93,3 @@
3 3 \N \N
4 4 \N \N
--- !sql1 --
-1 1 \N
-2 2 \N
-3 3 \N
-4 4 \N
-
--- !sql1 --
-1 1 20
-2 2 20
-3 3 \N
-4 4 \N
-
--- !sql1 --
-1 1 20
-2 2 20
-3 3 \N
-4 4 \N
-
--- !sql1 --
-3 3 2099-09-10T12:00:00.977174 \N
2099-09-10T12:00:00.977174
-4 4 2099-09-10T12:00:00.977174 \N
2099-09-10T12:00:00.977174
-
--- !sql2 --
-1 1
-2 2
-3 3
-4 4
-
--- !sql3 --
-1 1 999 999
-2 2 999 999
-3 3 999 999
-4 4 999 999
-
--- !sql3 --
-1 99
-2 99
-3 3 999 999
-4 4 999 999
-5 99
-
--- !sql4 --
-1 1 \N \N
-2 2 \N \N
-3 3 \N \N
-4 4 \N \N
-
--- !sql1 --
-1 1 \N
-2 2 \N
-3 3 \N
-4 4 \N
-
--- !sql1 --
-1 1 20
-2 2 20
-3 3 \N
-4 4 \N
-
--- !sql1 --
-1 1 20
-2 2 20
-3 3 \N
-4 4 \N
-
--- !sql1 --
-3 3 2099-09-10T12:00:00.977174 \N
2099-09-10T12:00:00.977174
-4 4 2099-09-10T12:00:00.977174 \N
2099-09-10T12:00:00.977174
-
--- !sql2 --
-1 1
-2 2
-3 3
-4 4
-
--- !sql3 --
-1 1 999 999
-2 2 999 999
-3 3 999 999
-4 4 999 999
-
--- !sql3 --
-1 99
-2 99
-3 3 999 999
-4 4 999 999
-5 99
-
--- !sql4 --
-1 1 \N \N
-2 2 \N \N
-3 3 \N \N
-4 4 \N \N
-
diff --git
a/regression-test/suites/unique_with_mow_p0/partial_update/test_partial_update_mow_with_sync_mv.groovy
b/regression-test/suites/unique_with_mow_p0/partial_update/test_partial_update_mow_with_sync_mv.groovy
index 34405843269..972542dcd34 100644
---
a/regression-test/suites/unique_with_mow_p0/partial_update/test_partial_update_mow_with_sync_mv.groovy
+++
b/regression-test/suites/unique_with_mow_p0/partial_update/test_partial_update_mow_with_sync_mv.groovy
@@ -72,22 +72,13 @@ suite("test_partial_update_mow_with_sync_mv") {
from test_partial_update_mow_with_sync_mv;
""")
-for (def use_nereids : [true, false]) {
-if (use_nereids) {
-sql "set enable_nereids_planner=true"
-sql "set enable_fallback_to_original_planner=false"
-} else {
-sql "set enable_nereids_planner=false"
-}
-
-sql "set enable_unique_key_partial_update=true;"
-sql "sync;"
+sql "set enable_unique_key_partial_update=true;"
+sql "sync;"
-test {
-sql """insert into
test_partial_update_mow_with_sync_mv(l_orderkey, l_linenumber, l_partkey,
l_suppkey, l_shipdate, l_returnflag) values
+test {
+sql """insert into test_partial_update_mow_with_sync_mv(l_orderkey,
l_linenumber, l_partkey, l_suppkey, l_shipdate, l_returnflag) values
(2, 3, 2, 1, '2023-10-18', 'k'); """
-exception "Can't do partial update on merge-on-write Unique table
with sync materialized view."
-}
+exception "Can't do partial update on merge-on
(doris) branch master updated: [fix](mtmv) Generate mtmv cache should use ADMIN user, and rewritten plan should not check privilege (#40374)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f3d92e48ed7 [fix](mtmv) Generate mtmv cache should use ADMIN user, and
rewritten plan should not check privilege (#40374)
f3d92e48ed7 is described below
commit f3d92e48ed781c138685b4cbb24d16f9c55c7820
Author: seawinde <149132972+seawi...@users.noreply.github.com>
AuthorDate: Thu Sep 19 15:52:09 2024 +0800
[fix](mtmv) Generate mtmv cache should use ADMIN user, and rewritten plan
should not check privilege (#40374)
1. In query rewrite by materialized view, If user has the select
privilege for table `lineitem` and `orders` but not mv1.
and mv def is as following:
CREATE MATERIALIZED VIEW mv1
BUILD IMMEDIATE REFRESH AUTO ON MANUAL
DISTRIBUTED BY RANDOM BUCKETS 1
PROPERTIES ('replication_num' = '1')
AS
select l_shipdate, o_orderdate, l_partkey, l_suppkey,
sum(o_totalprice) as sum_total,
max(o_totalprice) as max_total,
min(o_totalprice) as min_total,
count(*) as count_all,
bitmap_union(to_bitmap(case when o_shippriority > 1 and
o_orderkey IN (1, 3) then o_custkey else null end)) as bitmap_union_basic
from lineitem
left join orders on lineitem.l_orderkey = orders.o_orderkey and
l_shipdate = o_orderdate
group by
l_shipdate,
o_orderdate,
l_partkey,
l_suppkey;
if query is as follwoing, we should rewrite by mv1 sucessfully. Because
the privilege check for table is passed, though user has no select
privilege for mv1, but can use the rewritten plan with mv1.
select t1.l_partkey, t1.l_suppkey, o_orderdate,
sum(o_totalprice),
max(o_totalprice),
min(o_totalprice),
count(*),
count(distinct case when o_shippriority > 1 and o_orderkey IN
(1, 3) then o_custkey else null end)
from (select * from lineitem where l_shipdate = '2023-12-11') t1
left join orders on t1.l_orderkey = orders.o_orderkey and
t1.l_shipdate = o_orderdate
group by
o_orderdate,
l_partkey,
l_suppkey;
2. We use the `admin` user when refresh materialized view. So genenrate
mtmv cache, we also need to use the `admin` user.
---
.../main/java/org/apache/doris/catalog/MTMV.java | 8 +-
.../mysql/privilege/AccessControllerManager.java | 9 +
.../nereids/rules/analysis/UserAuthentication.java | 6 +-
.../exploration/mv/MaterializedViewUtils.java | 9 +-
.../nereids/mv/MtmvCacheNewConnectContextTest.java | 84 ++
.../mv/with_auth/with_select_table_auth.groovy | 184 +
6 files changed, 293 insertions(+), 7 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
index 0b7bde221c2..bbfed9cf82e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
@@ -300,7 +300,13 @@ public class MTMV extends OlapTable {
}
// Concurrent situations may result in duplicate cache generation,
// but we tolerate this in order to prevent nested use of readLock and
write MvLock for the table
-MTMVCache mtmvCache = MTMVCache.from(this, connectionContext, true);
+MTMVCache mtmvCache;
+try {
+// Should new context with ADMIN user
+mtmvCache = MTMVCache.from(this,
MTMVPlanUtil.createMTMVContext(this), true);
+} finally {
+connectionContext.setThreadLocalInfo();
+}
writeMvLock();
try {
this.cache = mtmvCache;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/AccessControllerManager.java
b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/AccessControllerManager.java
index ba23c91e27d..b4406caf99c 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/AccessControllerManager.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/AccessControllerManager.java
@@ -173,6 +173,15 @@ public class AccessControllerManager {
}
// Column
+// If param has ctx, we can skip auth by isSkipAuth field in ctx
+public void checkColumnsPriv(ConnectContext ctx, String ctl, String
qualifiedDb, String tbl, Set cols,
+PrivPredicate wanted) throws UserException {
+if (ctx.isSkipAuth()) {
+re
(doris) branch master updated (18048e24a26 -> 412ff703174)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 18048e24a26 [enhance](mtmv)tasks tvf not getLock (#40816) add 412ff703174 [enhance](mtmv)Improve the speed of obtaining table snapshots (#40717) No new revisions were added by this update. Summary of changes: .../apache/doris/datasource/hive/HMSExternalTable.java | 18 +- 1 file changed, 9 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (1da5de683c0 -> 18048e24a26)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 1da5de683c0 [enhance](mtmv)when drop temp partition, not trigger materialized view refresh (#40875) add 18048e24a26 [enhance](mtmv)tasks tvf not getLock (#40816) No new revisions were added by this update. Summary of changes: fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java | 5 + .../src/main/java/org/apache/doris/job/extensions/mtmv/MTMVJob.java | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [enhance](mtmv)when drop temp partition, not trigger materialized view refresh (#40875)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1da5de683c0 [enhance](mtmv)when drop temp partition, not trigger
materialized view refresh (#40875)
1da5de683c0 is described below
commit 1da5de683c0dd055972a42e467d0911031aa22fc
Author: zhangdong <493738...@qq.com>
AuthorDate: Thu Sep 19 15:19:47 2024 +0800
[enhance](mtmv)when drop temp partition, not trigger materialized view
refresh (#40875)
when drop temp partition, not trigger materialized view refresh
---
.../main/java/org/apache/doris/datasource/InternalCatalog.java | 2 +-
.../src/main/java/org/apache/doris/event/DropPartitionEvent.java | 9 -
fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVService.java | 4
3 files changed, 13 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
index 6569477ab9e..c3d7c74a615 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
@@ -2014,7 +2014,7 @@ public class InternalCatalog implements
CatalogIf {
// it does not affect the logic of deleting the partition
try {
Env.getCurrentEnv().getEventProcessor().processEvent(
-new DropPartitionEvent(db.getCatalog().getId(),
db.getId(), olapTable.getId()));
+new DropPartitionEvent(db.getCatalog().getId(),
db.getId(), olapTable.getId(), isTempPartition));
} catch (Throwable t) {
// According to normal logic, no exceptions will be thrown,
// but in order to avoid bugs affecting the original logic, all
exceptions are caught
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/event/DropPartitionEvent.java
b/fe/fe-core/src/main/java/org/apache/doris/event/DropPartitionEvent.java
index 598768aa8de..cea03e7ef79 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/event/DropPartitionEvent.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/event/DropPartitionEvent.java
@@ -20,7 +20,14 @@ package org.apache.doris.event;
import org.apache.doris.common.AnalysisException;
public class DropPartitionEvent extends TableEvent {
-public DropPartitionEvent(long ctlId, long dbId, long tableId) throws
AnalysisException {
+private boolean isTempPartition;
+
+public DropPartitionEvent(long ctlId, long dbId, long tableId, boolean
isTempPartition) throws AnalysisException {
super(EventType.DROP_PARTITION, ctlId, dbId, tableId);
+this.isTempPartition = isTempPartition;
+}
+
+public boolean isTempPartition() {
+return isTempPartition;
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVService.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVService.java
index 4b740b75ef8..278811d3a99 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVService.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVService.java
@@ -24,6 +24,7 @@ import org.apache.doris.catalog.TableIf;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.MetaNotFoundException;
+import org.apache.doris.event.DropPartitionEvent;
import org.apache.doris.event.Event;
import org.apache.doris.event.EventException;
import org.apache.doris.event.EventListener;
@@ -177,6 +178,9 @@ public class MTMVService implements EventListener {
if (!(event instanceof TableEvent)) {
return;
}
+if (event instanceof DropPartitionEvent && ((DropPartitionEvent)
event).isTempPartition()) {
+return;
+}
TableEvent tableEvent = (TableEvent) event;
LOG.info("processEvent, Event: {}", event);
TableIf table;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated (f483a7605c8 -> 8302261dd25)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git from f483a7605c8 [fix](nestedtypes) fix nested type with is_exclusive (#40434) add 8302261dd25 [Fix](nereids) set all nullable aggregate function to alwaysnullable in window expression (#40693) (#40809) No new revisions were added by this update. Summary of changes: .../ExtractAndNormalizeWindowExpression.java | 16 +- .../normalize_window_nullable_agg_test.out | 293 + .../normalize_window_nullable_agg_test.groovy | 71 + 3 files changed, 369 insertions(+), 11 deletions(-) create mode 100644 regression-test/data/nereids_rules_p0/normalize_window/normalize_window_nullable_agg_test.out create mode 100644 regression-test/suites/nereids_rules_p0/normalize_window/normalize_window_nullable_agg_test.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (b2a50a3eb78 -> 91d085c8fe9)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from b2a50a3eb78 [feature](functions) impl scalar functions translate and
url_encode (#40567)
add 91d085c8fe9 [enhance](mtmv) Refuse to execute insert overwrite on the
same table (#40558)
No new revisions were added by this update.
Summary of changes:
.../insertoverwrite/InsertOverwriteManager.java| 58
.../apache/doris/job/extensions/mtmv/MTMVTask.java | 4 ++
.../insert/InsertOverwriteTableCommand.java| 71 ++--
.../java/org/apache/doris/qe/StmtExecutor.java | 21 ++
.../InsertOverwriteManagerTest.java| 77 ++
.../org/apache/doris/regression/suite/Suite.groovy | 24 ++-
.../test_alter_job_mtmv.groovy}| 46 ++---
7 files changed, 269 insertions(+), 32 deletions(-)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/insertoverwrite/InsertOverwriteManagerTest.java
copy regression-test/suites/{mtmv_up_down_olap_p0/load.groovy =>
mtmv_p0/test_alter_job_mtmv.groovy} (61%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (552cb345be8 -> c5542f7c2f1)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 552cb345be8 [Refactor](scan) remove unless code in BE and FE (#40927) add c5542f7c2f1 [chore](test) remove enable_nereids_planner=false (#40934) No new revisions were added by this update. Summary of changes: .../duplicate/storage/test_duplicate_bitmap.out| 24 - .../duplicate/storage/test_duplicate_hll.out | 56 --- .../storage/test_duplicate_quantile_state.out | 42 -- .../data_model_p0/unique/test_unique_bitmap.out| 20 - .../data/data_model_p0/unique/test_unique_hll.out | 92 .../unique/test_unique_quantile_state.out | 60 --- .../unique/test_unique_table_auto_inc.out | 14 - .../data/datatype_p0/decimalv3/test_decimalv3.out | 12 - .../data/ddl_p0/test_create_or_replace_view.out| 3 - .../data/delete_p0/test_delete_on_mor.out | 25 -- .../data/insert_p0/insert_group_commit_into.out| 108 - .../insert_p0/insert_group_commit_into_unique.out | 88 .../insert_group_commit_into_unique_sync_mode.out | 88 .../data/insert_p0/insert_with_null.out| 20 - .../test_new_partial_update_delete.out | 52 --- .../test_partial_update_after_delete.out | 6 - .../test_partial_update_auto_inc.out | 36 -- .../test_partial_update_case_insensitivity.out | 38 -- .../partial_update/test_partial_update_delete.out | 196 .../test_partial_update_native_insert_stmt.out | 12 - .../data/update/test_unique_table_update.out | 10 - regression-test/data/update/test_update_mow.out| 30 -- .../account_p0/test_nereids_row_policy.groovy | 13 +- .../duplicate/storage/test_duplicate_bitmap.groovy | 52 +-- .../duplicate/storage/test_duplicate_hll.groovy| 76 ++-- .../storage/test_duplicate_quantile_state.groovy | 68 ++- .../data_model_p0/unique/test_unique_bitmap.groovy | 38 +- .../data_model_p0/unique/test_unique_hll.groovy| 76 ++-- .../unique/test_unique_quantile_state.groovy | 68 ++- .../unique/test_unique_table_auto_inc.groovy | 15 - .../datatype_p0/decimalv3/test_decimalv3.groovy| 12 - .../ddl_p0/test_create_or_replace_view.groovy | 18 - .../suites/delete_p0/test_delete.groovy| 1 - .../suites/delete_p0/test_delete_on_mor.groovy | 51 +-- .../group_commit/test_group_commit_error.groovy| 3 - .../insert_p0/insert_group_commit_into.groovy | 500 ++--- .../insert_group_commit_into_unique.groovy | 363 +++ ...nsert_group_commit_into_unique_sync_mode.groovy | 364 +++ .../insert_group_commit_with_large_data.groovy | 64 ++- .../suites/insert_p0/insert_with_null.groovy | 9 +- .../test_group_commit_data_bytes_property.groovy | 52 +-- .../test_group_commit_interval_ms_property.groovy | 28 +- .../insert/test_insert_default_value.groovy| 5 - .../suites/prepared_stmt_p0/prepared_stmt.groovy | 4 - .../test_new_partial_update_delete.groovy | 114 - .../test_partial_update_after_delete.groovy| 28 -- .../test_partial_update_auto_inc.groovy| 148 +++--- .../test_partial_update_case_insensitivity.groovy | 50 +-- .../test_partial_update_delete.groovy | 268 ++- .../test_partial_update_native_insert_stmt.groovy | 53 +-- .../suites/update/test_unique_table_update.groovy | 23 - .../suites/update/test_update_mow.groovy | 62 --- .../suites/variant_p0/column_name.groovy | 2 - 53 files changed, 1063 insertions(+), 2597 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (979cb012f88 -> 55727c312ea)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 979cb012f88 [Featrue](default value) Support `bitmap_empty` default
value (#40364)
add 55727c312ea [Enhancement](function) support last_query_id function
(#40739)
No new revisions were added by this update.
Summary of changes:
.../org/apache/doris/catalog/BuiltinScalarFunctions.java | 4 +++-
.../rules/expression/rules/FoldConstantRuleOnFE.java | 16 +++-
.../scalar/{SessionUser.java => LastQueryId.java}| 10 +-
.../trees/expressions/visitor/ScalarFunctionVisitor.java | 5 +
.../main/java/org/apache/doris/qe/ConnectContext.java| 8
.../suites/query_p0/system/test_query_sys.groovy | 2 ++
6 files changed, 38 insertions(+), 7 deletions(-)
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{SessionUser.java
=> LastQueryId.java} (89%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](Nereids) sink distribute is wrong when table is unpartitioned (#40541) (#40901)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 5c70fe35ea8 [fix](Nereids) sink distribute is wrong when table is
unpartitioned (#40541) (#40901)
5c70fe35ea8 is described below
commit 5c70fe35ea82bdac6cdf0c58c36b97010e6b30ef
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Wed Sep 18 15:18:41 2024 +0800
[fix](Nereids) sink distribute is wrong when table is unpartitioned
(#40541) (#40901)
pick from master #40541
---
.../doris/nereids/trees/plans/physical/PhysicalOlapTableSink.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalOlapTableSink.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalOlapTableSink.java
index dbe02884db2..a04912f5119 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalOlapTableSink.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalOlapTableSink.java
@@ -203,7 +203,7 @@ public class PhysicalOlapTableSink
extends PhysicalTabl
if (distributionInfo instanceof HashDistributionInfo) {
// Do not enable shuffle for duplicate key tables when its
tablet num is less than threshold.
if (targetTable.getKeysType() == KeysType.DUP_KEYS) {
-final long partitionNums =
targetTable.getPartitionInfo().getAllPartitions().size();
+final long partitionNums =
Math.max(targetTable.getPartitionInfo().getAllPartitions().size(), 1);
final long tabletNums = partitionNums *
distributionInfo.getBucketNum();
if (tabletNums < Config.min_tablets_for_dup_table_shuffle)
{
return PhysicalProperties.ANY;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](planner) throw NPE when insert overwrite partition(*) (#40656) (#40903)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 7d23a7c587c [fix](planner) throw NPE when insert overwrite
partition(*) (#40656) (#40903)
7d23a7c587c is described below
commit 7d23a7c587c2bcbb7768bd7622272698da1dcf54
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Wed Sep 18 13:48:48 2024 +0800
[fix](planner) throw NPE when insert overwrite partition(*) (#40656)
(#40903)
pick from master #40656
---
.../main/java/org/apache/doris/analysis/InsertOverwriteTableStmt.java | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/InsertOverwriteTableStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/InsertOverwriteTableStmt.java
index cfe030c428a..e32a38e4658 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/analysis/InsertOverwriteTableStmt.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/analysis/InsertOverwriteTableStmt.java
@@ -75,7 +75,9 @@ public class InsertOverwriteTableStmt extends DdlStmt {
}
public List getPartitionNames() {
-if (target.getPartitionNames() == null) {
+if (target.getPartitionNames() == null
+|| target.getPartitionNames().getPartitionNames() == null
+|| target.getPartitionNames().isStar()) {
return new ArrayList<>();
}
return target.getPartitionNames().getPartitionNames();
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](Nereids) avoid bad cast when compute scale for round (#40776) (#40904)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 3336e6acbb0 [fix](Nereids) avoid bad cast when compute scale for round
(#40776) (#40904)
3336e6acbb0 is described below
commit 3336e6acbb00c7b90a761000a3d3ff5b934e82a2
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Wed Sep 18 13:48:01 2024 +0800
[fix](Nereids) avoid bad cast when compute scale for round (#40776) (#40904)
pick from master #40776
for pass test case, also fix errors in computeResultInFe
computeResultInFe will generate wrong result set if output does not
match between final result and the node executing computeResultInFe
---
.../org/apache/doris/nereids/NereidsPlanner.java | 2 +-
.../functions/ComputePrecisionForRound.java| 4 +++-
.../nereids/trees/plans/ComputeResultSet.java | 5 +++-
.../plans/physical/PhysicalEmptyRelation.java | 6 ++---
.../plans/physical/PhysicalOneRowRelation.java | 28 +-
.../trees/plans/physical/PhysicalResultSink.java | 5 ++--
.../trees/plans/physical/PhysicalSqlCache.java | 2 +-
.../suites/correctness_p0/test_cast_decimal.groovy | 1 -
8 files changed, 31 insertions(+), 22 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
index 232ec168b3e..36ca1fa0992 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
@@ -579,7 +579,7 @@ public class NereidsPlanner extends Planner {
if (physicalPlan instanceof ComputeResultSet) {
Optional sqlCacheContext =
statementContext.getSqlCacheContext();
Optional resultSet = ((ComputeResultSet) physicalPlan)
-.computeResultInFe(cascadesContext, sqlCacheContext);
+.computeResultInFe(cascadesContext, sqlCacheContext,
physicalPlan.getOutput());
if (resultSet.isPresent()) {
return resultSet;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/ComputePrecisionForRound.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/ComputePrecisionForRound.java
index b47804e23ff..b07b7d384d8 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/ComputePrecisionForRound.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/ComputePrecisionForRound.java
@@ -40,8 +40,10 @@ public interface ComputePrecisionForRound extends
ComputePrecision {
// If scale arg is an integer literal, or it is a cast(Integer as
Integer)
// then we will try to use its value as result scale
// In any other cases, we will make sure result decimal has same
scale with input.
-if ((floatLength.isLiteral() && floatLength.getDataType()
instanceof Int32OrLessType)
+if ((floatLength.isLiteral() && !floatLength.isNullLiteral()
+&& floatLength.getDataType() instanceof Int32OrLessType)
|| (floatLength instanceof Cast &&
floatLength.child(0).isLiteral()
+&& !floatLength.child(0).isNullLiteral()
&& floatLength.child(0).getDataType() instanceof
Int32OrLessType)) {
if (floatLength instanceof Cast) {
scale = ((IntegerLikeLiteral)
floatLength.child(0)).getIntValue();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/ComputeResultSet.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/ComputeResultSet.java
index beee784ec9d..f86e143ca7b 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/ComputeResultSet.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/ComputeResultSet.java
@@ -19,8 +19,10 @@ package org.apache.doris.nereids.trees.plans;
import org.apache.doris.nereids.CascadesContext;
import org.apache.doris.nereids.SqlCacheContext;
+import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.qe.ResultSet;
+import java.util.List;
import java.util.Optional;
/**
@@ -51,5 +53,6 @@ import java.util.Optional;
*
*/
public interface ComputeResultSet {
-Optional computeResultInFe(CascadesContext cascadesContext,
Optional sqlCacheContext);
+Optional computeResultInFe(CascadesContext cascadesContext,
Optional sqlCacheContext,
+List outputSlots);
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalEmpt
(doris) branch audit_log deleted (was 7b10dd4ffee)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch audit_log in repository https://gitbox.apache.org/repos/asf/doris.git was 7b10dd4ffee [chore](audit) let line comment work well The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch 2.1_40066 deleted (was cf55d28ab2e)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch 2.1_40066 in repository https://gitbox.apache.org/repos/asf/doris.git was cf55d28ab2e [opt](Nereids) forbid one step topn with distribute as child (#40066) The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Fix](Nereids) fix append_trailing_char_if_absent function return null (#40820)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1501597e45f [Fix](Nereids) fix append_trailing_char_if_absent function
return null (#40820)
1501597e45f is described below
commit 1501597e45fa20adbe995b33f73a233032e32ec0
Author: LiBinfeng <46676950+libinfeng...@users.noreply.github.com>
AuthorDate: Wed Sep 18 11:01:25 2024 +0800
[Fix](Nereids) fix append_trailing_char_if_absent function return null
(#40820)
example: select append_trailing_char_if_absent('it','a') would return
null in original design, it can not return null when folding constant on
fe any time
---
.../functions/executable/ExecutableFunctions.java | 11 ---
.../functions/executable/StringArithmetic.java | 3 +
.../fold_constant_string_arithmatic.groovy | 88 ++
3 files changed, 91 insertions(+), 11 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/executable/ExecutableFunctions.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/executable/ExecutableFunctions.java
index a86e933c9ff..8195d456cc7 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/executable/ExecutableFunctions.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/executable/ExecutableFunctions.java
@@ -28,9 +28,7 @@ import
org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral;
import org.apache.doris.nereids.trees.expressions.literal.LargeIntLiteral;
import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
import org.apache.doris.nereids.trees.expressions.literal.SmallIntLiteral;
-import org.apache.doris.nereids.trees.expressions.literal.StringLikeLiteral;
import org.apache.doris.nereids.trees.expressions.literal.TinyIntLiteral;
-import org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral;
import org.apache.doris.nereids.types.DoubleType;
import java.math.BigInteger;
@@ -105,15 +103,6 @@ public class ExecutableFunctions {
}
}
-@ExecFunction(name = "append_trailing_char_if_absent")
-public static Expression appendTrailingIfCharAbsent(StringLikeLiteral
literal, StringLikeLiteral chr) {
-if (literal.getValue().length() != 1) {
-return null;
-}
-return literal.getValue().endsWith(chr.getValue()) ? literal
-: new VarcharLiteral(literal.getValue() + chr.getValue());
-}
-
@ExecFunction(name = "e")
public static Expression e() { // CHECKSTYLE IGNORE THIS LINE
return new DoubleLiteral(Math.E);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/executable/StringArithmetic.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/executable/StringArithmetic.java
index bc9cc29e7d0..5cf375aa3bb 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/executable/StringArithmetic.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/executable/StringArithmetic.java
@@ -794,6 +794,9 @@ public class StringArithmetic {
*/
@ExecFunction(name = "append_trailing_char_if_absent")
public static Expression appendTrailingCharIfAbsent(StringLikeLiteral
first, StringLikeLiteral second) {
+if (second.getValue().length() != 1) {
+return new NullLiteral(first.getDataType());
+}
if (first.getValue().endsWith(second.getValue())) {
return first;
} else {
diff --git
a/regression-test/suites/nereids_p0/expression/fold_constant/fold_constant_string_arithmatic.groovy
b/regression-test/suites/nereids_p0/expression/fold_constant/fold_constant_string_arithmatic.groovy
index 2bcdfc2fd24..02cf78f1cc6 100644
---
a/regression-test/suites/nereids_p0/expression/fold_constant/fold_constant_string_arithmatic.groovy
+++
b/regression-test/suites/nereids_p0/expression/fold_constant/fold_constant_string_arithmatic.groovy
@@ -197,6 +197,7 @@ suite("fold_constant_string_arithmatic") {
testFoldConst("select append_trailing_char_if_absent('a','c')")
testFoldConst("select append_trailing_char_if_absent('ac','c')")
+testFoldConst("select append_trailing_char_if_absent('it','a')")
testFoldConst("select ascii('1')")
testFoldConst("select ascii('a')")
@@ -684,4 +685,91 @@ suite("fold_constant_string_arithmatic") {
// fix problem of cast date and time function exception
testFoldConst("select ifnull(date
(doris) branch master updated (18a374f63c8 -> 55d6d6452f7)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 18a374f63c8 [fix](mtmv) Fix partition trace wrong when partition name is same from both side of join (#40485) add 55d6d6452f7 [Fix](nereids) set all nullable aggregate function to alwaysnullable in window expression (#40693) No new revisions were added by this update. Summary of changes: .../ExtractAndNormalizeWindowExpression.java | 16 +- .../normalize_window_nullable_agg_test.out | 293 + .../normalize_window_nullable_agg_test.groovy | 71 + 3 files changed, 369 insertions(+), 11 deletions(-) create mode 100644 regression-test/data/nereids_rules_p0/normalize_window/normalize_window_nullable_agg_test.out create mode 100644 regression-test/suites/nereids_rules_p0/normalize_window/normalize_window_nullable_agg_test.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (3af0943c392 -> 18a374f63c8)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 3af0943c392 [fix](restore) Fix clean restore with view (#40620) add 18a374f63c8 [fix](mtmv) Fix partition trace wrong when partition name is same from both side of join (#40485) No new revisions were added by this update. Summary of changes: .../rules/exploration/mv/MaterializedViewUtils.java | 18 ++ .../trees/plans/commands/info/CreateMTMVInfo.java| 4 +++- .../nereids/trees/plans/logical/LogicalOlapScan.java | 3 --- .../exploration/mv/MaterializedViewUtilsTest.java| 20 ...nable_date_non_deterministic_function_mtmv.groovy | 6 +++--- 5 files changed, 40 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (69b14b56a71 -> 7659de554ce)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 69b14b56a71 [Fix](cases) Make case success when meet expected error (#40713) add 7659de554ce [fix](Nereids) avoid bad cast when compute scale for round (#40776) No new revisions were added by this update. Summary of changes: .../org/apache/doris/nereids/NereidsPlanner.java | 2 +- .../functions/ComputePrecisionForRound.java| 4 +++- .../nereids/trees/plans/ComputeResultSet.java | 5 +++- .../plans/physical/PhysicalEmptyRelation.java | 6 ++--- .../plans/physical/PhysicalOneRowRelation.java | 28 +- .../trees/plans/physical/PhysicalResultSink.java | 5 ++-- .../trees/plans/physical/PhysicalSqlCache.java | 2 +- .../suites/correctness_p0/test_cast_decimal.groovy | 1 - 8 files changed, 31 insertions(+), 22 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [opt](Nereids) fix several insert into related issues (#40467) (#40755)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 51c8b62d1c4 [opt](Nereids) fix several insert into related issues
(#40467) (#40755)
51c8b62d1c4 is described below
commit 51c8b62d1c43bda2ae585a2fb4bf544bcf7c61eb
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Fri Sep 13 10:19:56 2024 +0800
[opt](Nereids) fix several insert into related issues (#40467) (#40755)
pick from master #40467
- http_stream TVF should always generate one fragment plan
- http_stream TVF plan should not check root as scan node
- distinguish group_commit TVF with normal insert statement
- index and generate slot should based on type cast base slot
- agg_state could cast from nullable to non-nullable
- colocated and bucket scan range compute should only on scan node
---
.../main/java/org/apache/doris/catalog/Type.java | 4 -
.../doris/nereids/parser/LogicalPlanBuilder.java | 2 +-
.../doris/nereids/rules/analysis/BindSink.java | 108 ++---
.../expressions/functions/table/HttpStream.java| 11 +++
.../trees/plans/commands/info/DMLCommandType.java | 2 +
.../main/java/org/apache/doris/qe/Coordinator.java | 5 +-
.../java/org/apache/doris/qe/StmtExecutor.java | 17 ++--
.../insert_into_table/insert_use_table_id.out | 48 -
.../insert_group_commit_into_unique.groovy | 9 +-
.../insert_into_table/insert_use_table_id.groovy | 107
10 files changed, 105 insertions(+), 208 deletions(-)
diff --git a/fe/fe-common/src/main/java/org/apache/doris/catalog/Type.java
b/fe/fe-common/src/main/java/org/apache/doris/catalog/Type.java
index 1831981f9f2..0d052d900b5 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/catalog/Type.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/catalog/Type.java
@@ -867,10 +867,6 @@ public abstract class Type {
return false;
}
for (int i = 0; i < sourceAggState.getSubTypes().size(); i++) {
-// target subtype is not null but source subtype is nullable
-if (!targetAggState.getSubTypeNullables().get(i) &&
sourceAggState.getSubTypeNullables().get(i)) {
-return false;
-}
if (!canCastTo(sourceAggState.getSubTypes().get(i),
targetAggState.getSubTypes().get(i))) {
return false;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
index 10523f0c76e..32cfe310f1e 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
@@ -572,7 +572,7 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor {
isAutoDetect,
isOverwrite,
ConnectContext.get().getSessionVariable().isEnableUniqueKeyPartialUpdate(),
-DMLCommandType.INSERT,
+ctx.tableId == null ? DMLCommandType.INSERT :
DMLCommandType.GROUP_COMMIT,
plan);
Optional cte = Optional.empty();
if (ctx.cte() != null) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java
index 0bda233e6f3..793ed5cc8f8 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java
@@ -67,6 +67,7 @@ import
org.apache.doris.nereids.trees.plans.visitor.InferPlanOutputAlias;
import org.apache.doris.nereids.types.DataType;
import org.apache.doris.nereids.types.StringType;
import org.apache.doris.nereids.types.coercion.CharacterType;
+import org.apache.doris.nereids.util.ExpressionUtils;
import org.apache.doris.nereids.util.RelationUtil;
import org.apache.doris.nereids.util.TypeCoercionUtils;
@@ -125,7 +126,8 @@ public class BindSink implements AnalysisRuleFactory {
&& table.getSequenceMapCol() != null
&& sink.getColNames().contains(table.getSequenceMapCol());
Pair, Integer> bindColumnsResult =
-bindTargetColumns(table, sink.getColNames(), childHasSeqCol,
needExtraSeqCol);
+bindTargetColumns(table, sink.getColNames(), childHasSeqCol,
needExtraSeqCol,
+sink.getDMLCommandType() ==
DMLCommandType.GROUP_COMMIT);
List bindColumns = bindColumnsResult.first;
int extraColumnsNum = bindColumnsResult.second;
(doris) branch master updated (4466541c1b4 -> 96c299e0d8e)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 4466541c1b4 [fix](statistics)Fix drop stats log editlog bug. Catch drop stats exception while truncate table. (#40738) add 96c299e0d8e [Fix](cases) fix test_use_mv (#40753) No new revisions were added by this update. Summary of changes: regression-test/suites/nereids_p0/hint/test_use_mv.groovy | 6 -- 1 file changed, 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [feature](Nereids) support query on ES table in internal catalog (#40449) (#40754)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 967801ca666 [feature](Nereids) support query on ES table in internal
catalog (#40449) (#40754)
967801ca666 is described below
commit 967801ca666f84a60010bef1f6ca5ca777d5b901
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Thu Sep 12 20:01:48 2024 +0800
[feature](Nereids) support query on ES table in internal catalog (#40449)
(#40754)
pick from master #40449
---
.../glue/translator/PhysicalPlanTranslator.java| 6 +-
.../doris/nereids/rules/analysis/BindRelation.java | 11 +---
.../nereids/trees/plans/logical/LogicalEsScan.java | 21 +++
.../trees/plans/physical/PhysicalEsScan.java | 11 +---
.../apache/doris/planner/StatisticDeriveTest.java | 27 -
.../test_push_conjunct_external_table.groovy | 67 --
6 files changed, 16 insertions(+), 127 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
index 129854b7ff9..78a6a083090 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
@@ -48,6 +48,7 @@ import org.apache.doris.catalog.TableIf;
import org.apache.doris.catalog.Type;
import org.apache.doris.datasource.ExternalTable;
import org.apache.doris.datasource.FileQueryScanNode;
+import org.apache.doris.datasource.es.EsExternalTable;
import org.apache.doris.datasource.es.source.EsScanNode;
import org.apache.doris.datasource.hive.HMSExternalTable;
import org.apache.doris.datasource.hive.HMSExternalTable.DLAType;
@@ -624,9 +625,10 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor slots = esScan.getOutput();
-ExternalTable table = esScan.getTable();
+TableIf table = esScan.getTable();
TupleDescriptor tupleDescriptor = generateTupleDesc(slots, table,
context);
-EsScanNode esScanNode = new EsScanNode(context.nextPlanNodeId(),
tupleDescriptor, true);
+EsScanNode esScanNode = new EsScanNode(context.nextPlanNodeId(),
tupleDescriptor,
+table instanceof EsExternalTable);
esScanNode.setNereidsId(esScan.getId());
esScanNode.addConjuncts(translateToLegacyConjuncts(esScan.getConjuncts()));
Utils.execWithUncheckedException(esScanNode::init);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java
index 67000b3fee9..4e18039c4b1 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java
@@ -33,7 +33,6 @@ import org.apache.doris.common.Config;
import org.apache.doris.common.Pair;
import org.apache.doris.common.util.Util;
import org.apache.doris.datasource.ExternalTable;
-import org.apache.doris.datasource.es.EsExternalTable;
import org.apache.doris.datasource.hive.HMSExternalTable;
import org.apache.doris.datasource.hive.HMSExternalTable.DLAType;
import org.apache.doris.nereids.CTEContext;
@@ -416,17 +415,11 @@ public class BindRelation extends OneAnalysisRuleFactory {
case ODBC:
return new
LogicalOdbcScan(unboundRelation.getRelationId(), table,
qualifierWithoutTableName);
case ES_EXTERNAL_TABLE:
-return new LogicalEsScan(unboundRelation.getRelationId(),
(EsExternalTable) table,
-qualifierWithoutTableName);
+case ELASTICSEARCH:
+return new LogicalEsScan(unboundRelation.getRelationId(),
table, qualifierWithoutTableName);
case TEST_EXTERNAL_TABLE:
return new
LogicalTestScan(unboundRelation.getRelationId(), table,
qualifierWithoutTableName);
default:
-try {
-// TODO: support other type table, such as
ELASTICSEARCH
-
cascadesContext.getConnectContext().getSessionVariable().enableFallbackToOriginalPlannerOnce();
-} catch (Exception e) {
-// ignore
-}
throw new AnalysisException("Unsupported tableType " +
table.getType());
}
} finally {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalEsScan.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalE
(doris) branch branch-2.1 updated: [opt](Nereids) support search from override udfs with same arity (#40432) (#40751)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new fedadbba6ec [opt](Nereids) support search from override udfs with same
arity (#40432) (#40751)
fedadbba6ec is described below
commit fedadbba6ece3bb744872f88261e5c923de23872
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Thu Sep 12 19:58:49 2024 +0800
[opt](Nereids) support search from override udfs with same arity (#40432)
(#40751)
pick from master #40432
create alias function f1(int) with parameter(id) as abs(id); create
alias function f1(string) with parameter(id) as substr(id, 2); select
f1('1'); -- bind on f1(string)
select f1(1); -- bind on f1(int)
test case already existed in P0
---
.../org/apache/doris/catalog/FunctionRegistry.java | 92 --
.../apache/doris/catalog/FunctionSignature.java| 34
.../trees/expressions/functions/udf/AliasUdf.java | 4 +-
.../expressions/functions/udf/AliasUdfBuilder.java | 20 ++---
.../expressions/functions/udf/JavaUdafBuilder.java | 6 ++
.../expressions/functions/udf/JavaUdfBuilder.java | 6 ++
.../expressions/functions/udf/UdfBuilder.java | 3 +
7 files changed, 124 insertions(+), 41 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionRegistry.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionRegistry.java
index 0e049b4b902..7280463b0f2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionRegistry.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionRegistry.java
@@ -21,7 +21,9 @@ import org.apache.doris.datasource.InternalCatalog;
import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.nereids.annotation.Developing;
import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.Expression;
import
org.apache.doris.nereids.trees.expressions.functions.AggCombinerFunctionBuilder;
+import
org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
import org.apache.doris.nereids.trees.expressions.functions.FunctionBuilder;
import org.apache.doris.nereids.trees.expressions.functions.udf.UdfBuilder;
import org.apache.doris.nereids.types.DataType;
@@ -156,18 +158,33 @@ public class FunctionRegistry {
+ "' which has " + arity + " arity. Candidate functions
are: " + candidateHints);
}
if (candidateBuilders.size() > 1) {
-String candidateHints = getCandidateHint(name, candidateBuilders);
-// TODO: NereidsPlanner not supported override function by the
same arity, we will support it later
-if (ConnectContext.get() != null) {
-try {
-
ConnectContext.get().getSessionVariable().enableFallbackToOriginalPlannerOnce();
-} catch (Throwable t) {
-// ignore error
+boolean needChooseOne = true;
+List signatures =
Lists.newArrayListWithCapacity(candidateBuilders.size());
+for (FunctionBuilder functionBuilder : candidateBuilders) {
+if (functionBuilder instanceof UdfBuilder) {
+signatures.addAll(((UdfBuilder)
functionBuilder).getSignatures());
+} else {
+needChooseOne = false;
+break;
+}
+}
+for (Object argument : arguments) {
+if (!(argument instanceof Expression)) {
+needChooseOne = false;
+break;
}
}
+if (needChooseOne) {
+FunctionSignature signature = new
UdfSignatureSearcher(signatures, (List) arguments).getSignature();
+for (int i = 0; i < signatures.size(); i++) {
+if (signatures.get(i).equals(signature)) {
+return candidateBuilders.get(i);
+}
+}
+}
+String candidateHints = getCandidateHint(name, candidateBuilders);
throw new AnalysisException("Function '" + qualifiedName + "' is
ambiguous: " + candidateHints);
}
-
return candidateBuilders.get(0);
}
@@ -235,4 +252,63 @@ public class FunctionRegistry {
.removeIf(builder -> ((UdfBuilder)
builder).getArgTypes().equals(argTypes));
}
}
+
+/**
+ * use for search appropriate signature for UDFs if candidate more than
one.
+ */
+static class UdfSignatureSearcher implements ExplicitlyCastableSignature {
+
+private final List signatures;
+private fin
(doris) branch branch-2.1 updated: [fix](nereids)let query be able to forwarded to master (#40119) (#40725)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 3484a6cfe3d [fix](nereids)let query be able to forwarded to master
(#40119) (#40725)
3484a6cfe3d is described below
commit 3484a6cfe3d9b024fbc5c5d70218316c6515385f
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Thu Sep 12 17:09:36 2024 +0800
[fix](nereids)let query be able to forwarded to master (#40119) (#40725)
pick from master #40119
---
.../main/java/org/apache/doris/qe/StmtExecutor.java | 21 +
.../doris/nereids/preprocess/SelectHintTest.java| 6 ++
2 files changed, 27 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
index 062f83443f9..ec28d53d90b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
@@ -744,6 +744,27 @@ public class StmtExecutor {
}
} else {
context.getState().setIsQuery(true);
+if (isForwardToMaster()) {
+// some times the follower's meta data is out of date.
+// so we need forward the query to master until the meta data
is sync with master
+if (context.getCommand() == MysqlCommand.COM_STMT_PREPARE) {
+throw new UserException("Forward master command is not
supported for prepare statement");
+}
+if (isProxy) {
+// This is already a stmt forwarded from other FE.
+// If we goes here, means we can't find a valid Master
FE(some error happens).
+// To avoid endless forward, throw exception here.
+throw new NereidsException(new UserException("The
statement has been forwarded to master FE("
++ Env.getCurrentEnv().getSelfNode().getHost() + ")
and failed to execute"
++ " because Master FE is not ready. You may need
to check FE's status"));
+}
+redirectStatus = RedirectStatus.NO_FORWARD;
+forwardToMaster();
+if (masterOpExecutor != null && masterOpExecutor.getQueryId()
!= null) {
+context.setQueryId(masterOpExecutor.getQueryId());
+}
+return;
+}
// create plan
// Query following createting table would throw table not exist
error.
// For example.
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/preprocess/SelectHintTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/preprocess/SelectHintTest.java
index f548a499cad..62965ac27b3 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/preprocess/SelectHintTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/preprocess/SelectHintTest.java
@@ -47,6 +47,12 @@ public class SelectHintTest {
return ctx;
}
};
+new MockUp() {
+@Mock
+public boolean isMaster() {
+return true;
+}
+};
}
@Test
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](nereids) build agg for random distributed agg table in bindRelation phase (#40181) (#40702)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 0f8176dee0a [fix](nereids) build agg for random distributed agg table
in bindRelation phase (#40181) (#40702)
0f8176dee0a is described below
commit 0f8176dee0a15893323b58b9e631dd9f45f1b6fb
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Thu Sep 12 14:08:50 2024 +0800
[fix](nereids) build agg for random distributed agg table in bindRelation
phase (#40181) (#40702)
pick from master #40181
---
.../doris/nereids/jobs/executor/Analyzer.java | 3 -
.../org/apache/doris/nereids/rules/RuleType.java | 4 -
.../doris/nereids/rules/analysis/BindRelation.java | 151 ++--
.../BuildAggForRandomDistributedTable.java | 271 -
.../doris/nereids/rules/analysis/CheckPolicy.java | 21 +-
.../nereids/rules/analysis/BindRelationTest.java | 31 ++-
.../nereids/rules/analysis/CheckRowPolicyTest.java | 97
.../aggregate/select_random_distributed_tbl.out| 14 +-
.../aggregate/select_random_distributed_tbl.groovy | 19 +-
9 files changed, 302 insertions(+), 309 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Analyzer.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Analyzer.java
index 605a848181c..1ffbac97d74 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Analyzer.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Analyzer.java
@@ -26,7 +26,6 @@ import org.apache.doris.nereids.rules.analysis.BindExpression;
import org.apache.doris.nereids.rules.analysis.BindRelation;
import
org.apache.doris.nereids.rules.analysis.BindRelation.CustomTableResolver;
import org.apache.doris.nereids.rules.analysis.BindSink;
-import
org.apache.doris.nereids.rules.analysis.BuildAggForRandomDistributedTable;
import org.apache.doris.nereids.rules.analysis.CheckAfterBind;
import org.apache.doris.nereids.rules.analysis.CheckAnalysis;
import org.apache.doris.nereids.rules.analysis.CheckPolicy;
@@ -163,8 +162,6 @@ public class Analyzer extends AbstractBatchJobExecutor {
topDown(new EliminateGroupByConstant()),
topDown(new SimplifyAggGroupBy()),
-// run BuildAggForRandomDistributedTable before NormalizeAggregate
in order to optimize the agg plan
-topDown(new BuildAggForRandomDistributedTable()),
topDown(new NormalizeAggregate()),
topDown(new HavingToFilter()),
bottomUp(new SemiJoinCommute()),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
index db0f0703dcb..082ee72fbed 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
@@ -336,10 +336,6 @@ public enum RuleType {
// topn opts
DEFER_MATERIALIZE_TOP_N_RESULT(RuleTypeClass.REWRITE),
-// pre agg for random distributed table
-BUILD_AGG_FOR_RANDOM_DISTRIBUTED_TABLE_PROJECT_SCAN(RuleTypeClass.REWRITE),
-BUILD_AGG_FOR_RANDOM_DISTRIBUTED_TABLE_FILTER_SCAN(RuleTypeClass.REWRITE),
-BUILD_AGG_FOR_RANDOM_DISTRIBUTED_TABLE_AGG_SCAN(RuleTypeClass.REWRITE),
// short circuit rule
SHOR_CIRCUIT_POINT_QUERY(RuleTypeClass.REWRITE),
// exploration rules
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java
index e6f550305e3..67000b3fee9 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java
@@ -17,10 +17,17 @@
package org.apache.doris.nereids.rules.analysis;
+import org.apache.doris.catalog.AggStateType;
+import org.apache.doris.catalog.AggregateType;
import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.DistributionInfo;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.FunctionRegistry;
+import org.apache.doris.catalog.KeysType;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.Partition;
import org.apache.doris.catalog.TableIf;
+import org.apache.doris.catalog.Type;
import org.apache.doris.catalog.View;
import org.apache.doris.common.Config;
import org.apache.doris.common.Pair;
@@ -44,13 +51,26 @@ import
org.apache.doris.nereids.properties.LogicalProperties;
import org.apache.doris.nereids.properties.PhysicalProperties;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
+
(doris) branch master updated: [refactor](nereids)use nereids to implement TableQueryPlanAction (#39627)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 89cf8d3f9b0 [refactor](nereids)use nereids to implement
TableQueryPlanAction (#39627)
89cf8d3f9b0 is described below
commit 89cf8d3f9b06d36a747b5a08372d1331a286395c
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Thu Sep 12 14:07:57 2024 +0800
[refactor](nereids)use nereids to implement TableQueryPlanAction (#39627)
---
.../doris/httpv2/rest/TableQueryPlanAction.java| 252 -
.../org/apache/doris/nereids/trees/TreeNode.java | 17 ++
.../doris/http/TableQueryPlanActionTest.java | 18 ++
3 files changed, 179 insertions(+), 108 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/TableQueryPlanAction.java
b/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/TableQueryPlanAction.java
index 6188b502692..8e03fef07f5 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/TableQueryPlanAction.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/TableQueryPlanAction.java
@@ -17,11 +17,7 @@
package org.apache.doris.httpv2.rest;
-import org.apache.doris.analysis.InlineViewRef;
-import org.apache.doris.analysis.SelectStmt;
import org.apache.doris.analysis.StatementBase;
-import org.apache.doris.analysis.TableName;
-import org.apache.doris.analysis.TableRef;
import org.apache.doris.catalog.Database;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.Table;
@@ -33,20 +29,31 @@ import org.apache.doris.common.util.NetUtils;
import org.apache.doris.httpv2.entity.ResponseEntityBuilder;
import org.apache.doris.httpv2.rest.manager.HttpUtils;
import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.analyzer.UnboundRelation;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.parser.NereidsParser;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.plans.commands.Command;
+import org.apache.doris.nereids.trees.plans.commands.ExplainCommand;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalResultSink;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSubQueryAlias;
+import org.apache.doris.nereids.util.RelationUtil;
import org.apache.doris.planner.PlanFragment;
-import org.apache.doris.planner.Planner;
import org.apache.doris.planner.ScanNode;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.GlobalVariable;
-import org.apache.doris.qe.OriginStatement;
-import org.apache.doris.qe.StmtExecutor;
+import org.apache.doris.qe.SessionVariable;
import org.apache.doris.thrift.TDataSink;
import org.apache.doris.thrift.TDataSinkType;
import org.apache.doris.thrift.TMemoryScratchSink;
import org.apache.doris.thrift.TNetworkAddress;
import org.apache.doris.thrift.TPaloScanRange;
import org.apache.doris.thrift.TPlanFragment;
-import org.apache.doris.thrift.TQueryOptions;
import org.apache.doris.thrift.TQueryPlanInfo;
import org.apache.doris.thrift.TScanRangeLocations;
import org.apache.doris.thrift.TTabletVersionInfo;
@@ -163,120 +170,149 @@ public class TableQueryPlanAction extends
RestBaseController {
*/
private void handleQuery(ConnectContext context, String requestDb, String
requestTable, String sql,
Map result) throws DorisHttpException {
-// use SE to resolve sql
-StmtExecutor stmtExecutor = new StmtExecutor(context, new
OriginStatement(sql, 0), false);
+List stmts = null;
+SessionVariable sessionVariable = context.getSessionVariable();
+boolean needSetParallelResultSinkToFalse = false;
try {
-TQueryOptions tQueryOptions =
context.getSessionVariable().toThrift();
-// Conduct Planner create SingleNodePlan#createPlanFragments
-tQueryOptions.num_nodes = 1;
-// analyze sql
-stmtExecutor.analyze(tQueryOptions);
-} catch (Exception e) {
-throw new DorisHttpException(HttpResponseStatus.BAD_REQUEST,
e.getMessage());
-}
-// the parsed logical statement
-StatementBase query = stmtExecutor.getParsedStmt();
-// only process select semantic
-if (!(query instanceof SelectStmt)) {
-throw new DorisHttpException(HttpResponseStatus.BAD_REQUEST,
-"Select statement needed, but found [" + sql + " ]");
-
(doris) branch master updated (de90051162d -> e20030c0da2)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from de90051162d [Fix](Job)Replaying logs should not modify the original information of the job (#40474) add e20030c0da2 [chore](Nereids) remove enable_fallback_to_original_planner variable (#40416) No new revisions were added by this update. Summary of changes: .../org/apache/doris/job/extensions/insert/InsertTask.java| 1 - .../main/java/org/apache/doris/load/ExportTaskExecutor.java | 1 - .../src/main/java/org/apache/doris/mtmv/MTMVPlanUtil.java | 1 - .../org/apache/doris/nereids/rules/analysis/BindSink.java | 7 +-- .../nereids/rules/analysis/EliminateLogicalSelectHint.java| 8 ++-- .../src/main/java/org/apache/doris/qe/ConnectProcessor.java | 8 +--- .../src/main/java/org/apache/doris/qe/SessionVariable.java| 11 +-- .../src/main/java/org/apache/doris/qe/StmtExecutor.java | 5 ++--- .../java/org/apache/doris/analysis/BulkLoadDataDescTest.java | 1 - .../java/org/apache/doris/catalog/CreateFunctionTest.java | 1 - .../apache/doris/datasource/hive/HiveDDLAndDMLPlanTest.java | 1 - .../java/org/apache/doris/external/hms/HmsCatalogTest.java| 1 - .../org/apache/doris/nereids/preprocess/SelectHintTest.java | 7 ++- .../agg_state/nereids/test_agg_state_nereids.groovy | 4 .../suites/datatype_p0/decimalv3/test_decimalv3.groovy| 2 -- regression-test/suites/javaudf_p0/test_javaudtf_int.groovy| 4 ++-- .../suites/json_p0/test_json_predict_is_null.groovy | 5 - regression-test/suites/jsonb_p0/test_jsonb_cast.groovy| 3 --- .../suites/jsonb_p0/test_jsonb_predict_is_null.groovy | 5 - .../suites/jsonb_p0/test_jsonb_with_unescaped_string.groovy | 3 --- .../suites/nereids_p0/insert_into_table/no_partition.groovy | 7 --- .../suites/nereids_p0/insert_into_table/partial_update.groovy | 1 - .../insert_into_table/partial_update_complex.groovy | 1 - .../suites/nereids_p0/insert_into_table/unsupport_type.groovy | 4 .../suites/nereids_p0/test_timeout_fallback.groovy| 1 - .../nereids_rules_p0/grouping_sets/valid_grouping.groovy | 5 - .../suites/nereids_syntax_p0/aggregate_strategies.groovy | 7 --- .../suites/nereids_syntax_p0/filter_to_select.groovy | 5 + regression-test/suites/nereids_syntax_p0/having.groovy| 3 --- .../suites/nereids_syntax_p0/null_aware_left_anti_join.groovy | 9 - .../partition_p0/multi_partition/test_multi_partition.groovy | 1 - .../partition_p0/multi_partition/test_range_partition.groovy | 2 -- regression-test/suites/point_query_p0/test_point_query.groovy | 1 - 33 files changed, 16 insertions(+), 110 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Fix](Job)Replace BlockingWaitStrategy with LiteTimeoutBlockingWaitStrategy to avoid deadlock issues. (#40625)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 087048f2b2f [Fix](Job)Replace BlockingWaitStrategy with
LiteTimeoutBlockingWaitStrategy to avoid deadlock issues. (#40625)
087048f2b2f is described below
commit 087048f2b2f06eae15e80e7ffa209610c9e3f173
Author: Calvin Kirs
AuthorDate: Thu Sep 12 00:18:48 2024 +0800
[Fix](Job)Replace BlockingWaitStrategy with LiteTimeoutBlockingWaitStrategy
to avoid deadlock issues. (#40625)
FYI https://issues.apache.org/jira/browse/LOG4J2-1221
- BlockingWaitStrategy is a wait strategy used in the Disruptor
framework that blocks the thread when the ring buffer is full or not yet
available for publishing.
When threads are blocked, they are waiting for space in the ring buffer
to become available, which can lead to potential deadlocks if not
managed properly.
Timeout Handling:
- LiteTimeoutBlockingWaitStrategy provides a timeout for waiting
threads. If the buffer is not ready within the timeout period, the
thread is released, preventing it from being blocked indefinitely.
Reduced Risk of Deadlocks:
- By avoiding indefinite blocking, this strategy reduces the risk of
deadlocks caused by threads waiting on each other. The timeout allows
the system to handle scenarios where resources are temporarily
---
.../apache/doris/job/disruptor/TaskDisruptor.java | 15 ---
.../doris/job/executor/DispatchTaskHandler.java| 5 -
.../doris/job/executor/TimerJobSchedulerTask.java | 5 -
.../job/manager/TaskDisruptorGroupManager.java | 22 +++---
.../apache/doris/job/scheduler/JobScheduler.java | 13 -
.../doris/scheduler/disruptor/TaskDisruptor.java | 10 +-
6 files changed, 44 insertions(+), 26 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/job/disruptor/TaskDisruptor.java
b/fe/fe-core/src/main/java/org/apache/doris/job/disruptor/TaskDisruptor.java
index 45564e99b17..6ca2924c593 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/job/disruptor/TaskDisruptor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/job/disruptor/TaskDisruptor.java
@@ -24,6 +24,8 @@ import com.lmax.disruptor.WaitStrategy;
import com.lmax.disruptor.WorkHandler;
import com.lmax.disruptor.dsl.Disruptor;
import com.lmax.disruptor.dsl.ProducerType;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
import java.util.concurrent.ThreadFactory;
@@ -33,6 +35,7 @@ import java.util.concurrent.ThreadFactory;
* @param the type of the event handled by the Disruptor
*/
public class TaskDisruptor {
+private static final Logger LOG =
LogManager.getLogger(TaskDisruptor.class);
private final Disruptor disruptor;
private final EventTranslatorVararg eventTranslator;
@@ -68,9 +71,15 @@ public class TaskDisruptor {
*
* @param args the arguments for the event
*/
-public void publishEvent(Object... args) {
-RingBuffer ringBuffer = disruptor.getRingBuffer();
-ringBuffer.publishEvent(eventTranslator, args);
+public boolean publishEvent(Object... args) {
+try {
+RingBuffer ringBuffer = disruptor.getRingBuffer();
+return ringBuffer.tryPublishEvent(eventTranslator, args);
+} catch (Exception e) {
+LOG.warn("Failed to publish event", e);
+// Handle the exception, e.g., retry or alert
+}
+return false;
}
/**
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/job/executor/DispatchTaskHandler.java
b/fe/fe-core/src/main/java/org/apache/doris/job/executor/DispatchTaskHandler.java
index e5933d133cb..d93393aa0ef 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/job/executor/DispatchTaskHandler.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/job/executor/DispatchTaskHandler.java
@@ -66,7 +66,10 @@ public class DispatchTaskHandler
implements WorkHandler
implements TimerTask {
log.info("job status is not running, job id is {}, skip
dispatch", this.job.getJobId());
return;
}
-dispatchDisruptor.publishEvent(this.job);
+if (!dispatchDisruptor.publishEvent(this.job)) {
+log.warn("dispatch timer job failed, job id is {}, job name is
{}",
+this.job.getJobId(), this.job.getJobName());
+}
} catch (Exception e) {
log.warn("dispatch timer job error, task id is {}",
this.job.getJobId(), e);
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/job/manager/TaskDisruptorGroupManager.java
b/fe/fe-core/src/main/java/org/apache/doris/job/manager/TaskDisruptorGroupMan
(doris) branch master updated: [fix](auth)show config command should check auth (#40533)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d0cb1283f5f [fix](auth)show config command should check auth (#40533)
d0cb1283f5f is described below
commit d0cb1283f5f7563a7f67186be4211fabcdc4d26e
Author: zhangdong <493738...@qq.com>
AuthorDate: Wed Sep 11 22:54:45 2024 +0800
[fix](auth)show config command should check auth (#40533)
The old optimizer checked permissions, but the new optimizer did not
check permissions,
We keep the permissions of the new optimizer and the old optimizer
consistent.
ps: We will add cases to other PRs as a whole
---
.../doris/nereids/trees/plans/commands/ShowConfigCommand.java | 7 +++
1 file changed, 7 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowConfigCommand.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowConfigCommand.java
index b139d0a5772..77bb5ee83d3 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowConfigCommand.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowConfigCommand.java
@@ -23,8 +23,11 @@ import org.apache.doris.catalog.ScalarType;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.CaseSensibility;
import org.apache.doris.common.ConfigBase;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
import org.apache.doris.common.PatternMatcher;
import org.apache.doris.common.PatternMatcherWrapper;
+import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.nereids.trees.plans.PlanType;
import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
import org.apache.doris.qe.ConnectContext;
@@ -157,6 +160,10 @@ public class ShowConfigCommand extends Command implements
NoForward {
@Override
public void run(ConnectContext ctx, StmtExecutor executor) throws
Exception {
+// check auth
+if
(!Env.getCurrentEnv().getAccessManager().checkGlobalPriv(ConnectContext.get(),
PrivPredicate.ADMIN)) {
+
ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR,
"ADMIN");
+}
if (nodeType == NodeType.FRONTEND) {
executor.sendResultSet(handShowFrontendConfig());
} else {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](default value) Avoid default precision value of 0 for default expressions during deserialization (#40671)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f597d72a828 [fix](default value) Avoid default precision value of 0
for default expressions during deserialization (#40671)
f597d72a828 is described below
commit f597d72a828e29405d8226c13e27fc2c9477a818
Author: zy-kkk
AuthorDate: Wed Sep 11 22:47:04 2024 +0800
[fix](default value) Avoid default precision value of 0 for default
expressions during deserialization (#40671)
This PR fixes the issue where the precision of default value expressions
was incorrectly set to `0` during deserialization. The `gsonPostProcess`
method no longer assigns a default precision of `0` if the precision is
not provided. Additionally, logic has been added to ensure that
expressions without precision do not incorrectly apply a precision of
`0`.
---
.../main/java/org/apache/doris/analysis/DefaultValueExprDef.java| 6 ++
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/DefaultValueExprDef.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/DefaultValueExprDef.java
index 30ab756bb51..44bc766ee47 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/analysis/DefaultValueExprDef.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/analysis/DefaultValueExprDef.java
@@ -64,7 +64,7 @@ public class DefaultValueExprDef implements Writable,
GsonPostProcessable {
*/
public FunctionCallExpr getExpr(Type type) {
List exprs = null;
-if (precision != null) {
+if (precision != null && precision != 0) {
exprs = Lists.newArrayList();
exprs.add(new IntLiteral(precision));
}
@@ -101,8 +101,6 @@ public class DefaultValueExprDef implements Writable,
GsonPostProcessable {
@Override
public void gsonPostProcess() throws IOException {
-if (precision == null) {
-precision = 0L;
-}
+// nothing to do
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [chore](audit) let line comment work well (#40599)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new aab6204bbb9 [chore](audit) let line comment work well (#40599)
aab6204bbb9 is described below
commit aab6204bbb9abf3466ae0608e35d31291a40de90
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Wed Sep 11 22:44:42 2024 +0800
[chore](audit) let line comment work well (#40599)
should replace new line character with '\n', not ' '
---
.../src/main/java/org/apache/doris/qe/AuditLogHelper.java| 12 ++--
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/AuditLogHelper.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/AuditLogHelper.java
index 904910822b8..c4b6d20688c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/AuditLogHelper.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/AuditLogHelper.java
@@ -89,14 +89,14 @@ public class AuditLogHelper {
}
int maxLen = GlobalVariable.auditPluginMaxSqlLength;
if (origStmt.length() <= maxLen) {
-return origStmt.replace("\n", " ")
-.replace("\t", " ")
-.replace("\r", " ");
+return origStmt.replace("\n", "\\n")
+.replace("\t", "\\t")
+.replace("\r", "\\r");
}
origStmt = truncateByBytes(origStmt)
-.replace("\n", " ")
-.replace("\t", " ")
-.replace("\r", " ");
+.replace("\n", "\\n")
+.replace("\t", "\\t")
+.replace("\r", "\\r");
int rowCnt = 0;
// old planner
if (parsedStmt instanceof NativeInsertStmt) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](planner) throw NPE when insert overwrite partition(*) (#40656)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 77fb56e0c1f [fix](planner) throw NPE when insert overwrite
partition(*) (#40656)
77fb56e0c1f is described below
commit 77fb56e0c1f35fcf5eaba8a9d56799fd074bb5fa
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Wed Sep 11 22:36:47 2024 +0800
[fix](planner) throw NPE when insert overwrite partition(*) (#40656)
---
.../main/java/org/apache/doris/analysis/InsertOverwriteTableStmt.java | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/InsertOverwriteTableStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/InsertOverwriteTableStmt.java
index a0fca62e284..aa805af6583 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/analysis/InsertOverwriteTableStmt.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/analysis/InsertOverwriteTableStmt.java
@@ -75,7 +75,9 @@ public class InsertOverwriteTableStmt extends DdlStmt
implements NotFallbackInPa
}
public List getPartitionNames() {
-if (target.getPartitionNames() == null) {
+if (target.getPartitionNames() == null
+|| target.getPartitionNames().getPartitionNames() == null
+|| target.getPartitionNames().isStar()) {
return new ArrayList<>();
}
return target.getPartitionNames().getPartitionNames();
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [improvement](mtmv) Support to add use_for_rewrite property when create materialized view (#40332)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new cd2062ae8df [improvement](mtmv) Support to add use_for_rewrite
property when create materialized view (#40332)
cd2062ae8df is described below
commit cd2062ae8dfca5895d41f651b250afa2969c2215
Author: seawinde <149132972+seawi...@users.noreply.github.com>
AuthorDate: Wed Sep 11 19:03:56 2024 +0800
[improvement](mtmv) Support to add use_for_rewrite property when create
materialized view (#40332)
Add `is_used_in_rewrite` property when create mv. Default true;
If `is_used_in_rewrite` is false which means the mv would not partion in
query rewrite.
Such as mv def is as following:
CREATE MATERIALIZED VIEW mv1
BUILD IMMEDIATE REFRESH AUTO ON MANUAL
DISTRIBUTED BY RANDOM BUCKETS 2
PROPERTIES (
'replication_num' = '1',
'use_for_rewrite' = 'false'
)
AS
SELECT k1, k2, count(*) from t1 group by k1, k2;
if we run query as following, this mv would not partition in the query
rewrite:
SELECT k2, count(*) from t1 group by k2;
---
.../main/java/org/apache/doris/catalog/MTMV.java | 13 +++
.../apache/doris/common/util/PropertyAnalyzer.java | 3 +
.../org/apache/doris/mtmv/MTMVPropertyUtil.java| 15 ++-
.../mv/InitMaterializationContextHook.java | 11 ++-
.../suites/mtmv_p0/test_use_for_rewrite.groovy | 103 +
5 files changed, 142 insertions(+), 3 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
index 3e60a489c93..f93ecc9475c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
@@ -245,6 +245,19 @@ public class MTMV extends OlapTable {
}
}
+public boolean isUseForRewrite() {
+readMvLock();
+try {
+if
(!StringUtils.isEmpty(mvProperties.get(PropertyAnalyzer.PROPERTIES_USE_FOR_REWRITE)))
{
+return
Boolean.valueOf(mvProperties.get(PropertyAnalyzer.PROPERTIES_USE_FOR_REWRITE));
+}
+// default is true
+return true;
+} finally {
+readMvUnlock();
+}
+}
+
public int getRefreshPartitionNum() {
readMvLock();
try {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java
b/fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java
index 8441c9307ca..3d95e69cb67 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java
@@ -186,6 +186,9 @@ public class PropertyAnalyzer {
public static final String PROPERTIES_ENABLE_NONDETERMINISTIC_FUNCTION =
"enable_nondeterministic_function";
+
+public static final String PROPERTIES_USE_FOR_REWRITE =
+"use_for_rewrite";
public static final String PROPERTIES_EXCLUDED_TRIGGER_TABLES =
"excluded_trigger_tables";
public static final String PROPERTIES_REFRESH_PARTITION_NUM =
"refresh_partition_num";
public static final String PROPERTIES_WORKLOAD_GROUP = "workload_group";
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPropertyUtil.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPropertyUtil.java
index 12287183886..dbaccaf9247 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPropertyUtil.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPropertyUtil.java
@@ -38,7 +38,8 @@ public class MTMVPropertyUtil {
PropertyAnalyzer.PROPERTIES_PARTITION_SYNC_LIMIT,
PropertyAnalyzer.PROPERTIES_PARTITION_TIME_UNIT,
PropertyAnalyzer.PROPERTIES_PARTITION_DATE_FORMAT,
-PropertyAnalyzer.PROPERTIES_ENABLE_NONDETERMINISTIC_FUNCTION
+PropertyAnalyzer.PROPERTIES_ENABLE_NONDETERMINISTIC_FUNCTION,
+PropertyAnalyzer.PROPERTIES_USE_FOR_REWRITE
);
public static void analyzeProperty(String key, String value) {
@@ -65,6 +66,10 @@ public class MTMVPropertyUtil {
analyzePartitionSyncLimit(value);
break;
case PropertyAnalyzer.PROPERTIES_ENABLE_NONDETERMINISTIC_FUNCTION:
+analyzeBooleanProperty(value,
PropertyAnalyzer.PROPERTIES_ENABLE_NONDETERMINISTIC_FUNCTION);
+break;
+case PropertyAnalyzer.PROPERTIES_USE_FOR_REWRITE:
+analyzeBooleanProperty(value,
PropertyAnalyzer.PROPERTIES_USE_FOR_REWR
(doris) branch master updated: [fix](nereids) build agg for random distributed agg table in bindRelation phase (#40181)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new dca6b8175d7 [fix](nereids) build agg for random distributed agg table
in bindRelation phase (#40181)
dca6b8175d7 is described below
commit dca6b8175d7557b0b91c7a5c97d2656b02dced6f
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Wed Sep 11 18:16:03 2024 +0800
[fix](nereids) build agg for random distributed agg table in bindRelation
phase (#40181)
it's better to build agg for random distributed agg table in
bindRelation phase instead of in BuildAggForRandomDistributedTable RULE
---
.../doris/nereids/jobs/executor/Analyzer.java | 3 -
.../org/apache/doris/nereids/rules/RuleType.java | 4 -
.../doris/nereids/rules/analysis/BindRelation.java | 125 +-
.../BuildAggForRandomDistributedTable.java | 271 -
.../doris/nereids/rules/analysis/CheckPolicy.java | 21 +-
.../nereids/rules/analysis/BindRelationTest.java | 23 ++
.../nereids/rules/analysis/CheckRowPolicyTest.java | 97
.../aggregate/select_random_distributed_tbl.out| 14 +-
.../aggregate/select_random_distributed_tbl.groovy | 19 +-
9 files changed, 285 insertions(+), 292 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Analyzer.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Analyzer.java
index 605a848181c..1ffbac97d74 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Analyzer.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Analyzer.java
@@ -26,7 +26,6 @@ import org.apache.doris.nereids.rules.analysis.BindExpression;
import org.apache.doris.nereids.rules.analysis.BindRelation;
import
org.apache.doris.nereids.rules.analysis.BindRelation.CustomTableResolver;
import org.apache.doris.nereids.rules.analysis.BindSink;
-import
org.apache.doris.nereids.rules.analysis.BuildAggForRandomDistributedTable;
import org.apache.doris.nereids.rules.analysis.CheckAfterBind;
import org.apache.doris.nereids.rules.analysis.CheckAnalysis;
import org.apache.doris.nereids.rules.analysis.CheckPolicy;
@@ -163,8 +162,6 @@ public class Analyzer extends AbstractBatchJobExecutor {
topDown(new EliminateGroupByConstant()),
topDown(new SimplifyAggGroupBy()),
-// run BuildAggForRandomDistributedTable before NormalizeAggregate
in order to optimize the agg plan
-topDown(new BuildAggForRandomDistributedTable()),
topDown(new NormalizeAggregate()),
topDown(new HavingToFilter()),
bottomUp(new SemiJoinCommute()),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
index b6ab6e2dac2..d345d9057e9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
@@ -342,10 +342,6 @@ public enum RuleType {
// topn opts
DEFER_MATERIALIZE_TOP_N_RESULT(RuleTypeClass.REWRITE),
-// pre agg for random distributed table
-BUILD_AGG_FOR_RANDOM_DISTRIBUTED_TABLE_PROJECT_SCAN(RuleTypeClass.REWRITE),
-BUILD_AGG_FOR_RANDOM_DISTRIBUTED_TABLE_FILTER_SCAN(RuleTypeClass.REWRITE),
-BUILD_AGG_FOR_RANDOM_DISTRIBUTED_TABLE_AGG_SCAN(RuleTypeClass.REWRITE),
// short circuit rule
SHOR_CIRCUIT_POINT_QUERY(RuleTypeClass.REWRITE),
// exploration rules
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java
index cedc4e92ff1..c81fcc25b83 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java
@@ -17,10 +17,17 @@
package org.apache.doris.nereids.rules.analysis;
+import org.apache.doris.catalog.AggStateType;
+import org.apache.doris.catalog.AggregateType;
import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.DistributionInfo;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.FunctionRegistry;
+import org.apache.doris.catalog.KeysType;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.Partition;
import org.apache.doris.catalog.TableIf;
+import org.apache.doris.catalog.Type;
import org.apache.doris.catalog.View;
import org.apache.doris.common.Config;
import org.apache.doris.common.Pair;
@@ -43,13 +50,26 @@ import
org.apache.doris.nereids.properties.LogicalProperties;
import org.apache.doris.nereids.properties.PhysicalProperties;
import org.apache.doris.nereids
(doris) branch master updated (36828f2994e -> 898d5254ccb)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 36828f2994e [Fix](load) fix wrong replica number when loading to table with no partition (#40635) add 898d5254ccb [fix](Nereids) sink distribute is wrong when table is unpartitioned (#40541) No new revisions were added by this update. Summary of changes: .../doris/nereids/trees/plans/physical/PhysicalOlapTableSink.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (d8c34beffa5 -> 3be6ed40eae)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from d8c34beffa5 [fix](ES Catalog)Support parse single value for array
column (#40614)
add 3be6ed40eae [improvement](jdbc catalog) support jdbc external catalog
insert stmt in nereids (#39813)
No new revisions were added by this update.
Summary of changes:
.../nereids/analyzer/UnboundJdbcTableSink.java | 84
.../nereids/analyzer/UnboundTableSinkCreator.java | 20 +--
.../glue/translator/PhysicalPlanTranslator.java| 20 +++
.../pre/TurnOffPageCacheForInsertIntoSelect.java | 8 ++
.../nereids/properties/RequestPropertyDeriver.java | 9 ++
.../org/apache/doris/nereids/rules/RuleSet.java| 2 +
.../org/apache/doris/nereids/rules/RuleType.java | 2 +
.../doris/nereids/rules/analysis/BindSink.java | 77 ++-
...ogicalJdbcTableSinkToPhysicalJdbcTableSink.java | 48 +++
.../apache/doris/nereids/trees/plans/PlanType.java | 3 +
.../insert/BaseExternalTableInsertExecutor.java| 2 +-
.../commands/insert/InsertIntoTableCommand.java| 27 +++-
.../trees/plans/commands/insert/InsertUtils.java | 9 ++
.../commands/insert/JdbcInsertCommandContext.java} | 10 +-
.../plans/commands/insert/JdbcInsertExecutor.java | 113 +++
.../trees/plans/logical/LogicalJdbcTableSink.java | 151 +
.../physical/PhysicalBaseExternalTableSink.java| 4 +
.../plans/physical/PhysicalJdbcTableSink.java | 109 +++
.../nereids/trees/plans/visitor/SinkVisitor.java | 15 ++
...actionType.java => JdbcTransactionManager.java} | 26 +++-
.../apache/doris/transaction/TransactionType.java | 3 +-
21 files changed, 719 insertions(+), 23 deletions(-)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundJdbcTableSink.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalJdbcTableSinkToPhysicalJdbcTableSink.java
copy
fe/fe-core/src/main/java/org/apache/doris/{transaction/TransactionType.java =>
nereids/trees/plans/commands/insert/JdbcInsertCommandContext.java} (81%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/JdbcInsertExecutor.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalJdbcTableSink.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalJdbcTableSink.java
copy
fe/fe-core/src/main/java/org/apache/doris/transaction/{TransactionType.java =>
JdbcTransactionManager.java} (67%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [Fix](nereids) fix partition prune not expression compute logic (#40619)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 37e5e99e80e [Fix](nereids) fix partition prune not expression compute
logic (#40619)
37e5e99e80e is described below
commit 37e5e99e80e24175bfa2a0f9008f6fcc8b252412
Author: feiniaofeiafei <53502832+feiniaofeia...@users.noreply.github.com>
AuthorDate: Wed Sep 11 11:49:00 2024 +0800
[Fix](nereids) fix partition prune not expression compute logic (#40619)
---
.../rules/OneRangePartitionEvaluator.java | 18 ++---
.../test_reject_not_default_false.out | 4 ++
.../test_reject_not_default_false.groovy | 43 ++
3 files changed, 59 insertions(+), 6 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/OneRangePartitionEvaluator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/OneRangePartitionEvaluator.java
index a3f646fb74e..adbf97091cd 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/OneRangePartitionEvaluator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/OneRangePartitionEvaluator.java
@@ -251,6 +251,7 @@ public class OneRangePartitionEvaluator
Map leftColumnRanges =
result.childrenResult.get(0).columnRanges;
ColumnRange greaterThenRange =
ColumnRange.greaterThan((Literal) greaterThan.right());
result = intersectSlotRange(result, leftColumnRanges, slot,
greaterThenRange);
+result = result.withRejectNot(true);
}
} else if (greaterThan.left() instanceof Literal &&
greaterThan.right() instanceof Slot) {
Slot slot = (Slot) greaterThan.right();
@@ -258,6 +259,7 @@ public class OneRangePartitionEvaluator
Map rightColumnRanges =
result.childrenResult.get(1).columnRanges;
ColumnRange lessThenRange = ColumnRange.lessThen((Literal)
greaterThan.left());
result = intersectSlotRange(result, rightColumnRanges, slot,
lessThenRange);
+result = result.withRejectNot(true);
}
}
return result;
@@ -276,6 +278,7 @@ public class OneRangePartitionEvaluator
Map leftColumnRanges =
result.childrenResult.get(0).columnRanges;
ColumnRange atLeastRange = ColumnRange.atLeast((Literal)
greaterThanEqual.right());
result = intersectSlotRange(result, leftColumnRanges, slot,
atLeastRange);
+result = result.withRejectNot(true);
}
} else if (greaterThanEqual.left() instanceof Literal &&
greaterThanEqual.right() instanceof Slot) {
Slot slot = (Slot) greaterThanEqual.right();
@@ -283,6 +286,7 @@ public class OneRangePartitionEvaluator
Map rightColumnRanges =
result.childrenResult.get(1).columnRanges;
ColumnRange atMostRange = ColumnRange.atMost((Literal)
greaterThanEqual.left());
result = intersectSlotRange(result, rightColumnRanges, slot,
atMostRange);
+result = result.withRejectNot(true);
}
}
return result;
@@ -301,6 +305,7 @@ public class OneRangePartitionEvaluator
Map leftColumnRanges =
result.childrenResult.get(0).columnRanges;
ColumnRange greaterThenRange = ColumnRange.lessThen((Literal)
lessThan.right());
result = intersectSlotRange(result, leftColumnRanges, slot,
greaterThenRange);
+result = result.withRejectNot(true);
}
} else if (lessThan.left() instanceof Literal && lessThan.right()
instanceof Slot) {
Slot slot = (Slot) lessThan.right();
@@ -308,6 +313,7 @@ public class OneRangePartitionEvaluator
Map rightColumnRanges =
result.childrenResult.get(1).columnRanges;
ColumnRange lessThenRange = ColumnRange.greaterThan((Literal)
lessThan.left());
result = intersectSlotRange(result, rightColumnRanges, slot,
lessThenRange);
+result = result.withRejectNot(true);
}
}
return result;
@@ -326,6 +332,7 @@ public class OneRangePartitionEvaluator
Map leftColumnRanges =
result.childrenResult.get(0).columnRanges;
ColumnRange atLeastRange = ColumnRange.atMost((Literal)
lessThanEqual.right());
result = intersectSlotRange(result, leftColumnRanges, slot,
atLeastRange);
+result = result.withRejectNot(true);
}
} else if (lessThanEqual.left() instanceof Literal &&
lessThanEqual.right() instanceof Slot) {
Slot slot = (Slot) lessThanEqual.rig
(doris) branch master updated (cd902b61194 -> 948fb2ab57a)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from cd902b61194 [refactor](pipeline) Refactor logics (#40576) add 948fb2ab57a [improvement](mtmv) Optimize the materialized view hint info when explain (#39998) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/doris/nereids/NereidsPlanner.java | 10 ++ .../nereids/rules/exploration/mv/MaterializationContext.java | 2 +- .../nereids_rules_p0/mv/same_name/sync_async_same_name.groovy | 2 +- 3 files changed, 8 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch audit_log created (now 0ead57a060c)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch audit_log in repository https://gitbox.apache.org/repos/asf/doris.git at 0ead57a060c [chore](audit) let line comment work well This branch includes the following new commits: new 0ead57a060c [chore](audit) let line comment work well The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [fix](auth) ordinary users can see the processes of other users (#39747) (#40417)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 921901ac454 [fix](auth) ordinary users can see the processes of other
users (#39747) (#40417)
921901ac454 is described below
commit 921901ac454dbef2a41dc3fd34a6b8a60d23833b
Author: zhangdong <493738...@qq.com>
AuthorDate: Mon Sep 9 19:18:22 2024 +0800
[fix](auth) ordinary users can see the processes of other users (#39747)
(#40417)
pick from master #39747
---
.../src/main/java/org/apache/doris/qe/ConnectScheduler.java| 10 +-
fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java | 1 +
.../java/org/apache/doris/service/FrontendServiceImpl.java | 6 +-
gensrc/thrift/FrontendService.thrift | 1 +
4 files changed, 16 insertions(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectScheduler.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectScheduler.java
index 4000ac7b1d4..da3fc49aaf7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectScheduler.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectScheduler.java
@@ -17,6 +17,7 @@
package org.apache.doris.qe;
+import org.apache.doris.analysis.UserIdentity;
import org.apache.doris.catalog.Env;
import org.apache.doris.common.Config;
import org.apache.doris.common.ThreadPoolManager;
@@ -161,10 +162,17 @@ public class ConnectScheduler {
}
// used for thrift
-public List> listConnectionWithoutAuth(boolean isShowFullSql,
boolean isShowFeHost) {
+public List> listConnectionForRpc(UserIdentity userIdentity,
boolean isShowFullSql,
+boolean isShowFeHost) {
List> list = new ArrayList<>();
long nowMs = System.currentTimeMillis();
for (ConnectContext ctx : connectionMap.values()) {
+// Check auth
+if (!ctx.getCurrentUserIdentity().equals(userIdentity) &&
!Env.getCurrentEnv()
+.getAccessManager()
+.checkGlobalPriv(userIdentity, PrivPredicate.GRANT)) {
+continue;
+}
list.add(ctx.toThreadInfo(isShowFullSql).toRow(-1, nowMs,
isShowFeHost));
}
return list;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
index 40445cc9a21..69c92d0f166 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
@@ -482,6 +482,7 @@ public class ShowExecutor {
try {
TShowProcessListRequest request = new
TShowProcessListRequest();
request.setShowFullSql(isShowFullSql);
+
request.setCurrentUserIdent(ConnectContext.get().getCurrentUserIdentity().toThrift());
List> frontends =
FrontendsProcNode.getFrontendWithRpcPort(Env.getCurrentEnv(),
false);
FrontendService.Client client = null;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
b/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
index cbbd49f69a8..7f81104d943 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
@@ -3316,8 +3316,12 @@ public class FrontendServiceImpl implements
FrontendService.Iface {
if (request.isSetShowFullSql()) {
isShowFullSql = request.isShowFullSql();
}
+UserIdentity userIdentity = UserIdentity.ROOT;
+if (request.isSetCurrentUserIdent()) {
+userIdentity =
UserIdentity.fromThrift(request.getCurrentUserIdent());
+}
List> processList =
ExecuteEnv.getInstance().getScheduler()
-.listConnectionWithoutAuth(isShowFullSql, true);
+.listConnectionForRpc(userIdentity, isShowFullSql, true);
TShowProcessListResult result = new TShowProcessListResult();
result.setProcessList(processList);
return result;
diff --git a/gensrc/thrift/FrontendService.thrift
b/gensrc/thrift/FrontendService.thrift
index a67bb59b80e..69aa37964dc 100644
--- a/gensrc/thrift/FrontendService.thrift
+++ b/gensrc/thrift/FrontendService.thrift
@@ -1254,6 +1254,7 @@ struct TGetBackendMetaResult {
struct TShowProcessListRequest {
1: optional bool show_full_sql
+2: optional Types.TUserIdentity current_user_ident
}
struct TShowProcessListResult {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [fix](test) table name in nereids_rules_p0/adjust_nullable/set_operation.groovy is wrong (#40536)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 40ac550f1c9 [fix](test) table name in
nereids_rules_p0/adjust_nullable/set_operation.groovy is wrong (#40536)
40ac550f1c9 is described below
commit 40ac550f1c98f612123ce54393e5a92fbbe64aac
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Mon Sep 9 15:42:14 2024 +0800
[fix](test) table name in
nereids_rules_p0/adjust_nullable/set_operation.groovy is wrong (#40536)
---
.../suites/nereids_rules_p0/adjust_nullable/set_operation.groovy| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/regression-test/suites/nereids_rules_p0/adjust_nullable/set_operation.groovy
b/regression-test/suites/nereids_rules_p0/adjust_nullable/set_operation.groovy
index 0d3017b939c..2bbb717716d 100644
---
a/regression-test/suites/nereids_rules_p0/adjust_nullable/set_operation.groovy
+++
b/regression-test/suites/nereids_rules_p0/adjust_nullable/set_operation.groovy
@@ -41,7 +41,7 @@ suite("test_set_operation_adjust_nullable") {
"""
sql """
-SELECT c1, c1 FROM t1 EXCEPT SELECT c2, c2 FROM t2;
+SELECT c1, c1 FROM set_operation_t1 EXCEPT SELECT c2, c2 FROM
set_operation_t2;
"""
// do not use regulator child output nullable as init nullable info
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [opt](Nereids) fix several insert into related issues (#40467)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 5dd08b6b460 [opt](Nereids) fix several insert into related issues
(#40467)
5dd08b6b460 is described below
commit 5dd08b6b4603ff8ef3c8a76d6a160fbce79fc953
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Mon Sep 9 12:19:56 2024 +0800
[opt](Nereids) fix several insert into related issues (#40467)
- http_stream TVF should always generate one fragment plan
- http_stream TVF plan should not check root as scan node
- distinguish group_commit TVF with normal insert statement
- index and generate slot should based on type cast base slot
- agg_state could cast from nullable to non-nullable
- colocated and bucket scan range compute should only on scan node
---
.../main/java/org/apache/doris/catalog/Type.java | 4 -
.../doris/nereids/parser/LogicalPlanBuilder.java | 2 +-
.../doris/nereids/rules/analysis/BindSink.java | 81 +++-
.../expressions/functions/table/HttpStream.java| 11 +++
.../trees/plans/commands/info/DMLCommandType.java | 2 +
.../main/java/org/apache/doris/qe/Coordinator.java | 5 +-
.../java/org/apache/doris/qe/StmtExecutor.java | 17 ++--
.../insert_into_table/insert_use_table_id.out | 48 -
.../agg_state/max/test_agg_state_max.groovy| 2 +-
.../insert_group_commit_into_unique.groovy | 9 +-
.../insert_into_table/insert_use_table_id.groovy | 107 -
11 files changed, 90 insertions(+), 198 deletions(-)
diff --git a/fe/fe-common/src/main/java/org/apache/doris/catalog/Type.java
b/fe/fe-common/src/main/java/org/apache/doris/catalog/Type.java
index 1dcd062261b..7a8dda5aabe 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/catalog/Type.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/catalog/Type.java
@@ -866,10 +866,6 @@ public abstract class Type {
return false;
}
for (int i = 0; i < sourceAggState.getSubTypes().size(); i++) {
-// target subtype is not null but source subtype is nullable
-if (!targetAggState.getSubTypeNullables().get(i) &&
sourceAggState.getSubTypeNullables().get(i)) {
-return false;
-}
if (!canCastTo(sourceAggState.getSubTypes().get(i),
targetAggState.getSubTypes().get(i))) {
return false;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
index 324ab808226..67ef1ca48c2 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
@@ -593,7 +593,7 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor {
isAutoDetect,
isOverwrite,
ConnectContext.get().getSessionVariable().isEnableUniqueKeyPartialUpdate(),
-DMLCommandType.INSERT,
+ctx.tableId == null ? DMLCommandType.INSERT :
DMLCommandType.GROUP_COMMIT,
plan);
Optional cte = Optional.empty();
if (ctx.cte() != null) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java
index c9e7f07f5d0..6d8ad94242b 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java
@@ -68,6 +68,7 @@ import
org.apache.doris.nereids.trees.plans.visitor.InferPlanOutputAlias;
import org.apache.doris.nereids.types.DataType;
import org.apache.doris.nereids.types.StringType;
import org.apache.doris.nereids.types.coercion.CharacterType;
+import org.apache.doris.nereids.util.ExpressionUtils;
import org.apache.doris.nereids.util.RelationUtil;
import org.apache.doris.nereids.util.TypeCoercionUtils;
@@ -126,7 +127,8 @@ public class BindSink implements AnalysisRuleFactory {
&& table.getSequenceMapCol() != null
&& sink.getColNames().contains(table.getSequenceMapCol());
Pair, Integer> bindColumnsResult =
-bindTargetColumns(table, sink.getColNames(), childHasSeqCol,
needExtraSeqCol);
+bindTargetColumns(table, sink.getColNames(), childHasSeqCol,
needExtraSeqCol,
+sink.getDMLCommandType() ==
DMLCommandType.GROUP_COMMIT);
List bindColumns = bindColumnsResult.first;
int extraColumnsNum = bindColumnsResult.second;
(doris) branch master updated: [enhance](mtmv) During cache generation, no longer hold the write lock for mtmv (#40402)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e23696cd000 [enhance](mtmv) During cache generation, no longer hold
the write lock for mtmv (#40402)
e23696cd000 is described below
commit e23696cd000b4e8834ee805d3520fe6af5db96b6
Author: zhangdong <493738...@qq.com>
AuthorDate: Fri Sep 6 11:34:23 2024 +0800
[enhance](mtmv) During cache generation, no longer hold the write lock for
mtmv (#40402)
During cache generation, no longer hold the write Lock for mv to avoid
changes in the logic of cache generation in the future, internal calls
to other locks, and deadlocks
---
.../main/java/org/apache/doris/catalog/MTMV.java | 47 ++
1 file changed, 31 insertions(+), 16 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
index c3d36ea3971..3e60a489c93 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
@@ -176,6 +176,19 @@ public class MTMV extends OlapTable {
public void addTaskResult(MTMVTask task, MTMVRelation relation,
Map partitionSnapshots) {
+MTMVCache mtmvCache = null;
+boolean needUpdateCache = false;
+if (task.getStatus() == TaskStatus.SUCCESS && !Env.isCheckpointThread()
+&& !Config.enable_check_compatibility_mode) {
+needUpdateCache = true;
+try {
+// shouldn't do this while holding mvWriteLock
+mtmvCache = MTMVCache.from(this,
MTMVPlanUtil.createMTMVContext(this), true);
+} catch (Throwable e) {
+mtmvCache = null;
+LOG.warn("generate cache failed", e);
+}
+}
writeMvLock();
try {
if (task.getStatus() == TaskStatus.SUCCESS) {
@@ -183,13 +196,8 @@ public class MTMV extends OlapTable {
this.status.setSchemaChangeDetail(null);
this.status.setRefreshState(MTMVRefreshState.SUCCESS);
this.relation = relation;
-if (!Env.isCheckpointThread() &&
!Config.enable_check_compatibility_mode) {
-try {
-this.cache = MTMVCache.from(this,
MTMVPlanUtil.createMTMVContext(this), true);
-} catch (Throwable e) {
-this.cache = null;
-LOG.warn("generate cache failed", e);
-}
+if (needUpdateCache) {
+this.cache = mtmvCache;
}
} else {
this.status.setRefreshState(MTMVRefreshState.FAIL);
@@ -268,17 +276,24 @@ public class MTMV extends OlapTable {
* Called when in query, Should use one connection context in query
*/
public MTMVCache getOrGenerateCache(ConnectContext connectionContext)
throws AnalysisException {
-if (cache == null) {
-writeMvLock();
-try {
-if (cache == null) {
-this.cache = MTMVCache.from(this, connectionContext, true);
-}
-} finally {
-writeMvUnlock();
+readMvLock();
+try {
+if (cache != null) {
+return cache;
}
+} finally {
+readMvUnlock();
+}
+// Concurrent situations may result in duplicate cache generation,
+// but we tolerate this in order to prevent nested use of readLock and
write MvLock for the table
+MTMVCache mtmvCache = MTMVCache.from(this, connectionContext, true);
+writeMvLock();
+try {
+this.cache = mtmvCache;
+return cache;
+} finally {
+writeMvUnlock();
}
-return cache;
}
public MTMVCache getCache() {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [opt](Nereids) support search from override udfs with same arity (#40432)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 82c2650322d [opt](Nereids) support search from override udfs with same
arity (#40432)
82c2650322d is described below
commit 82c2650322d57ed0028655d39f551649d59faaa1
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Thu Sep 5 20:21:19 2024 +0800
[opt](Nereids) support search from override udfs with same arity (#40432)
create alias function f1(int) with parameter(id) as abs(id);
create alias function f1(string) with parameter(id) as substr(id, 2);
select f1('1'); -- bind on f1(string)
select f1(1); -- bind on f1(int)
test case already existed in P0
---
.../org/apache/doris/catalog/FunctionRegistry.java | 92 --
.../apache/doris/catalog/FunctionSignature.java| 34
.../trees/expressions/functions/udf/AliasUdf.java | 4 +-
.../expressions/functions/udf/AliasUdfBuilder.java | 20 ++---
.../expressions/functions/udf/JavaUdafBuilder.java | 6 ++
.../expressions/functions/udf/JavaUdfBuilder.java | 6 ++
.../expressions/functions/udf/JavaUdtfBuilder.java | 6 ++
.../expressions/functions/udf/UdfBuilder.java | 3 +
8 files changed, 130 insertions(+), 41 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionRegistry.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionRegistry.java
index 0e049b4b902..7280463b0f2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionRegistry.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionRegistry.java
@@ -21,7 +21,9 @@ import org.apache.doris.datasource.InternalCatalog;
import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.nereids.annotation.Developing;
import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.Expression;
import
org.apache.doris.nereids.trees.expressions.functions.AggCombinerFunctionBuilder;
+import
org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
import org.apache.doris.nereids.trees.expressions.functions.FunctionBuilder;
import org.apache.doris.nereids.trees.expressions.functions.udf.UdfBuilder;
import org.apache.doris.nereids.types.DataType;
@@ -156,18 +158,33 @@ public class FunctionRegistry {
+ "' which has " + arity + " arity. Candidate functions
are: " + candidateHints);
}
if (candidateBuilders.size() > 1) {
-String candidateHints = getCandidateHint(name, candidateBuilders);
-// TODO: NereidsPlanner not supported override function by the
same arity, we will support it later
-if (ConnectContext.get() != null) {
-try {
-
ConnectContext.get().getSessionVariable().enableFallbackToOriginalPlannerOnce();
-} catch (Throwable t) {
-// ignore error
+boolean needChooseOne = true;
+List signatures =
Lists.newArrayListWithCapacity(candidateBuilders.size());
+for (FunctionBuilder functionBuilder : candidateBuilders) {
+if (functionBuilder instanceof UdfBuilder) {
+signatures.addAll(((UdfBuilder)
functionBuilder).getSignatures());
+} else {
+needChooseOne = false;
+break;
+}
+}
+for (Object argument : arguments) {
+if (!(argument instanceof Expression)) {
+needChooseOne = false;
+break;
}
}
+if (needChooseOne) {
+FunctionSignature signature = new
UdfSignatureSearcher(signatures, (List) arguments).getSignature();
+for (int i = 0; i < signatures.size(); i++) {
+if (signatures.get(i).equals(signature)) {
+return candidateBuilders.get(i);
+}
+}
+}
+String candidateHints = getCandidateHint(name, candidateBuilders);
throw new AnalysisException("Function '" + qualifiedName + "' is
ambiguous: " + candidateHints);
}
-
return candidateBuilders.get(0);
}
@@ -235,4 +252,63 @@ public class FunctionRegistry {
.removeIf(builder -> ((UdfBuilder)
builder).getArgTypes().equals(argTypes));
}
}
+
+/**
+ * use for search appropriate signature for UDFs if candidate more than
one.
+ */
+static class UdfSignatureSearcher implements ExplicitlyCastableSignature {
+
+private final List signatures;
+private fin
(doris) branch master updated (518a0fc018b -> f830527af68)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 518a0fc018b [opt](mtmv) Support to contain select constant clause when
create async materialized view (#40244)
add f830527af68 [Feat](nereids) support pull up predicate from set
operator (#39450)
No new revisions were added by this update.
Summary of changes:
.../doris/nereids/jobs/executor/Rewriter.java | 31 +-
.../org/apache/doris/nereids/rules/RuleSet.java|9 +-
.../rules/rewrite/EliminateEmptyRelation.java |4 +
.../nereids/rules/rewrite/InferPredicates.java | 45 +
.../nereids/rules/rewrite/PullUpPredicates.java| 139 +++
.../rules/rewrite/PushProjectIntoUnion.java| 13 +-
.../apache/doris/nereids/util/ExpressionUtils.java | 29 +
.../nereids/rules/rewrite/InferPredicatesTest.java | 131 +-
.../data/nereids_hint_tpcds_p0/shape/query8.out|9 +-
.../push_down_expression_in_hash_join.out |4 +-
.../filter_push_down/push_filter_through.out |6 +-
.../infer_predicate/infer_intersect_except.out | 155 +++
.../infer_predicate/pull_up_predicate_literal.out | 1258 ++--
.../infer_predicate/pull_up_predicate_set_op.out | 563 +
.../infer_set_operator_distinct.out| 27 +-
.../bs_downgrade_shape/query8.out |9 +-
.../nereids_tpcds_shape_sf1000_p0/shape/query8.out |9 +-
.../noStatsRfPrune/query8.out |9 +-
.../no_stats_shape/query8.out |9 +-
.../rf_prune/query8.out|9 +-
.../nereids_tpcds_shape_sf100_p0/shape/query8.out |9 +-
.../tpcds_sf100/noStatsRfPrune/query8.out |9 +-
.../tpcds_sf100/no_stats_shape/query8.out |9 +-
.../new_shapes_p0/tpcds_sf100/rf_prune/query8.out |9 +-
.../new_shapes_p0/tpcds_sf100/shape/query8.out |9 +-
.../tpcds_sf1000/bs_downgrade_shape/query8.out |9 +-
.../new_shapes_p0/tpcds_sf1000/shape/query8.out|9 +-
.../txn_insert_values_with_schema_change.out |0
.../adjust_nullable/set_operation.groovy |4 +-
.../infer_predicate/infer_intersect_except.groovy | 208
.../pull_up_predicate_literal.groovy |9 +-
.../pull_up_predicate_set_op.groovy| 469
.../infer_set_operator_distinct.groovy |1 -
33 files changed, 2240 insertions(+), 982 deletions(-)
create mode 100644
regression-test/data/nereids_rules_p0/infer_predicate/infer_intersect_except.out
create mode 100644
regression-test/data/nereids_rules_p0/infer_predicate/pull_up_predicate_set_op.out
copy regression-test/{data/insert_p0/transaction =>
regression-test/realData/insert_p0}/txn_insert_values_with_schema_change.out
(100%)
create mode 100644
regression-test/suites/nereids_rules_p0/infer_predicate/infer_intersect_except.groovy
create mode 100644
regression-test/suites/nereids_rules_p0/infer_predicate/pull_up_predicate_set_op.groovy
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [Fix](planner) legacy planner repeat group by has grouping report error (#40281) (#40306)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new b97e0438a20 [Fix](planner) legacy planner repeat group by has grouping
report error (#40281) (#40306)
b97e0438a20 is described below
commit b97e0438a20cd0fb5bb1bef6392407c9ef43fadc
Author: feiniaofeiafei <53502832+feiniaofeia...@users.noreply.github.com>
AuthorDate: Wed Sep 4 12:04:51 2024 +0800
[Fix](planner) legacy planner repeat group by has grouping report error
(#40281) (#40306)
cherry-pick from master #40281
---
.../org/apache/doris/analysis/GroupByClause.java | 5 +++
.../grouping_sets/valid_grouping.groovy| 51 ++
2 files changed, 56 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/GroupByClause.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/GroupByClause.java
index f6305e611da..9b505594467 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/GroupByClause.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/GroupByClause.java
@@ -192,6 +192,11 @@ public class GroupByClause implements ParseNode {
"GROUP BY expression must not contain aggregate
functions: "
+ groupingExpr.toSql());
}
+if (groupingExpr.contains(GroupingFunctionCallExpr.class)) {
+throw new AnalysisException(
+"GROUP BY expression must not contain grouping scalar
functions: "
++ groupingExpr.toSql());
+}
if (groupingExpr.contains(AnalyticExpr.class)) {
// reference the original expr in the error msg
throw new AnalysisException(
diff --git
a/regression-test/suites/nereids_rules_p0/grouping_sets/valid_grouping.groovy
b/regression-test/suites/nereids_rules_p0/grouping_sets/valid_grouping.groovy
new file mode 100644
index 000..dd69f23f027
--- /dev/null
+++
b/regression-test/suites/nereids_rules_p0/grouping_sets/valid_grouping.groovy
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("valid_grouping"){
+sql "SET enable_fallback_to_original_planner=true"
+sql "drop table if exists valid_grouping"
+sql """
+CREATE TABLE `valid_grouping` (
+ `a` INT NULL,
+ `b` VARCHAR(10) NULL,
+ `c` INT NULL,
+ `d` INT NULL
+) ENGINE=OLAP
+DUPLICATE KEY(`a`, `b`)
+DISTRIBUTED BY RANDOM BUCKETS AUTO
+PROPERTIES (
+"replication_allocation" = "tag.location.default: 1"
+);
+"""
+sql "insert into valid_grouping values(1,'d2',3,5);"
+test {
+sql """select
+b, 'day' as DT_TYPE
+from valid_grouping
+group by grouping sets ( (grouping_id(b)),(b));"""
+exception("GROUP BY expression must not contain grouping scalar
functions: grouping_id(`b`)")
+}
+
+test {
+sql """select
+b, 'day' as DT_TYPE
+from valid_grouping
+group by grouping sets ( (grouping(b)),(b));"""
+exception("GROUP BY expression must not contain grouping scalar
functions: grouping(`b`)")
+}
+
+}
\ No newline at end of file
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (9a92b4c3809 -> bfbb065d0e0)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 9a92b4c3809 [opt](function)Some geo functions incorrectly used get. (#40107) add bfbb065d0e0 [Feat](nereids) add max/min filter push down rewrite rule (#39252) No new revisions were added by this update. Summary of changes: .../org/apache/doris/nereids/rules/RuleSet.java| 2 + .../org/apache/doris/nereids/rules/RuleType.java | 1 + .../rules/rewrite/MaxMinFilterPushDown.java| 133 ++ .../rules/rewrite/MaxMinFilterPushDownTest.java| 115 .../eager_aggregate/push_down_max_through_join.out | 20 +- .../max_min_filter_push_down.out | 290 + .../max_min_filter_push_down.groovy| 214 +++ 7 files changed, 765 insertions(+), 10 deletions(-) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MaxMinFilterPushDown.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/MaxMinFilterPushDownTest.java create mode 100644 regression-test/data/nereids_rules_p0/max_min_filter_push_down/max_min_filter_push_down.out create mode 100644 regression-test/suites/nereids_rules_p0/max_min_filter_push_down/max_min_filter_push_down.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch r_cte created (now 140081381c)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch r_cte in repository https://gitbox.apache.org/repos/asf/doris-website.git at 140081381c [fix](cte) doris do not support recursive CTE, remove it in docs This branch includes the following new commits: new 140081381c [fix](cte) doris do not support recursive CTE, remove it in docs The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [opt](Nereids) add show statement syntax (#40288)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new cdfe3c24cd3 [opt](Nereids) add show statement syntax (#40288) cdfe3c24cd3 is described below commit cdfe3c24cd32c2569f8cdaa10fabf4558c7d4759 Author: morrySnow <101034200+morrys...@users.noreply.github.com> AuthorDate: Tue Sep 3 18:27:27 2024 +0800 [opt](Nereids) add show statement syntax (#40288) --- .../antlr4/org/apache/doris/nereids/DorisLexer.g4 | 1 + .../antlr4/org/apache/doris/nereids/DorisParser.g4 | 136 - fe/fe-core/src/main/cup/sql_parser.cup | 4 +- .../org/apache/doris/analysis/AlterRoleStmt.java | 2 +- .../org/apache/doris/analysis/AnalyzeStmt.java | 2 +- .../org/apache/doris/analysis/CreateRoleStmt.java | 2 +- .../java/org/apache/doris/analysis/DdlStmt.java| 2 +- .../apache/doris/analysis/DiagnoseTabletStmt.java | 2 +- .../org/apache/doris/analysis/DropUserStmt.java| 2 +- .../java/org/apache/doris/analysis/ExportStmt.java | 2 +- .../doris/analysis/InsertOverwriteTableStmt.java | 2 +- .../java/org/apache/doris/analysis/SelectStmt.java | 2 +- .../doris/analysis/SetDefaultStorageVaultStmt.java | 2 +- .../apache/doris/analysis/SetOperationStmt.java| 2 +- .../apache/doris/analysis/SetUserPropertyStmt.java | 2 +- .../org/apache/doris/analysis/ShowAlterStmt.java | 2 +- .../org/apache/doris/analysis/ShowAnalyzeStmt.java | 2 +- .../doris/analysis/ShowAnalyzeTaskStatus.java | 2 +- .../org/apache/doris/analysis/ShowAuthorStmt.java | 2 +- .../doris/analysis/ShowAutoAnalyzeJobsStmt.java| 2 +- .../apache/doris/analysis/ShowBackendsStmt.java| 2 +- .../org/apache/doris/analysis/ShowBackupStmt.java | 2 +- .../org/apache/doris/analysis/ShowBrokerStmt.java | 2 +- .../apache/doris/analysis/ShowBuildIndexStmt.java | 2 +- .../doris/analysis/ShowCacheHotSpotStmt.java | 2 +- .../doris/analysis/ShowCatalogRecycleBinStmt.java | 2 +- .../org/apache/doris/analysis/ShowCatalogStmt.java | 2 +- .../org/apache/doris/analysis/ShowCharsetStmt.java | 2 +- .../apache/doris/analysis/ShowCloudWarmUpStmt.java | 2 +- .../org/apache/doris/analysis/ShowClusterStmt.java | 2 +- .../apache/doris/analysis/ShowCollationStmt.java | 2 +- .../apache/doris/analysis/ShowColumnHistStmt.java | 2 +- .../apache/doris/analysis/ShowColumnStatsStmt.java | 2 +- .../org/apache/doris/analysis/ShowColumnStmt.java | 2 +- .../apache/doris/analysis/ShowConvertLSCStmt.java | 2 +- .../org/apache/doris/analysis/ShowCopyStmt.java| 2 +- .../doris/analysis/ShowCreateCatalogStmt.java | 2 +- .../apache/doris/analysis/ShowCreateDbStmt.java| 2 +- .../doris/analysis/ShowCreateFunctionStmt.java | 2 +- .../apache/doris/analysis/ShowCreateMTMVStmt.java | 2 +- .../analysis/ShowCreateMaterializedViewStmt.java | 2 +- .../doris/analysis/ShowCreateRepositoryStmt.java | 2 +- .../apache/doris/analysis/ShowCreateTableStmt.java | 2 +- .../apache/doris/analysis/ShowDataSkewStmt.java| 2 +- .../org/apache/doris/analysis/ShowDataStmt.java| 2 +- .../apache/doris/analysis/ShowDataTypesStmt.java | 2 +- .../org/apache/doris/analysis/ShowDbIdStmt.java| 2 +- .../java/org/apache/doris/analysis/ShowDbStmt.java | 2 +- .../org/apache/doris/analysis/ShowDeleteStmt.java | 2 +- .../doris/analysis/ShowDynamicPartitionStmt.java | 2 +- .../apache/doris/analysis/ShowEncryptKeysStmt.java | 2 +- .../org/apache/doris/analysis/ShowEnginesStmt.java | 2 +- .../org/apache/doris/analysis/ShowEventsStmt.java | 2 +- .../org/apache/doris/analysis/ShowExportStmt.java | 2 +- .../apache/doris/analysis/ShowFrontendsStmt.java | 2 +- .../apache/doris/analysis/ShowFunctionsStmt.java | 2 +- .../org/apache/doris/analysis/ShowGrantsStmt.java | 2 +- .../org/apache/doris/analysis/ShowIndexStmt.java | 2 +- .../apache/doris/analysis/ShowLastInsertStmt.java | 2 +- .../apache/doris/analysis/ShowLoadProfileStmt.java | 2 +- .../org/apache/doris/analysis/ShowLoadStmt.java| 2 +- .../doris/analysis/ShowLoadWarningsStmt.java | 2 +- .../apache/doris/analysis/ShowOpenTableStmt.java | 2 +- .../apache/doris/analysis/ShowPartitionIdStmt.java | 2 +- .../apache/doris/analysis/ShowPartitionsStmt.java | 2 +- .../org/apache/doris/analysis/ShowPluginsStmt.java | 2 +- .../org/apache/doris/analysis/ShowPolicyStmt.java | 2 +- .../apache/doris/analysis/ShowPrivilegesStmt.java | 2 +- .../org/apache/doris/analysis/ShowProcStmt.java| 2 +- .../apache/doris/analysis/ShowProcedureStmt.java | 2 +- .../apache/doris/analysis/ShowProcesslistStmt.java | 2 +- .../doris/analysis/ShowQueryProfileStmt.java | 2 +- .../apache/doris/an
(doris) branch master updated (a4956d6d378 -> b7e6d7350bc)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from a4956d6d378 [feature](metacache) add system table catalog_meta_cache_statistics (#40155) add b7e6d7350bc [fix](testcase) fix unstable case (#40124) No new revisions were added by this update. Summary of changes: regression-test/suites/insert_p0/insert_group_commit_into.groovy | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [enhance](mtmv)Optimize the speed of obtaining the last update time of Hive (#40169)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 5ea6e4bc035 [enhance](mtmv)Optimize the speed of obtaining the last
update time of Hive (#40169)
5ea6e4bc035 is described below
commit 5ea6e4bc03575444688dc4672d099354ba6b17e6
Author: zhangdong <493738...@qq.com>
AuthorDate: Tue Sep 3 11:43:49 2024 +0800
[enhance](mtmv)Optimize the speed of obtaining the last update time of Hive
(#40169)
Previously, to obtain the last update time of a hive table, it was
necessary to obtain the last update time of all partitions under the
table, which required generating a large map.
---
.../doris/datasource/hive/HMSExternalTable.java| 79 --
.../doris/datasource/hive/HiveMetaStoreCache.java | 4 ++
2 files changed, 46 insertions(+), 37 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
index a9f2da13b40..8217f1c3a36 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
@@ -53,6 +53,7 @@ import com.google.common.collect.BiMap;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
import com.google.common.collect.Sets;
+import org.apache.commons.collections.CollectionUtils;
import org.apache.commons.collections.MapUtils;
import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.hive.metastore.api.ColumnStatisticsData;
@@ -741,18 +742,17 @@ public class HMSExternalTable extends ExternalTable
implements MTMVRelatedTableI
return res;
}
-private HiveMetaStoreCache.HivePartitionValues getHivePartitionValues() {
-HiveMetaStoreCache cache = Env.getCurrentEnv().getExtMetaCacheMgr()
-.getMetaStoreCache((HMSExternalCatalog) getCatalog());
-return cache.getPartitionValues(
-getDbName(), getName(), getPartitionColumnTypes());
-}
-
@Override
public MTMVSnapshotIf getPartitionSnapshot(String partitionName,
MTMVRefreshContext context)
throws AnalysisException {
-long partitionLastModifyTime =
getPartitionLastModifyTime(partitionName);
-return new MTMVTimestampSnapshot(partitionLastModifyTime);
+HiveMetaStoreCache cache = Env.getCurrentEnv().getExtMetaCacheMgr()
+.getMetaStoreCache((HMSExternalCatalog) getCatalog());
+HiveMetaStoreCache.HivePartitionValues hivePartitionValues =
cache.getPartitionValues(
+getDbName(), getName(), getPartitionColumnTypes());
+Long partitionId =
getPartitionIdByNameOrAnalysisException(partitionName, hivePartitionValues);
+HivePartition hivePartition =
getHivePartitionByIdOrAnalysisException(partitionId,
+hivePartitionValues, cache);
+return new MTMVTimestampSnapshot(hivePartition.getLastModifiedTime());
}
@Override
@@ -760,45 +760,50 @@ public class HMSExternalTable extends ExternalTable
implements MTMVRelatedTableI
if (getPartitionType() == PartitionType.UNPARTITIONED) {
return new MTMVMaxTimestampSnapshot(getName(), getLastDdlTime());
}
-String partitionName = "";
+Long maxPartitionId = 0L;
long maxVersionTime = 0L;
long visibleVersionTime;
-for (Entry entry :
getAndCopyPartitionItems().entrySet()) {
-visibleVersionTime = getPartitionLastModifyTime(entry.getKey());
+HiveMetaStoreCache cache = Env.getCurrentEnv().getExtMetaCacheMgr()
+.getMetaStoreCache((HMSExternalCatalog) getCatalog());
+HiveMetaStoreCache.HivePartitionValues hivePartitionValues =
cache.getPartitionValues(
+getDbName(), getName(), getPartitionColumnTypes());
+BiMap idToName =
hivePartitionValues.getPartitionNameToIdMap().inverse();
+if (MapUtils.isEmpty(idToName)) {
+throw new AnalysisException("partitions is empty for : " +
getName());
+}
+for (Long partitionId : idToName.keySet()) {
+visibleVersionTime =
getHivePartitionByIdOrAnalysisException(partitionId, hivePartitionValues,
+cache).getLastModifiedTime();
if (visibleVersionTime > maxVersionTime) {
maxVersionTime = visibleVersionTime;
-partitionName = entry.getKey();
+maxPartitionId = partitionId;
}
}
-return new MTMVMaxTimestampSnapshot(partitionName, maxVersionTime);
+return new MTMVMaxTimestampSnapshot(idToName.get(maxPartitionId),
maxVersionTime);
}
-private long getP
(doris) branch master updated (ee3be4fa769 -> a1ea5c467ab)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from ee3be4fa769 [fix](index compaction)Fix core when having multiple dest
segments (#40254)
add a1ea5c467ab [Fix](planner) legacy planner repeat group by has grouping
report error (#40281)
No new revisions were added by this update.
Summary of changes:
.../org/apache/doris/analysis/GroupByClause.java | 5 +++
.../grouping_sets/valid_grouping.groovy} | 47 --
2 files changed, 31 insertions(+), 21 deletions(-)
copy regression-test/suites/{datatype_p0/ip/test_ip_implicit_cast.groovy =>
nereids_rules_p0/grouping_sets/valid_grouping.groovy} (51%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [opt](Nereids) forbid one step topn with distribute as child (#40066) (#40277)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new dbd20f23ae1 [opt](Nereids) forbid one step topn with distribute as
child (#40066) (#40277)
dbd20f23ae1 is described below
commit dbd20f23ae1d731ebee096b072cf46d40f616cbc
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Mon Sep 2 21:46:38 2024 +0800
[opt](Nereids) forbid one step topn with distribute as child (#40066)
(#40277)
pick from master #40066
---
.../properties/ChildrenPropertiesRegulator.java | 19 +++
1 file changed, 19 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildrenPropertiesRegulator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildrenPropertiesRegulator.java
index d17d7485cce..a435d09eb80 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildrenPropertiesRegulator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildrenPropertiesRegulator.java
@@ -43,6 +43,7 @@ import
org.apache.doris.nereids.trees.plans.physical.PhysicalNestedLoopJoin;
import org.apache.doris.nereids.trees.plans.physical.PhysicalOlapScan;
import org.apache.doris.nereids.trees.plans.physical.PhysicalProject;
import org.apache.doris.nereids.trees.plans.physical.PhysicalSetOperation;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalTopN;
import org.apache.doris.nereids.trees.plans.physical.PhysicalUnion;
import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
import org.apache.doris.nereids.util.JoinUtils;
@@ -468,6 +469,24 @@ public class ChildrenPropertiesRegulator extends
PlanVisitor {
return true;
}
+@Override
+public Boolean visitPhysicalTopN(PhysicalTopN topN, Void
context) {
+// process must shuffle
+visit(topN, context);
+
+// If child is DistributionSpecGather, topN should forbid two-phase
topN
+if (topN.getSortPhase() == SortPhase.LOCAL_SORT
+&&
childrenProperties.get(0).getDistributionSpec().equals(DistributionSpecGather.INSTANCE))
{
+return false;
+}
+// forbid one step topn with distribute as child
+if (topN.getSortPhase() == SortPhase.GATHER_SORT
+&& children.get(0).getPlan() instanceof PhysicalDistribute) {
+return false;
+}
+return true;
+}
+
/**
* check both side real output hash key order are same or not.
*
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [fix](Nereids) datetimev2 literal equals should compare microsecond (#40121) (#40279)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 6c34de6adf3 [fix](Nereids) datetimev2 literal equals should compare
microsecond (#40121) (#40279)
6c34de6adf3 is described below
commit 6c34de6adf3f1633fa6f7492735331761b78239b
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Mon Sep 2 21:45:49 2024 +0800
[fix](Nereids) datetimev2 literal equals should compare microsecond
(#40121) (#40279)
pick from master #40121
---
.../nereids/trees/expressions/literal/DateTimeV2Literal.java | 7 ++-
.../nereids/trees/expressions/literal/DateTimeLiteralTest.java | 10 ++
2 files changed, 16 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeV2Literal.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeV2Literal.java
index 784167c81ca..fea92fd86ec 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeV2Literal.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeV2Literal.java
@@ -99,6 +99,11 @@ public class DateTimeV2Literal extends DateTimeLiteral {
getDataType().toCatalogDataType());
}
+@Override
+public double getDouble() {
+return super.getDouble() + microSecond / 100.0;
+}
+
@Override
public String toString() {
return getStringValue();
@@ -233,6 +238,6 @@ public class DateTimeV2Literal extends DateTimeLiteral {
return false;
}
DateTimeV2Literal literal = (DateTimeV2Literal) o;
-return Objects.equals(dataType, literal.dataType);
+return Objects.equals(dataType, literal.dataType) &&
Objects.equals(microSecond, literal.microSecond);
}
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteralTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteralTest.java
index a7553ef658b..e7606cca352 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteralTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteralTest.java
@@ -485,4 +485,14 @@ class DateTimeLiteralTest {
Assertions.assertEquals(1, literal.roundCeiling(0).month);
Assertions.assertEquals(2001, literal.roundCeiling(0).year);
}
+
+@Test
+void testEquals() {
+DateTimeV2Literal l1 = new DateTimeV2Literal(1, 1, 1, 1, 1, 1, 1);
+DateTimeV2Literal l2 = new DateTimeV2Literal(1, 1, 1, 1, 1, 1, 1);
+DateTimeV2Literal l3 = new DateTimeV2Literal(1, 1, 1, 1, 1, 1, 2);
+
+Assertions.assertEquals(l1, l2);
+Assertions.assertNotEquals(l1, l3);
+}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch 2.1_40066 updated (b0e4a7a7fe1 -> cf55d28ab2e)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch 2.1_40066 in repository https://gitbox.apache.org/repos/asf/doris.git omit b0e4a7a7fe1 [opt](Nereids) forbid one step topn with distribute as child (#40066) add cf55d28ab2e [opt](Nereids) forbid one step topn with distribute as child (#40066) This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (b0e4a7a7fe1) \ N -- N -- N refs/heads/2.1_40066 (cf55d28ab2e) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: fe/.idea/vcs.xml | 22 ++ 1 file changed, 18 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch 2.1_40066 updated (01d246c39e0 -> b0e4a7a7fe1)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch 2.1_40066 in repository https://gitbox.apache.org/repos/asf/doris.git discard 01d246c39e0 [opt](Nereids) forbid one step topn with distribute as child (#40066) new b0e4a7a7fe1 [opt](Nereids) forbid one step topn with distribute as child (#40066) This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (01d246c39e0) \ N -- N -- N refs/heads/2.1_40066 (b0e4a7a7fe1) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: fe/.idea/vcs.xml | 22 -- .../properties/ChildrenPropertiesRegulator.java| 6 -- 2 files changed, 4 insertions(+), 24 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 01/01: [opt](Nereids) forbid one step topn with distribute as child (#40066)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch 2.1_40066
in repository https://gitbox.apache.org/repos/asf/doris.git
commit b0e4a7a7fe1fe574574f3f69cf974c459ccd2843
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Thu Aug 29 21:31:20 2024 +0800
[opt](Nereids) forbid one step topn with distribute as child (#40066)
pick from master #40066
---
fe/.idea/vcs.xml | 22 --
.../properties/ChildrenPropertiesRegulator.java| 6 +-
2 files changed, 9 insertions(+), 19 deletions(-)
diff --git a/fe/.idea/vcs.xml b/fe/.idea/vcs.xml
index 7b2cdb1cbbd..8c0f59e92e6 100644
--- a/fe/.idea/vcs.xml
+++ b/fe/.idea/vcs.xml
@@ -1,20 +1,4 @@
-
@@ -27,6 +11,8 @@
-
+
+
+
-
+
\ No newline at end of file
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildrenPropertiesRegulator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildrenPropertiesRegulator.java
index c83c0d3582e..5ba1bd87b05 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildrenPropertiesRegulator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildrenPropertiesRegulator.java
@@ -518,12 +518,16 @@ public class ChildrenPropertiesRegulator extends
PlanVisitor {
public Boolean visitPhysicalTopN(PhysicalTopN topN, Void
context) {
// process must shuffle
visit(topN, context);
-
// If child is DistributionSpecGather, topN should forbid two-phase
topN
if (topN.getSortPhase() == SortPhase.LOCAL_SORT
&&
childrenProperties.get(0).getDistributionSpec().equals(DistributionSpecGather.INSTANCE))
{
return false;
}
+// forbid one step topn with distribute as child
+if (topN.getSortPhase() == SortPhase.GATHER_SORT
+&& children.get(0).getPlan() instanceof PhysicalDistribute) {
+return false;
+}
return true;
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 01/01: [opt](Nereids) forbid one step topn with distribute as child (#40066)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch 2.1_40066
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 01d246c39e08cfce1f8433a3e2d74a9ef56dea49
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Thu Aug 29 21:31:20 2024 +0800
[opt](Nereids) forbid one step topn with distribute as child (#40066)
pick from master #40066
---
.../doris/nereids/properties/ChildrenPropertiesRegulator.java | 10 ++
1 file changed, 10 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildrenPropertiesRegulator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildrenPropertiesRegulator.java
index c83c0d3582e..f84e319f345 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildrenPropertiesRegulator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildrenPropertiesRegulator.java
@@ -519,11 +519,21 @@ public class ChildrenPropertiesRegulator extends
PlanVisitor {
// process must shuffle
visit(topN, context);
+int sortPhaseNum =
jobContext.getCascadesContext().getConnectContext().getSessionVariable().sortPhaseNum;
+// if control sort phase, forbid nothing
+if (sortPhaseNum == 1 || sortPhaseNum == 2) {
+return true;
+}
// If child is DistributionSpecGather, topN should forbid two-phase
topN
if (topN.getSortPhase() == SortPhase.LOCAL_SORT
&&
childrenProperties.get(0).getDistributionSpec().equals(DistributionSpecGather.INSTANCE))
{
return false;
}
+// forbid one step topn with distribute as child
+if (topN.getSortPhase() == SortPhase.GATHER_SORT
+&& children.get(0).getPlan() instanceof PhysicalDistribute) {
+return false;
+}
return true;
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch 2.1_40066 created (now 01d246c39e0)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch 2.1_40066 in repository https://gitbox.apache.org/repos/asf/doris.git at 01d246c39e0 [opt](Nereids) forbid one step topn with distribute as child (#40066) This branch includes the following new commits: new 01d246c39e0 [opt](Nereids) forbid one step topn with distribute as child (#40066) The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [fix](nereids) fix union all instance number (#39999) (#40100)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new a4d124e4d3f [fix](nereids) fix union all instance number (#3)
(#40100)
a4d124e4d3f is described below
commit a4d124e4d3f2552f06217650c4fddc824c3c6d54
Author: xzj7019 <13794+xzj7...@users.noreply.github.com>
AuthorDate: Mon Sep 2 18:11:56 2024 +0800
[fix](nereids) fix union all instance number (#3) (#40100)
pick from master #3
---
.../main/java/org/apache/doris/qe/Coordinator.java | 43 ++-
.../query_p0/union/test_union_instance.groovy | 88 ++
2 files changed, 111 insertions(+), 20 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
index 85f4201eb1e..f3b6dabbf46 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
@@ -1657,7 +1657,6 @@ public class Coordinator implements CoordInterface {
}
Pair pairNodes =
findLeftmostNode(fragment.getPlanRoot());
-PlanNode fatherNode = pairNodes.first;
PlanNode leftMostNode = pairNodes.second;
/*
@@ -1672,25 +1671,8 @@ public class Coordinator implements CoordInterface {
// (Case B)
// there is no leftmost scan; we assign the same hosts as
those of our
// input fragment which has a higher instance_number
-
-int inputFragmentIndex = 0;
-int maxParallelism = 0;
-// If the fragment has three children, then the first child
and the second child are
-// the children(both exchange node) of shuffle HashJoinNode,
-// and the third child is the right child(ExchangeNode) of
broadcast HashJoinNode.
-// We only need to pay attention to the maximum parallelism
among
-// the two ExchangeNodes of shuffle HashJoinNode.
-int childrenCount = (fatherNode != null) ?
fatherNode.getChildren().size() : 1;
-for (int j = 0; j < childrenCount; j++) {
-int currentChildFragmentParallelism
-=
fragmentExecParamsMap.get(fragment.getChild(j).getFragmentId()).instanceExecParams.size();
-if (currentChildFragmentParallelism > maxParallelism) {
-maxParallelism = currentChildFragmentParallelism;
-inputFragmentIndex = j;
-}
-}
-
-PlanFragmentId inputFragmentId =
fragment.getChild(inputFragmentIndex).getFragmentId();
+int maxParallelFragmentIndex =
findMaxParallelFragmentIndex(fragment);
+PlanFragmentId inputFragmentId =
fragment.getChild(maxParallelFragmentIndex).getFragmentId();
// AddAll() soft copy()
int exchangeInstances = -1;
if (ConnectContext.get() != null &&
ConnectContext.get().getSessionVariable() != null) {
@@ -1838,6 +1820,27 @@ public class Coordinator implements CoordInterface {
}
}
+private int findMaxParallelFragmentIndex(PlanFragment fragment) {
+Preconditions.checkState(!fragment.getChildren().isEmpty(), "fragment
has no children");
+
+// exclude broadcast join right side's child fragments
+List childFragmentCandidates =
fragment.getChildren().stream()
+.filter(e -> e.getOutputPartition() !=
DataPartition.UNPARTITIONED)
+.collect(Collectors.toList());
+
+int maxParallelism = 0;
+int maxParaIndex = 0;
+for (int i = 0; i < childFragmentCandidates.size(); i++) {
+PlanFragmentId childFragmentId =
childFragmentCandidates.get(i).getFragmentId();
+int currentChildFragmentParallelism =
fragmentExecParamsMap.get(childFragmentId).instanceExecParams.size();
+if (currentChildFragmentParallelism > maxParallelism) {
+maxParallelism = currentChildFragmentParallelism;
+maxParaIndex = i;
+}
+}
+return maxParaIndex;
+}
+
// Traverse the expected runtimeFilterID in each fragment, and establish
the corresponding relationship
// between runtimeFilterID and fragment instance addr and select the merge
instance of runtimeFilter
private void assignRuntimeFilterAddr() throws Exception {
diff --git a/regression-test/suites/query_p0/union/test_union_instance.groovy
b/regression-test/suites/query_p0/union/test_union_instance.groovy
new file mode 100644
index 000..6829a48989b
--- /dev/null
+++ b/regression-test/suites/query
(doris) branch master updated: [opt](Nereids) add job, analyze, load and cancel statements syntax (#40203)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 4c16768f7c5 [opt](Nereids) add job, analyze, load and cancel statements syntax (#40203) 4c16768f7c5 is described below commit 4c16768f7c5ed63f6b573c88264ad1a3aca01149 Author: morrySnow <101034200+morrys...@users.noreply.github.com> AuthorDate: Mon Sep 2 14:36:27 2024 +0800 [opt](Nereids) add job, analyze, load and cancel statements syntax (#40203) --- .../antlr4/org/apache/doris/nereids/DorisParser.g4 | 171 ++--- .../apache/doris/analysis/AlterJobStatusStmt.java | 2 +- .../org/apache/doris/analysis/AnalyzeDBStmt.java | 2 +- .../org/apache/doris/analysis/AnalyzeTblStmt.java | 2 +- .../doris/analysis/CancelAlterSystemStmt.java | 2 +- .../doris/analysis/CancelAlterTableStmt.java | 2 +- .../apache/doris/analysis/CancelBackupStmt.java| 2 +- .../doris/analysis/CancelCloudWarmUpStmt.java | 2 +- .../apache/doris/analysis/CancelExportStmt.java| 2 +- .../apache/doris/analysis/CancelJobTaskStmt.java | 2 +- .../org/apache/doris/analysis/CancelLoadStmt.java | 2 +- .../org/apache/doris/analysis/CleanLabelStmt.java | 2 +- .../apache/doris/analysis/CleanProfileStmt.java| 2 +- .../apache/doris/analysis/CleanQueryStatsStmt.java | 2 +- .../org/apache/doris/analysis/CreateJobStmt.java | 2 +- .../doris/analysis/CreateRoutineLoadStmt.java | 2 +- .../apache/doris/analysis/ImportColumnsStmt.java | 2 +- .../apache/doris/analysis/ImportDeleteOnStmt.java | 2 +- .../apache/doris/analysis/ImportSequenceStmt.java | 2 +- .../org/apache/doris/analysis/ImportWhereStmt.java | 2 +- .../apache/doris/analysis/KillAnalysisJobStmt.java | 2 +- .../java/org/apache/doris/analysis/LoadStmt.java | 2 +- .../doris/analysis/PauseRoutineLoadStmt.java | 2 +- .../apache/doris/analysis/PauseSyncJobStmt.java| 2 +- .../apache/doris/analysis/RefreshCatalogStmt.java | 2 +- .../org/apache/doris/analysis/RefreshDbStmt.java | 2 +- .../org/apache/doris/analysis/RefreshLdapStmt.java | 2 +- .../apache/doris/analysis/RefreshTableStmt.java| 2 +- .../doris/analysis/ResumeRoutineLoadStmt.java | 2 +- .../apache/doris/analysis/ResumeSyncJobStmt.java | 2 +- .../apache/doris/analysis/ShowCreateLoadStmt.java | 2 +- .../doris/analysis/ShowCreateRoutineLoadStmt.java | 2 +- .../apache/doris/analysis/ShowRoutineLoadStmt.java | 2 +- .../doris/analysis/ShowRoutineLoadTaskStmt.java| 2 +- .../apache/doris/analysis/StopRoutineLoadStmt.java | 2 +- .../org/apache/doris/analysis/StopSyncJobStmt.java | 2 +- .../java/org/apache/doris/analysis/SyncStmt.java | 2 +- .../org/apache/doris/analysis/UnifiedLoadStmt.java | 2 +- .../doris/nereids/parser/LogicalPlanBuilder.java | 5 +- .../nereids/trees/plans/commands/LoadCommand.java | 4 +- 40 files changed, 194 insertions(+), 60 deletions(-) diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 index 32f5293c00e..b46b55a96e0 100644 --- a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 +++ b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 @@ -70,6 +70,11 @@ unsupportedStatement | unsupportedAdminStatement | unsupportedTransactionStatement | unsupportedRecoverStatement +| unsupportedCancelStatement +| unsupportedJobStatement +| unsupportedCleanStatement +| unsupportedRefreshStatement +| unsupportedLoadStatement | unsupportedOtherStatement ; @@ -121,14 +126,11 @@ supportedDmlStatement partitionSpec? tableAlias (USING relations)? whereClause? #delete -| LOAD LABEL lableName=identifier +| LOAD LABEL lableName=multipartIdentifier LEFT_PAREN dataDescs+=dataDesc (COMMA dataDescs+=dataDesc)* RIGHT_PAREN (withRemoteStorageSystem)? -(PROPERTIES LEFT_PAREN properties=propertyItemList RIGHT_PAREN)? +propertyClause? (commentSpec)? #load -| LOAD mysqlDataDesc -(PROPERTIES LEFT_PAREN properties=propertyItemList RIGHT_PAREN)? -(commentSpec)? #mysqlLoad | EXPORT TABLE tableName=multipartIdentifier (PARTITION partition=identifierList)? (whereClause)? @@ -200,6 +202,130 @@ lockTable (READ (LOCAL)? | (LOW_PRIORITY)? WRITE) ; +unsupportedLoadStatement +: LOAD mysqlDataDesc +(PROPERTIES LEFT_PAREN properties=propertyItemList RIGHT_PAREN)? +(comme
(doris) branch mtmv_view created (now 1bc6d78ab43)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch mtmv_view in repository https://gitbox.apache.org/repos/asf/doris.git at 1bc6d78ab43 [feat](nereids) adjust stats derive by delta row (#39222) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch 2.1_35899 deleted (was 243a90ff3ee)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch 2.1_35899 in repository https://gitbox.apache.org/repos/asf/doris.git was 243a90ff3ee [fix](Nereids) not do distinct when aggregate with distinct project The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch 2.1_35863 deleted (was d1f0f9ee3bb)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch 2.1_35863 in repository https://gitbox.apache.org/repos/asf/doris.git was d1f0f9ee3bb [opt](Nereids) use date signature for date arithmetic as far as possible The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch 2.1_39499 deleted (was 0579d1ecae2)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch 2.1_39499 in repository https://gitbox.apache.org/repos/asf/doris.git was 0579d1ecae2 [opt](Nereids) do not fallback if nereids failed because timeout (#39499) The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [feature](statistics) Support get row count for JDBC external table. (#38889)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 8e44e50e592 [feature](statistics) Support get row count for JDBC
external table. (#38889)
8e44e50e592 is described below
commit 8e44e50e592476cf78487ccda7ed20fd14272cef
Author: Jibing-Li <64681310+jibing...@users.noreply.github.com>
AuthorDate: Fri Aug 30 19:24:09 2024 +0800
[feature](statistics) Support get row count for JDBC external table.
(#38889)
Support get row count for JDBC external table by executing select
count(1) from table.
Return -1 when external table row count is unknown.
---
.../doris/datasource/jdbc/JdbcExternalTable.java | 58 ++
.../jdbc/test_mysql_jdbc_statistics.groovy | 47 +++---
2 files changed, 88 insertions(+), 17 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalTable.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalTable.java
index 21d73499f2e..495311bc087 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalTable.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalTable.java
@@ -17,19 +17,28 @@
package org.apache.doris.datasource.jdbc;
+import org.apache.doris.analysis.StatementBase;
import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.JdbcResource;
import org.apache.doris.catalog.JdbcTable;
import org.apache.doris.datasource.ExternalTable;
import org.apache.doris.datasource.SchemaCacheValue;
+import org.apache.doris.qe.AutoCloseConnectContext;
+import org.apache.doris.qe.StmtExecutor;
import org.apache.doris.statistics.AnalysisInfo;
import org.apache.doris.statistics.BaseAnalysisTask;
import org.apache.doris.statistics.ExternalAnalysisTask;
+import org.apache.doris.statistics.ResultRow;
+import org.apache.doris.statistics.util.StatisticsUtil;
import org.apache.doris.thrift.TTableDescriptor;
+import org.apache.commons.text.StringSubstitutor;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import java.util.Optional;
/**
@@ -38,6 +47,9 @@ import java.util.Optional;
public class JdbcExternalTable extends ExternalTable {
private static final Logger LOG =
LogManager.getLogger(JdbcExternalTable.class);
+public static final String MYSQL_ROW_COUNT_SQL = "SELECT * FROM QUERY"
++ "(\"catalog\"=\"${ctlName}\", \"query\"=\"show table status from
`${dbName}` like '${tblName}'\");";
+
private JdbcTable jdbcTable;
/**
@@ -98,4 +110,50 @@ public class JdbcExternalTable extends ExternalTable {
makeSureInitialized();
return new ExternalAnalysisTask(info);
}
+
+@Override
+public long fetchRowCount() {
+Map params = new HashMap<>();
+params.put("ctlName", catalog.getName());
+params.put("dbName", dbName);
+params.put("tblName", name);
+switch (((JdbcExternalCatalog) catalog).getDatabaseTypeName()) {
+case JdbcResource.MYSQL:
+try (AutoCloseConnectContext r =
StatisticsUtil.buildConnectContext(false, false)) {
+StringSubstitutor stringSubstitutor = new
StringSubstitutor(params);
+String sql =
stringSubstitutor.replace(MYSQL_ROW_COUNT_SQL);
+StmtExecutor stmtExecutor = new
StmtExecutor(r.connectContext, sql);
+List resultRows =
stmtExecutor.executeInternalQuery();
+if (resultRows == null || resultRows.size() != 1) {
+LOG.info("No mysql status found for table {}.{}.{}",
catalog.getName(), dbName, name);
+return -1;
+}
+StatementBase parsedStmt = stmtExecutor.getParsedStmt();
+if (parsedStmt == null || parsedStmt.getColLabels() ==
null) {
+LOG.info("No column label found for table {}.{}.{}",
catalog.getName(), dbName, name);
+return -1;
+}
+ResultRow resultRow = resultRows.get(0);
+List colLabels = parsedStmt.getColLabels();
+int index = colLabels.indexOf("TABLE_ROWS");
+if (index == -1) {
+LOG.info("No TABLE_ROWS in status for table {}.{}.{}",
catalog.getName(), dbName, name);
+return -1;
+}
+long rows = Long.parseLong(resultRow.get(index));
+
(doris) branch master updated: [opt](Nereids) support some statements syntax (#40151)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new d413b0be247 [opt](Nereids) support some statements syntax (#40151) d413b0be247 is described below commit d413b0be24766ea3913a1513e6838eaa6732941f Author: morrySnow <101034200+morrys...@users.noreply.github.com> AuthorDate: Fri Aug 30 14:16:09 2024 +0800 [opt](Nereids) support some statements syntax (#40151) - copy into - backup - help - install plugin - uninstall plugin - lock tables - unlock tables - recover - start transaction --- .../antlr4/org/apache/doris/nereids/DorisParser.g4 | 46 +- .../java/org/apache/doris/analysis/BackupStmt.java | 2 +- .../java/org/apache/doris/analysis/CopyStmt.java | 2 +- .../java/org/apache/doris/analysis/HelpStmt.java | 2 +- .../apache/doris/analysis/InstallPluginStmt.java | 2 +- .../org/apache/doris/analysis/LockTablesStmt.java | 2 +- .../org/apache/doris/analysis/RecoverDbStmt.java | 2 +- .../doris/analysis/RecoverPartitionStmt.java | 2 +- .../apache/doris/analysis/RecoverTableStmt.java| 2 +- .../org/apache/doris/analysis/RestoreStmt.java | 2 +- .../apache/doris/analysis/UninstallPluginStmt.java | 2 +- .../apache/doris/analysis/UnlockTablesStmt.java| 2 +- .../org/apache/doris/analysis/UnsupportedStmt.java | 2 +- .../apache/doris/analysis/WarmUpClusterStmt.java | 2 +- 14 files changed, 58 insertions(+), 14 deletions(-) diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 index f79088b73aa..32f5293c00e 100644 --- a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 +++ b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 @@ -69,6 +69,8 @@ unsupportedStatement | unsupportedGrantRevokeStatement | unsupportedAdminStatement | unsupportedTransactionStatement +| unsupportedRecoverStatement +| unsupportedOtherStatement ; materailizedViewStatement @@ -172,6 +174,40 @@ supportedDropStatement : DROP CATALOG RECYCLE BIN WHERE idType=STRING_LITERAL EQ id=INTEGER_VALUE #dropCatalogRecycleBin ; +unsupportedOtherStatement +: HELP mark=identifierOrText #help +| INSTALL PLUGIN FROM source=identifierOrText properties=propertyClause? #installPlugin +| UNINSTALL PLUGIN name=identifierOrText #uninstallPlugin +| LOCK TABLES (lockTable (COMMA lockTable)*)? #lockTables +| UNLOCK TABLES #unlockTables +| WARM UP CLUSTER destination=identifier WITH +(CLUSTER source=identifier | (warmUpItem (COMMA warmUpItem)*)) FORCE? #warmUpCluster +| BACKUP SNAPSHOT label=multipartIdentifier TO repo=identifier +((ON | EXCLUDE) LEFT_PAREN baseTableRef (COMMA baseTableRef)* RIGHT_PAREN)? +properties=propertyClause? #backup +| RESTORE SNAPSHOT label=multipartIdentifier FROM repo=identifier +((ON | EXCLUDE) LEFT_PAREN baseTableRef (COMMA baseTableRef)* RIGHT_PAREN)? +properties=propertyClause? #restore +| START TRANSACTION (WITH CONSISTENT SNAPSHOT)? #unsupportedStartTransaction +; + +warmUpItem +: TABLE tableName=multipartIdentifier (PARTITION partitionName=identifier)? +; + +lockTable +: name=multipartIdentifier (AS alias=identifierOrText)? +(READ (LOCAL)? | (LOW_PRIORITY)? WRITE) +; + +unsupportedRecoverStatement +: RECOVER DATABASE name=identifier id=INTEGER_VALUE? (AS alias=identifier)? #recoverDatabase +| RECOVER TABLE name=multipartIdentifier +id=INTEGER_VALUE? (AS alias=identifier)? #recoverTable +| RECOVER PARTITION name=identifier id=INTEGER_VALUE? (AS alias=identifier)? +FROM tableName=multipartIdentifier #recoverPartition +; + unsupportedAdminStatement : ADMIN SHOW REPLICA STATUS FROM baseTableRef wildWhere? #adminShowReplicaStatus | ADMIN SHOW REPLICA DISTRIBUTION FROM baseTableRef #adminShowReplicaDistribution @@ -551,7 +587,15 @@ unsupportedUseStatement ; unsupportedDmlStatement -: TRUNCATE TABLE multipartIdentifier specifiedPartition? # truncateTable +: TRUNCATE TABLE multipartIdentifier specifiedPartition? #truncateTable +| COPY INTO selectHint? name=multipartIdentifier columns=identifierList
