[Impala-ASF-CR] IMPALA-9448: (Addendum) Avoid generic recreate for Ozone bucket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19082 ) Change subject: IMPALA-9448: (Addendum) Avoid generic recreate for Ozone bucket .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11494/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19082 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I31a3813fac6f2d68859ba7fed53b5c5a82502780 Gerrit-Change-Number: 19082 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Sep 2022 05:47:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9448: (Addendum) Avoid generic recreate for Ozone bucket
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19082 to look at the new patch set (#2). Change subject: IMPALA-9448: (Addendum) Avoid generic recreate for Ozone bucket .. IMPALA-9448: (Addendum) Avoid generic recreate for Ozone bucket Avoids using `hadoop fs` to recreate an Ozone bucket during snapshot load, as this removes the encryption key. Change-Id: I31a3813fac6f2d68859ba7fed53b5c5a82502780 --- M testdata/bin/load-test-warehouse-snapshot.sh 1 file changed, 7 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/19082/2 -- To view, visit http://gerrit.cloudera.org:8080/19082 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I31a3813fac6f2d68859ba7fed53b5c5a82502780 Gerrit-Change-Number: 19082 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-3119: DDL support for bucketed tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19055 ) Change subject: IMPALA-3119: DDL support for bucketed tables .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11493/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I919b4d4139bc3a7784fa6fdb6f064e25666d548e Gerrit-Change-Number: 19055 Gerrit-PatchSet: 5 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Sep 2022 04:35:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3119: DDL support for bucketed tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19055 ) Change subject: IMPALA-3119: DDL support for bucketed tables .. Patch Set 5: (3 comments) http://gerrit.cloudera.org:8080/#/c/19055/5/common/thrift/CatalogObjects.thrift File common/thrift/CatalogObjects.thrift: http://gerrit.cloudera.org:8080/#/c/19055/5/common/thrift/CatalogObjects.thrift@196 PS5, Line 196: // When create bucketd table, need to know about bucket's type, bucket's columns and number. line too long (92 > 90) http://gerrit.cloudera.org:8080/#/c/19055/5/fe/src/main/java/org/apache/impala/analysis/TableDef.java File fe/src/main/java/org/apache/impala/analysis/TableDef.java: http://gerrit.cloudera.org:8080/#/c/19055/5/fe/src/main/java/org/apache/impala/analysis/TableDef.java@792 PS5, Line 792: if (bucketDesc.getBucket_columns() == null || bucketDesc.getBucket_columns().size() == 0) { line too long (95 > 90) http://gerrit.cloudera.org:8080/#/c/19055/5/fe/src/main/java/org/apache/impala/util/BucketUtils.java File fe/src/main/java/org/apache/impala/util/BucketUtils.java: http://gerrit.cloudera.org:8080/#/c/19055/5/fe/src/main/java/org/apache/impala/util/BucketUtils.java@40 PS5, Line 40: TBucketType bucketType = TBucketType.valueOf(params.get(Table.HIVE_IMPALA_BUCKET_TYPE)); line too long (92 > 90) -- To view, visit http://gerrit.cloudera.org:8080/19055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I919b4d4139bc3a7784fa6fdb6f064e25666d548e Gerrit-Change-Number: 19055 Gerrit-PatchSet: 5 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Sep 2022 04:29:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3119: DDL support for bucketed tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19055 ) Change subject: IMPALA-3119: DDL support for bucketed tables .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8643/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I919b4d4139bc3a7784fa6fdb6f064e25666d548e Gerrit-Change-Number: 19055 Gerrit-PatchSet: 5 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Sep 2022 04:23:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11606: add 'untracked memory' metric.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19029 ) Change subject: IMPALA-11606: add 'untracked memory' metric. .. Patch Set 3: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8641/ -- To view, visit http://gerrit.cloudera.org:8080/19029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib16e00109d732f759c96c7a967eb1cc32124a03f Gerrit-Change-Number: 19029 Gerrit-PatchSet: 3 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Fri, 30 Sep 2022 04:23:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3119: DDL support for bucketed tables
Baike Xia has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/19055 ) Change subject: IMPALA-3119: DDL support for bucketed tables .. IMPALA-3119: DDL support for bucketed tables Add syntactic support for creating bucketed table. The specific syntax is as follows: CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name( col_name data_type [constraint_specification] [COMMENT 'col_comment'] [, ...] ) [PARTITIONED BY (col_name data_type [COMMENT 'col_comment'], ...)] [BUCKETED BY HASH([column [, column ...]])|RANDOM INTO 24 BUCKETS [SORT BY ([column [, column ...]])] [COMMENT 'table_comment'] [ROW FORMAT row_format] [WITH SERDEPROPERTIES ('key1'='value1', 'key2'='value2', ...)] [STORED AS file_format] [LOCATION 'hdfs_path'] [CACHED IN 'pool_name' [WITH REPLICATION = integer] | UNCACHED] [TBLPROPERTIES ('key1'='value1', 'key2'='value2', ...)] Instructions: 1. CLUSTERED BY of Hive is not supported, because HINT has the keyword; 2. The bucket partitioning algorithm contains HASH, RANDOM, KUDU_HASH. The default value is HASH; 3. INTO 24 BUCKETS, specifies the number of buckets, the default value is 16; 4. Create Bucketed Table statements that do not support Kudu and Iceberg tables, but for a Kudu table, the hash partition is equivalent to a bucket, and the optimization rule applies to join query; 5. In the current version, alter operations(add/drop/change/replace columns) on bucketed tables are not supported; This COMMIT is the first subtask of IMPALA-3118. Change-Id: I919b4d4139bc3a7784fa6fdb6f064e25666d548e --- M common/thrift/CatalogObjects.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A fe/src/main/java/org/apache/impala/util/BucketUtils.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M testdata/workloads/functional-query/queries/QueryTest/create-table.test 15 files changed, 408 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/19055/5 -- To view, visit http://gerrit.cloudera.org:8080/19055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I919b4d4139bc3a7784fa6fdb6f064e25666d548e Gerrit-Change-Number: 19055 Gerrit-PatchSet: 5 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11555: external Udf support decimal(*, *)
Minghui Zhu has posted comments on this change. ( http://gerrit.cloudera.org:8080/19021 ) Change subject: IMPALA-11555: external Udf support decimal(*, *) .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/19021/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19021/3//COMMIT_MSG@9 PS3, Line 9: Increase syntax parsing and remove some restrictions. > To help others to understand the sense of "decimal(*, *)", could you please I have written in https://issues.apache.org/jira/browse/IMPALA-11555 -- To view, visit http://gerrit.cloudera.org:8080/19021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I09f19da5b1660ed0b882497fc321e6cc1b9b55d0 Gerrit-Change-Number: 19021 Gerrit-PatchSet: 3 Gerrit-Owner: Minghui Zhu Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Minghui Zhu Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 30 Sep 2022 04:04:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11606: add 'untracked memory' metric.
Xianqing He has posted comments on this change. ( http://gerrit.cloudera.org:8080/19029 ) Change subject: IMPALA-11606: add 'untracked memory' metric. .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib16e00109d732f759c96c7a967eb1cc32124a03f Gerrit-Change-Number: 19029 Gerrit-PatchSet: 3 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Fri, 30 Sep 2022 04:03:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
Xiang Yang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 ) Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses .. Patch Set 4: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb Gerrit-Change-Number: 19046 Gerrit-PatchSet: 4 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Fri, 30 Sep 2022 03:51:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11235: Support Pluggable Authentication for Impala
Xiang Yang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19042 ) Change subject: IMPALA-11235: Support Pluggable Authentication for Impala .. Patch Set 3: (9 comments) Thanks for your contribution, minghui! http://gerrit.cloudera.org:8080/#/c/19042/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19042/3//COMMIT_MSG@1 PS3, Line 1: Parent: 296e9441 (IMPALA-11599/IMPALA-11605: GCC 10: Fix gdb and change to flatbuffers 1.9) can we give this authentication implementation a more explict name instead of the 'plugin'? such as commandline authentication .etc. can you add docs like 'https://impala.apache.org/docs/build/html/topics/impala_ldap.html'? http://gerrit.cloudera.org:8080/#/c/19042/3/be/src/rpc/authentication.cc File be/src/rpc/authentication.cc: http://gerrit.cloudera.org:8080/#/c/19042/3/be/src/rpc/authentication.cc@309 PS3, Line 309: LOG(ERROR) << "Plugin authentication failed for empty password。"; nit: don't use chinese punctuation. http://gerrit.cloudera.org:8080/#/c/19042/3/bin/plugin_for_test.sh File bin/plugin_for_test.sh: http://gerrit.cloudera.org:8080/#/c/19042/3/bin/plugin_for_test.sh@1 PS3, Line 1: #!/usr/bin/env bash lack of apache License header. http://gerrit.cloudera.org:8080/#/c/19042/3/bin/plugin_for_test.sh@18 PS3, Line 18: exit 1 use a different exit code to distinguish from line 4. http://gerrit.cloudera.org:8080/#/c/19042/3/fe/src/test/java/org/apache/impala/customcluster/PluginHS2Test.java File fe/src/test/java/org/apache/impala/customcluster/PluginHS2Test.java: http://gerrit.cloudera.org:8080/#/c/19042/3/fe/src/test/java/org/apache/impala/customcluster/PluginHS2Test.java@1 PS3, Line 1: // or more contributor license agreements. See the NOTICE file Incomplete lincense header. http://gerrit.cloudera.org:8080/#/c/19042/3/fe/src/test/java/org/apache/impala/customcluster/PluginHS2Test.java@53 PS3, Line 53: * plugin authentication is being used. nit: is used. http://gerrit.cloudera.org:8080/#/c/19042/3/fe/src/test/java/org/apache/impala/customcluster/PluginHS2Test.java@67 PS3, Line 67: static void verifySuccess(TStatus status) throws Exception { can use org.apache.hive.jdbc.Utils.verifySuccess() instead. http://gerrit.cloudera.org:8080/#/c/19042/3/fe/src/test/java/org/apache/impala/customcluster/PluginHS2Test.java@155 PS3, Line 155: transport.open(); do we need close THttpClient at the end of the test? http://gerrit.cloudera.org:8080/#/c/19042/3/fe/src/test/java/org/apache/impala/customcluster/PluginImpylaHttpTest.java File fe/src/test/java/org/apache/impala/customcluster/PluginImpylaHttpTest.java: http://gerrit.cloudera.org:8080/#/c/19042/3/fe/src/test/java/org/apache/impala/customcluster/PluginImpylaHttpTest.java@38 PS3, Line 38: * Impyla HTTP connectivity tests with LDAP authentication. What's the difference with LdapImpylaHttpTest? I see many Duplicate codes, can we refactor it to base class and inherited class mode? -- To view, visit http://gerrit.cloudera.org:8080/19042 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieaaef6bffe641ece30e7fc3756688ef30722dfe1 Gerrit-Change-Number: 19042 Gerrit-PatchSet: 3 Gerrit-Owner: Minghui Zhu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Minghui Zhu Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Fri, 30 Sep 2022 03:47:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3119: DDL support for bucketed tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19055 ) Change subject: IMPALA-3119: DDL support for bucketed tables .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/11492/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I919b4d4139bc3a7784fa6fdb6f064e25666d548e Gerrit-Change-Number: 19055 Gerrit-PatchSet: 4 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Sep 2022 03:30:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3119: DDL support for bucketed tables
Baike Xia has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19055 Change subject: IMPALA-3119: DDL support for bucketed tables .. IMPALA-3119: DDL support for bucketed tables Add syntactic support for creating bucketed table. The specific syntax is as follows: CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name( col_name data_type [constraint_specification] [COMMENT 'col_comment'] [, ...] ) [PARTITIONED BY (col_name data_type [COMMENT 'col_comment'], ...)] [BUCKETED BY HASH([column [, column ...]])|RANDOM INTO 24 BUCKETS [SORT BY ([column [, column ...]])] [COMMENT 'table_comment'] [ROW FORMAT row_format] [WITH SERDEPROPERTIES ('key1'='value1', 'key2'='value2', ...)] [STORED AS file_format] [LOCATION 'hdfs_path'] [CACHED IN 'pool_name' [WITH REPLICATION = integer] | UNCACHED] [TBLPROPERTIES ('key1'='value1', 'key2'='value2', ...)] Instructions: 1. CLUSTERED BY of Hive is not supported, because HINT has the keyword; 2. The bucket partitioning algorithm contains HASH, RANDOM, KUDU_HASH. The default value is HASH; 3. INTO 24 BUCKETS, specifies the number of buckets, the default value is 16; 4. Create Bucketed Table statements that do not support Kudu and Iceberg tables, but for a Kudu table, the hash partition is equivalent to a bucket, and the optimization rule applies to join query; 5. In the current version, alter operations(add/drop/change/replace columns) on bucketed tables are not supported; This COMMIT is the first subtask of IMPALA-3118. Change-Id: I919b4d4139bc3a7784fa6fdb6f064e25666d548e --- M common/thrift/CatalogObjects.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A fe/src/main/java/org/apache/impala/util/BucketUtils.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M testdata/workloads/functional-query/queries/QueryTest/create-table.test 16 files changed, 411 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/19055/4 -- To view, visit http://gerrit.cloudera.org:8080/19055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I919b4d4139bc3a7784fa6fdb6f064e25666d548e Gerrit-Change-Number: 19055 Gerrit-PatchSet: 4 Gerrit-Owner: Baike Xia
[Impala-ASF-CR] IMPALA-11566: Provide SQL parsing capabilities to other applications in the form of jar packages
Xiang Yang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19023 ) Change subject: IMPALA-11566: Provide SQL parsing capabilities to other applications in the form of jar packages .. Patch Set 2: (4 comments) Hi minghui, thanks for your contribution! I post some comments. http://gerrit.cloudera.org:8080/#/c/19023/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19023/2//COMMIT_MSG@1 PS2, Line 1: Parent: 19114c72 (IMPALA-11578: Exclude locality test for remote FS) can you add some UT? http://gerrit.cloudera.org:8080/#/c/19023/2/fe/src/main/java/org/apache/impala/analysis/StmtNode.java File fe/src/main/java/org/apache/impala/analysis/StmtNode.java: http://gerrit.cloudera.org:8080/#/c/19023/2/fe/src/main/java/org/apache/impala/analysis/StmtNode.java@49 PS2, Line 49: @SuppressWarnings("unchecked") why use @SuppressWarnings here? http://gerrit.cloudera.org:8080/#/c/19023/2/fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java File fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java: http://gerrit.cloudera.org:8080/#/c/19023/2/fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java@82 PS2, Line 82: if (BackendConfig.INSTANCE != null && BackendConfig.INSTANCE.skipLookupSymbol()) { what is skipLookupSymbol() for? http://gerrit.cloudera.org:8080/#/c/19023/2/fe/src/main/java/org/apache/impala/util/StmtVisitor.java File fe/src/main/java/org/apache/impala/util/StmtVisitor.java: http://gerrit.cloudera.org:8080/#/c/19023/2/fe/src/main/java/org/apache/impala/util/StmtVisitor.java@26 PS2, Line 26: List visit(StmtNode node); Why don't you use or extend org.apache.impala.util.Visitor? If this interface is necessary, can you add some description javadoc? -- To view, visit http://gerrit.cloudera.org:8080/19023 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I246b088310f25b809d9e598c0ecc6613d9749ee3 Gerrit-Change-Number: 19023 Gerrit-PatchSet: 2 Gerrit-Owner: Minghui Zhu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Minghui Zhu Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Fri, 30 Sep 2022 02:43:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11612: fix ORDER BY expression rewrite bug
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19049 ) Change subject: IMPALA-11612: fix ORDER BY expression rewrite bug .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8642/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I438c61c647e3f99665c94ea15e3a9d98b9ff9912 Gerrit-Change-Number: 19049 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 30 Sep 2022 02:10:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. Patch Set 1: (3 comments) This is a pretty nice fix! http://gerrit.cloudera.org:8080/#/c/19052/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19052/1//COMMIT_MSG@16 PS1, Line 16: - Run existing test_load.py We also need tests to verify the INSERT events. Could you add some tests in tests/metadata/test_event_processing.py, e.g. add some cases for LOAD DATA in test_event_based_replication? http://gerrit.cloudera.org:8080/#/c/19052/1/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/19052/1/be/src/service/client-request-state.cc@806 PS1, Line 806: response.partition_name nit: Could you add a comment mentioning that the partition_name is an empty string for unpartitioned tables? It's the same as DataSink::ROOT_PARTITION_KEY. http://gerrit.cloudera.org:8080/#/c/19052/1/be/src/service/client-request-state.cc@809 PS1, Line 809: catalog_update.__set_header(TCatalogServiceRequestHeader()); : catalog_update.header.__set_requesting_user(effective_user()); : catalog_update.header.__set_client_ip(session()->network_address.hostname); : catalog_update.header.__set_want_minimal_response(FLAGS_use_local_catalog); : catalog_update.header.__set_redacted_sql_stmt( : query_ctx_.client_request.__isset.redacted_stmt ? : query_ctx_.client_request.redacted_stmt : query_ctx_.client_request.stmt); nit: these duplicate the code in ClientRequestState::ExecLoadDataRequestImpl(). It'd be nice to extract them into a method like GetCatalogServiceRequestHeader(), and simplify the code to catalog_update.__set_header(GetCatalogServiceRequestHeader()) -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 1 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Fri, 30 Sep 2022 01:55:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10436: Require lower privilege for external Kudu table creation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17640 ) Change subject: IMPALA-10436: Require lower privilege for external Kudu table creation .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11491/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17640 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7936e1d8c48696169f7ad7ad92abe44a26eea3c4 Gerrit-Change-Number: 17640 Gerrit-PatchSet: 7 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 30 Sep 2022 01:18:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10436: Require lower privilege for external Kudu table creation
Fang-Yu Rao has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/17640 ) Change subject: IMPALA-10436: Require lower privilege for external Kudu table creation .. IMPALA-10436: Require lower privilege for external Kudu table creation This patch lowers the privilege requirement for external Kudu table creation. Before this patch, a user was required to have the ALL privilege on SERVER if the user wanted to create an external Kudu table. In this patch we introduce a new type of resources called storage handler URI and a new access type called RWSTORAGE that will be supported by Apache Ranger once RANGER-3281 is resolved, which in turn depends on the release of Apache Hive 4.0 that consists of HIVE-24705. Specifically, after this patch, a user will be allowed to create an external Kudu table as long as the user is granted the RWSTORAGE privilege on the resource specified by a storage handler URI that points to an existing Kudu table. For instance, in order for a user 'non_owner' to create an external Kudu table based on an existing Kudu table 'impala::tpch_kudu.nation', it suffices to execute the following command as an administrator to grant the necessary privilege to the requesting user, where "localhost" is the default address of Kudu master host assuming there is only one single master host in this example. GRANT RWSTORAGE ON STORAGEHANDLER_URI 'kudu://localhost/impala::tpch_kudu.nation' TO USER non_owner One may be wondering why we do not simply cancel the privilege check that required the ALL privilege on SERVER for external Kudu table creation. One scenario in which such relaxation is not secure is when the owner or the creator of the existing Kudu table is different from the requesting user who wants to create an external Kudu table in Impala. Not requiring any additional privilege check would allow a user without any privilege to retrieve the contents of the existing Kudu table. On the other hand, after this patch we still require a user to have the ALL privilege on SERVER when the table property of 'kudu.master_addresses' is specified in a query that tries to create a Kudu table whether or not the table is external. To be more specific, the user 'non_owner' would be able to create an external Kudu table using the following statement once being granted the RWSTORAGE privilege on the specified storage handler URI above. CREATE EXTERNAL TABLE default.kudu_tbl STORED AS KUDU TBLPROPERTIES ('kudu.table_name'='impala::tpch_kudu.nation') However, the following query submitted by the same user would be rejected due to the user 'non_owner' not being granted the ALL privilege on SERVER. CREATE EXTERNAL TABLE default.kudu_tbl STORED AS KUDU TBLPROPERTIES ('kudu.table_name'='impala::tpch_kudu.nation', 'kudu.master_addresses'='localhost') We do not relax such a requirement in that specifying the addresses of Kudu master hosts to connect should still be considered as an administrative operation. Testing: - Added various FE and E2E tests to verify Impala's behavior after this patch with respect to external Kudu table creation. - Verified that this patch passes the core tests in the DEBUG build. Change-Id: I7936e1d8c48696169f7ad7ad92abe44a26eea3c4 --- M common/thrift/CatalogObjects.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/ShowGrantPrincipalStmt.java A fe/src/main/java/org/apache/impala/analysis/StorageHandlerUri.java M fe/src/main/java/org/apache/impala/authorization/Authorizable.java M fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java A fe/src/main/java/org/apache/impala/authorization/AuthorizableStorageHandlerUri.java M fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java M fe/src/main/java/org/apache/impala/authorization/Privilege.java M fe/src/main/java/org/apache/impala/authorization/PrivilegeRequestBuilder.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerUtil.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M testdata/workloads/functional-query/queries/QueryTest/grant_revoke.test M
[Impala-ASF-CR] IMPALA-9448: (Addendum) Avoid generic recreate for Ozone bucket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19082 ) Change subject: IMPALA-9448: (Addendum) Avoid generic recreate for Ozone bucket .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11490/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19082 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I31a3813fac6f2d68859ba7fed53b5c5a82502780 Gerrit-Change-Number: 19082 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Sep 2022 00:16:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11562: (Addendum) Update core-site to include ofs impl
Hello weic...@apache.org, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19081 to look at the new patch set (#2). Change subject: IMPALA-11562: (Addendum) Update core-site to include ofs impl .. IMPALA-11562: (Addendum) Update core-site to include ofs impl Updates core-site to include the ofs implementation rather than o3fs. I believe this still worked because both ofs and o3fs implementations live in the same jar file, so including either statement results in both classes loading. However we should use the correct statement to avoid issues in the future. Change-Id: I41cbc875bef29e730ee2afcbda42f07017119593 --- M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/19081/2 -- To view, visit http://gerrit.cloudera.org:8080/19081 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I41cbc875bef29e730ee2afcbda42f07017119593 Gerrit-Change-Number: 19081 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-9448: (Addendum) Avoid generic recreate for Ozone bucket
Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19082 Change subject: IMPALA-9448: (Addendum) Avoid generic recreate for Ozone bucket .. IMPALA-9448: (Addendum) Avoid generic recreate for Ozone bucket Avoids using `hadoop fs` to recreate an Ozone bucket during snapshot load, as this removes the encryption key. Change-Id: I31a3813fac6f2d68859ba7fed53b5c5a82502780 --- M testdata/bin/load-test-warehouse-snapshot.sh 1 file changed, 7 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/19082/1 -- To view, visit http://gerrit.cloudera.org:8080/19082 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I31a3813fac6f2d68859ba7fed53b5c5a82502780 Gerrit-Change-Number: 19082 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith
[Impala-ASF-CR] IMPALA-11606: add 'untracked memory' metric.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19029 ) Change subject: IMPALA-11606: add 'untracked memory' metric. .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8641/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib16e00109d732f759c96c7a967eb1cc32124a03f Gerrit-Change-Number: 19029 Gerrit-PatchSet: 3 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Thu, 29 Sep 2022 23:21:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11627: Build Impala with cdw dependencies
Yu-Wen Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/18994 ) Change subject: IMPALA-11627: Build Impala with cdw dependencies .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/18994/6/java/pom.xml File java/pom.xml: http://gerrit.cloudera.org:8080/#/c/18994/6/java/pom.xml@109 PS6, Line 109: nit: wrong indentation? -- To view, visit http://gerrit.cloudera.org:8080/18994 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id379030f4b314e139c875584eee438b7416d89a4 Gerrit-Change-Number: 18994 Gerrit-PatchSet: 6 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yu-Wen Lai Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 29 Sep 2022 23:03:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11562: Update core-site to include ofs impl
weic...@apache.org has posted comments on this change. ( http://gerrit.cloudera.org:8080/19081 ) Change subject: IMPALA-11562: Update core-site to include ofs impl .. Patch Set 1: I guess it was working before because we actually specified it in Hadoop's core-default.xml: https://github.com/apache/hadoop/pull/2276/files -- To view, visit http://gerrit.cloudera.org:8080/19081 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I41cbc875bef29e730ee2afcbda42f07017119593 Gerrit-Change-Number: 19081 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 29 Sep 2022 22:35:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11562: Update core-site to include ofs impl
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19081 ) Change subject: IMPALA-11562: Update core-site to include ofs impl .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11489/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19081 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I41cbc875bef29e730ee2afcbda42f07017119593 Gerrit-Change-Number: 19081 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 29 Sep 2022 22:06:11 + Gerrit-HasComments: No
[Impala-ASF-CR] Speed up default configuration for Docker-based tests
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19038 ) Change subject: Speed up default configuration for Docker-based tests .. Speed up default configuration for Docker-based tests Docker-based parallelized test runs have proven themselves to be quite a bit faster than regular core or exhaustive mode builds. While regular sequential builds have also enjoyed shorter runtimes recently, Docker-based parallel builds still enjoy a speed advantage. Scheduling the parallel build segments is currently driven from the test driver script test-with-docker.py, and the order in which the segments are considered is currently hard-coded. The ordering was originally devised experimentally, by timing several test runs, then ordering the test segments based on expected duration, from longest to shortest. The average wall-clock run times for various test segments have changed since this original ordering was committed: FE tests have gotten significantly longer, while upgrading the default worker instance type cut shortened the serial phase(s) of E2E tests. This patch makes two changes to achieve a shorter overall run time for the Docker-based tests: 1. Reorders the default scheduling order of the test segments, based on currently measured durations 2. Increases the default suite concurrency for execution hosts: bumps suite concurrency from 4 to 5 for machines with memory sizes between 96 and 140 GBs (the currently used worker size) The latter change is also based on measurements: memory usage reports for total peak memory (RSS) and peak memory (RSS) per test segment both showed significant amounts of unused memory on the current default worker instance size (having 32 CPUs and 128 GB of RAM). Experiments showed that this machine size can reliable handle five concurrent containerized test sessions with some safety margin remaining, so the patch increases the default concurrency for this machine category. with both changes applied the duration of a core-mode test run with default settings is reduced from 2h45 to 2h25 (on average). Tested by running the Docker-based default test suite in core mode, with Ubuntu 16.04 and Rocky Linux 8.5 base images. Change-Id: Ifb609bcfb10e9f9b281cc6b375c36c9638db168b Reviewed-on: http://gerrit.cloudera.org:8080/19038 Reviewed-by: Michael Smith Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins --- M docker/test-with-docker.py 1 file changed, 3 insertions(+), 3 deletions(-) Approvals: Michael Smith: Looks good to me, but someone else must approve Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/19038 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ifb609bcfb10e9f9b281cc6b375c36c9638db168b Gerrit-Change-Number: 19038 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-11562: Update core-site to include ofs impl
Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19081 Change subject: IMPALA-11562: Update core-site to include ofs impl .. IMPALA-11562: Update core-site to include ofs impl Updates core-site to include the ofs implementation rather than o3fs. I believe this still worked because both ofs and o3fs implementations live in the same jar file, so including either statement results in both classes loading. However we should use the correct statement to avoid issues in the future. Change-Id: I41cbc875bef29e730ee2afcbda42f07017119593 --- M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/19081/1 -- To view, visit http://gerrit.cloudera.org:8080/19081 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I41cbc875bef29e730ee2afcbda42f07017119593 Gerrit-Change-Number: 19081 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Smith
[Impala-ASF-CR] IMPALA-10851: Codegen for structs
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18526 ) Change subject: IMPALA-10851: Codegen for structs .. Patch Set 12: Code-Review-1 (2 comments) http://gerrit.cloudera.org:8080/#/c/18526/12/be/src/runtime/descriptors.cc File be/src/runtime/descriptors.cc: http://gerrit.cloudera.org:8080/#/c/18526/12/be/src/runtime/descriptors.cc@972 PS12, Line 972: case TYPE_ARRAY: { // CollectionVal has same memory layout as StringVal. My point in an earlier comment was that https://gerrit.cloudera.org/c/18736/ added TYPE_MAP to some of these cases, and I think it needs to be added here and line 1024 after the rebase. http://gerrit.cloudera.org:8080/#/c/18526/10/tests/query_test/test_nested_types.py File tests/query_test/test_nested_types.py: http://gerrit.cloudera.org:8080/#/c/18526/10/tests/query_test/test_nested_types.py@143 PS10, Line 143: if new_vector.get_value('exec_option')['disable_codegen'] == 'False': > I reversed the order because in non-exhaustive builds only a subset of the Reversing order seems fine, but I'd add a comment why. I'd also add the other options always set to 0 (even when they're not used) because I think it's less confusing if they're in the initial setup and not conditional down in the tests. -- To view, visit http://gerrit.cloudera.org:8080/18526 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5272c3f095fd9f07877104ee03c8e43d0c4ec0b6 Gerrit-Change-Number: 18526 Gerrit-PatchSet: 12 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 29 Sep 2022 18:16:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11606: add 'untracked memory' metric.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19029 ) Change subject: IMPALA-11606: add 'untracked memory' metric. .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8640/ -- To view, visit http://gerrit.cloudera.org:8080/19029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib16e00109d732f759c96c7a967eb1cc32124a03f Gerrit-Change-Number: 19029 Gerrit-PatchSet: 3 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Thu, 29 Sep 2022 18:08:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19051 ) Change subject: IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2 .. IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2 In HIVE-22193, Hive added graceful shutdown for HiveServer2. This modified some of the startup scripts to maintain a pid file. On startup, it verified that the pid is not already running. Impala uses the hive startup script, but it kills HiveServer2 with testdata/bin/kill-java-service.sh. If the pid file outlives the process, then the OS may reuse the pid and this can cause problems on startup. This modifies testdata/bin/kill-hive-server.sh to remove the pid file. testdata/bin/kill-java-service.sh would fail if it did not kill HiveServer2, so it should be safe to remove the pid file after the kill-java-service.sh call. Testing: - Verified the hiveserver2.pid file is removed after shutting down. Change-Id: I813626d06829a86854c6d2c1715f0c5f5109836d Reviewed-on: http://gerrit.cloudera.org:8080/19051 Reviewed-by: Joe McDonnell Tested-by: Joe McDonnell --- M testdata/bin/kill-hive-server.sh 1 file changed, 5 insertions(+), 0 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I813626d06829a86854c6d2c1715f0c5f5109836d Gerrit-Change-Number: 19051 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/19051 ) Change subject: IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2 .. Patch Set 2: Verified+1 Code-Review+2 Only commit message changed, going ahead. -- To view, visit http://gerrit.cloudera.org:8080/19051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I813626d06829a86854c6d2c1715f0c5f5109836d Gerrit-Change-Number: 19051 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 29 Sep 2022 16:11:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2
Hello Quanlong Huang, Csaba Ringhofer, Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19051 to look at the new patch set (#2). Change subject: IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2 .. IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2 In HIVE-22193, Hive added graceful shutdown for HiveServer2. This modified some of the startup scripts to maintain a pid file. On startup, it verified that the pid is not already running. Impala uses the hive startup script, but it kills HiveServer2 with testdata/bin/kill-java-service.sh. If the pid file outlives the process, then the OS may reuse the pid and this can cause problems on startup. This modifies testdata/bin/kill-hive-server.sh to remove the pid file. testdata/bin/kill-java-service.sh would fail if it did not kill HiveServer2, so it should be safe to remove the pid file after the kill-java-service.sh call. Testing: - Verified the hiveserver2.pid file is removed after shutting down. Change-Id: I813626d06829a86854c6d2c1715f0c5f5109836d --- M testdata/bin/kill-hive-server.sh 1 file changed, 5 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/19051/2 -- To view, visit http://gerrit.cloudera.org:8080/19051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I813626d06829a86854c6d2c1715f0c5f5109836d Gerrit-Change-Number: 19051 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8639/ -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 7 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Thu, 29 Sep 2022 16:03:14 + Gerrit-HasComments: No
[Impala-ASF-CR] Prototype to provide an option to install Java 11
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19031 ) Change subject: Prototype to provide an option to install Java 11 .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11488/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328 Gerrit-Change-Number: 19031 Gerrit-PatchSet: 11 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 29 Sep 2022 15:55:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10851: Codegen for structs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18526 ) Change subject: IMPALA-10851: Codegen for structs .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11487/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18526 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5272c3f095fd9f07877104ee03c8e43d0c4ec0b6 Gerrit-Change-Number: 18526 Gerrit-PatchSet: 12 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 29 Sep 2022 15:56:13 + Gerrit-HasComments: No
[Impala-ASF-CR] Prototype to provide an option to install Java 11
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19031 to look at the new patch set (#11). Change subject: Prototype to provide an option to install Java 11 .. Prototype to provide an option to install Java 11 Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328 --- M bin/impala-config.sh M docker/CMakeLists.txt M docker/daemon_entrypoint.sh M docker/install_os_packages.sh 4 files changed, 111 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/19031/11 -- To view, visit http://gerrit.cloudera.org:8080/19031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328 Gerrit-Change-Number: 19031 Gerrit-PatchSet: 11 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-10851: Codegen for structs
Daniel Becker has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/18526 ) Change subject: IMPALA-10851: Codegen for structs .. IMPALA-10851: Codegen for structs IMPALA-9495 added support for struct types in SELECT lists but only with codegen turned off. This commit implements codegen for struct types. To facilitate this, code generation for reading and writing 'AnyVal's has been refactored. A new class, 'CodegenAnyValReadWriteInfo' is introduced. This class is an interface between sources and destinations, one of which is an 'AnyVal' object: sources generate an instance of this class and destinations take that instance and use it to write the value. The other side can for example be tuples from which we read (in the case of 'SlotRef') or tuples we write into (in case of materialisation, see Tuple::CodegenMaterializeExprs()). The main advantage is that sources do not have to know how to write their destinations, only how to read the values (and vice versa). Testing: - enabling tests for structs in select list in codegen mode in tests/query_test/test_nested_types.py Change-Id: I5272c3f095fd9f07877104ee03c8e43d0c4ec0b6 --- M be/src/codegen/CMakeLists.txt A be/src/codegen/codegen-anyval-read-write-info.cc A be/src/codegen/codegen-anyval-read-write-info.h M be/src/codegen/codegen-anyval.cc M be/src/codegen/codegen-anyval.h M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/impala-ir.cc M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h M be/src/exec/aggregator.cc M be/src/exec/filter-context.cc M be/src/exec/hash-table.cc M be/src/exec/hash-table.h M be/src/exprs/CMakeLists.txt M be/src/exprs/hive-udf-call-ir.cc M be/src/exprs/hive-udf-call.cc M be/src/exprs/hive-udf-call.h M be/src/exprs/kudu-partition-expr.cc A be/src/exprs/scalar-expr-evaluator-ir.cc M be/src/exprs/scalar-expr-evaluator.h M be/src/exprs/slot-ref.cc M be/src/exprs/slot-ref.h M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M be/src/runtime/fragment-state.cc M be/src/runtime/fragment-state.h M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/tuple.cc M be/src/udf/udf-internal.h M be/src/udf/udf-ir.cc M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test M tests/authorization/test_ranger.py M tests/query_test/test_nested_types.py 35 files changed, 2,200 insertions(+), 808 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/18526/12 -- To view, visit http://gerrit.cloudera.org:8080/18526 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5272c3f095fd9f07877104ee03c8e43d0c4ec0b6 Gerrit-Change-Number: 18526 Gerrit-PatchSet: 12 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/19051 ) Change subject: IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2 .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/19051/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19051/1//COMMIT_MSG@23 PS1, Line 23: remove > typo Thanks for pointing that out, will fix. -- To view, visit http://gerrit.cloudera.org:8080/19051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I813626d06829a86854c6d2c1715f0c5f5109836d Gerrit-Change-Number: 19051 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 29 Sep 2022 15:34:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11627: Build Impala with cdw dependencies
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18994 to look at the new patch set (#6). Change subject: IMPALA-11627: Build Impala with cdw dependencies .. IMPALA-11627: Build Impala with cdw dependencies The change switches to download cdw versions of Hive dependencies. As cdw depends on older versions of some components than cdp, this is a step back from some components, but this didn't lead to test failures. For maven artifects both cdp and cdw versions are downloaded and added as repositiories, so all component can find the versions it depends on. Other changes: - updated resource-requirements.test due to different number of files tpch_orc_def.lineitem - added redirections/implementations for new HMS APIs in MetastoreService - fixes the bug mentioned in IMPALA-11397 Change-Id: Id379030f4b314e139c875584eee438b7416d89a4 --- M bin/impala-config.sh M bin/jenkins/archive_m2_directory.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/catalog/metastore/HmsApiNameEnum.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M java/pom.xml M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test 8 files changed, 249 insertions(+), 39 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/18994/6 -- To view, visit http://gerrit.cloudera.org:8080/18994 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id379030f4b314e139c875584eee438b7416d89a4 Gerrit-Change-Number: 18994 Gerrit-PatchSet: 6 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11606: add 'untracked memory' metric.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19029 ) Change subject: IMPALA-11606: add 'untracked memory' metric. .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11486/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib16e00109d732f759c96c7a967eb1cc32124a03f Gerrit-Change-Number: 19029 Gerrit-PatchSet: 3 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Thu, 29 Sep 2022 13:19:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11606: add 'untracked memory' metric.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19029 ) Change subject: IMPALA-11606: add 'untracked memory' metric. .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8640/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib16e00109d732f759c96c7a967eb1cc32124a03f Gerrit-Change-Number: 19029 Gerrit-PatchSet: 3 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Thu, 29 Sep 2022 13:05:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11606: add 'untracked memory' metric.
Hello Xianqing He, Jian Zhang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19029 to look at the new patch set (#3). Change subject: IMPALA-11606: add 'untracked memory' metric. .. IMPALA-11606: add 'untracked memory' metric. Add a gauge metric 'untracked memroy' to record the memory size that not tracked by the 'process' mem-tracker. Change-Id: Ib16e00109d732f759c96c7a967eb1cc32124a03f --- M be/src/runtime/mem-tracker.cc M be/src/runtime/mem-tracker.h M common/thrift/metrics.json 3 files changed, 41 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/19029/3 -- To view, visit http://gerrit.cloudera.org:8080/19029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib16e00109d732f759c96c7a967eb1cc32124a03f Gerrit-Change-Number: 19029 Gerrit-PatchSet: 3 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Xianqing He
[Impala-ASF-CR] IMPALA-10851: Codegen for structs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18526 ) Change subject: IMPALA-10851: Codegen for structs .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11485/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18526 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5272c3f095fd9f07877104ee03c8e43d0c4ec0b6 Gerrit-Change-Number: 18526 Gerrit-PatchSet: 11 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 29 Sep 2022 12:50:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10851: Codegen for structs
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/18526 ) Change subject: IMPALA-10851: Codegen for structs .. Patch Set 11: Thanks Michael Smith for the review. First I rebased the change on master, I will address the comments in the next patch set. -- To view, visit http://gerrit.cloudera.org:8080/18526 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5272c3f095fd9f07877104ee03c8e43d0c4ec0b6 Gerrit-Change-Number: 18526 Gerrit-PatchSet: 11 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 29 Sep 2022 12:28:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10851: Codegen for structs
Daniel Becker has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/18526 ) Change subject: IMPALA-10851: Codegen for structs .. IMPALA-10851: Codegen for structs IMPALA-9495 added support for struct types in SELECT lists but only with codegen turned off. This commit implements codegen for struct types. To facilitate this, code generation for reading and writing 'AnyVal's has been refactored. A new class, 'CodegenAnyValReadWriteInfo' is introduced. This class is an interface between sources and destinations, one of which is an 'AnyVal' object: sources generate an instance of this class and destinations take that instance and use it to write the value. The other side can for example be tuples from which we read (in the case of 'SlotRef') or tuples we write into (in case of materialisation, see Tuple::CodegenMaterializeExprs()). The main advantage is that sources do not have to know how to write their destinations, only how to read the values (and vice versa). Testing: - enabling tests for structs in select list in codegen mode in tests/query_test/test_nested_types.py Change-Id: I5272c3f095fd9f07877104ee03c8e43d0c4ec0b6 --- M be/src/codegen/CMakeLists.txt A be/src/codegen/codegen-anyval-read-write-info.cc A be/src/codegen/codegen-anyval-read-write-info.h M be/src/codegen/codegen-anyval.cc M be/src/codegen/codegen-anyval.h M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/impala-ir.cc M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h M be/src/exec/aggregator.cc M be/src/exec/filter-context.cc M be/src/exec/hash-table.cc M be/src/exec/hash-table.h M be/src/exprs/CMakeLists.txt M be/src/exprs/hive-udf-call-ir.cc M be/src/exprs/hive-udf-call.cc M be/src/exprs/hive-udf-call.h M be/src/exprs/kudu-partition-expr.cc A be/src/exprs/scalar-expr-evaluator-ir.cc M be/src/exprs/scalar-expr-evaluator.h M be/src/exprs/slot-ref.cc M be/src/exprs/slot-ref.h M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M be/src/runtime/fragment-state.cc M be/src/runtime/fragment-state.h M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/tuple.cc M be/src/udf/udf-internal.h M be/src/udf/udf-ir.cc M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test M tests/authorization/test_ranger.py M tests/query_test/test_nested_types.py 35 files changed, 2,190 insertions(+), 807 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/18526/11 -- To view, visit http://gerrit.cloudera.org:8080/18526 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5272c3f095fd9f07877104ee03c8e43d0c4ec0b6 Gerrit-Change-Number: 18526 Gerrit-PatchSet: 11 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 ) Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses .. Patch Set 4: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/19046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb Gerrit-Change-Number: 19046 Gerrit-PatchSet: 4 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Thu, 29 Sep 2022 12:14:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11484/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Thu, 29 Sep 2022 11:11:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Daniel Becker has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value If the query contains a VALUES clause with a single value, the 'analyzed query' in the level 2 explain plan contains a UNION where both sides are the value from the VALUES clause. For INSERT INTO double_tbl VALUES (-0.43149576573887316); we get SELECT CAST(-0.43149576573887316 AS DECIMAL(17,17)) UNION SELECT CAST(-0.43149576573887316 AS DECIMAL(17,17)); This is because of a bug in 'SetOperationStmt::toSql()'; 'ValuesStmt's are implemented as special 'UnionStmt's, which derive from 'SetOperationStmt'. 'SetOperationStmt::toSql()' implicitly assumes that there are at least 2 operands and always prints the first one and the last one separately, even if they are the same. Note that 'ValuesStmt' is the only 'SetOperationStmt' that may contain a single operand - it is syntactically impossible in SQL to create other 'SetOperationStmt's with only one operand. This patch modifies 'SetOperationStmt::toSql()' so that if there is only one operand, only that operand is printed (so there is no reference to the set operation in the resulting string). This change does not affect how such queries are handled, only how the analyzed query is printed. Testing: - Extended AnalyzeStmtsTest.TestValuesStmt so it now also checks what VALUES clauses with one and two operands are rewritten to during analysis. Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f --- M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test 4 files changed, 31 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/19048/6 -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8639/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 7 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Thu, 29 Sep 2022 10:51:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 7 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Thu, 29 Sep 2022 10:51:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/19048 ) Change subject: IMPALA-10356: Fix SetOperationStmt::toSql() for VALUES statement with single value .. Patch Set 5: The error is strange, doesn't seem to be related: /home/ubuntu/Impala/toolchain/toolchain-packages-gcc10.4.0/binutils-2.35.1/bin/ld.gold: out of memory collect2: error: ld returned 1 exit status -- To view, visit http://gerrit.cloudera.org:8080/19048 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I952377ed14eba26e3774e7776eb81a95d1d8e76f Gerrit-Change-Number: 19048 Gerrit-PatchSet: 5 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Peter Rozsa Gerrit-Comment-Date: Thu, 29 Sep 2022 10:41:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11624: Bump Impyla dependency to 0.18.0
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/18456 ) Change subject: IMPALA-11624: Bump Impyla dependency to 0.18.0 .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/18456/6/bin/impala-config.sh File bin/impala-config.sh: http://gerrit.cloudera.org:8080/#/c/18456/6/bin/impala-config.sh@179 PS6, Line 179: shell/ext-py Hi Csaba. Just wondering, what will happen with shell/ext-py/thrift-0.14.2 ? Do we need to replace it with source code from thrift-0.16.0 ? -- To view, visit http://gerrit.cloudera.org:8080/18456 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7265558b0e07959c606cba73cd251c3edfcb3ed5 Gerrit-Change-Number: 18456 Gerrit-PatchSet: 6 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 29 Sep 2022 10:04:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11624: Bump Impyla dependency to 0.18.0
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18456 ) Change subject: IMPALA-11624: Bump Impyla dependency to 0.18.0 .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/18456/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18456/5//COMMIT_MSG@7 PS5, Line 7: IMPALA-11624: Bump Impyla dependency to 0.18.0 > nit: Should you add Jira number? Created a Jira about this. -- To view, visit http://gerrit.cloudera.org:8080/18456 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7265558b0e07959c606cba73cd251c3edfcb3ed5 Gerrit-Change-Number: 18456 Gerrit-PatchSet: 6 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 29 Sep 2022 09:17:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11624: Bump Impyla dependency to 0.18.0
Hello Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18456 to look at the new patch set (#6). Change subject: IMPALA-11624: Bump Impyla dependency to 0.18.0 .. IMPALA-11624: Bump Impyla dependency to 0.18.0 IMPALA_THRIFT_PY_VERSION is also bumped to 0.16.0p3. As 0.16.0p3 Thrift does not contain Python related patches and Impyla 0.18.0 depends on Thrift 0.16.0, now we are consistently using Thrift 0.16.0 in all Python code. Change-Id: I7265558b0e07959c606cba73cd251c3edfcb3ed5 --- M bin/impala-config.sh M infra/python/deps/requirements.txt M shell/packaging/requirements.txt 3 files changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/18456/6 -- To view, visit http://gerrit.cloudera.org:8080/18456 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7265558b0e07959c606cba73cd251c3edfcb3ed5 Gerrit-Change-Number: 18456 Gerrit-PatchSet: 6 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/19051 ) Change subject: IMPALA-11621: Remove hiveserver2.pid when shutting down HiveServer2 .. Patch Set 1: Code-Review+2 (1 comment) I think I have bumped into the issue in https://gerrit.cloudera.org/#/c/18994/ http://gerrit.cloudera.org:8080/#/c/19051/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19051/1//COMMIT_MSG@23 PS1, Line 23: remove typo -- To view, visit http://gerrit.cloudera.org:8080/19051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I813626d06829a86854c6d2c1715f0c5f5109836d Gerrit-Change-Number: 19051 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 29 Sep 2022 08:19:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10902: In the case of outer join connections, skip unnecessary BE check
Xianqing He has posted comments on this change. ( http://gerrit.cloudera.org:8080/17830 ) Change subject: IMPALA-10902: In the case of outer join connections, skip unnecessary BE check .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/17830/4/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/17830/4/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@3737 PS4, Line 3737: } else if (expr instanceof BetweenPredicate) { Just wonder whether exists this case since BetweenPredicate rewrites into an equivalent conjunctive/disjunctive CompoundPredicate in analyzer -- To view, visit http://gerrit.cloudera.org:8080/17830 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6f4db9b5aa00ee5117118696c58c07ffaaf0b66 Gerrit-Change-Number: 17830 Gerrit-PatchSet: 4 Gerrit-Owner: pengdou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Thu, 29 Sep 2022 08:18:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11612: fix ORDER BY expression rewrite bug
Hello Quanlong Huang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19049 to look at the new patch set (#3). Change subject: IMPALA-11612: fix ORDER BY expression rewrite bug .. IMPALA-11612: fix ORDER BY expression rewrite bug When the query SQL is overwritten, both the selectList and groupByList fields are preprocessed except for the orderByList fields, the result of rewoten orderByList is not same as selectList and groupByList. It occur exception in method SelectStmt.verifyAggregation. So we should add preprocessing before the orderByList be rewrote. Change-Id: I438c61c647e3f99665c94ea15e3a9d98b9ff9912 --- M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java 2 files changed, 20 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/19049/3 -- To view, visit http://gerrit.cloudera.org:8080/19049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I438c61c647e3f99665c94ea15e3a9d98b9ff9912 Gerrit-Change-Number: 19049 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] Prototype to provide an option to install Java 11
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19031 ) Change subject: Prototype to provide an option to install Java 11 .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11483/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc1dbd3f6a2279840218dc1da2b60077e211a328 Gerrit-Change-Number: 19031 Gerrit-PatchSet: 10 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 29 Sep 2022 06:12:12 + Gerrit-HasComments: No