[Impala-ASF-CR] IMPALA-11051: Add support for 'void' Iceberg partition transform
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18102 ) Change subject: IMPALA-11051: Add support for 'void' Iceberg partition transform .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7744/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icbe11d56cdeb82aaadedfdb3ad61dd7cc4c2f4d0 Gerrit-Change-Number: 18102 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 13 Jan 2022 11:09:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11051: Add support for 'void' Iceberg partition transform
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18102 ) Change subject: IMPALA-11051: Add support for 'void' Iceberg partition transform .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icbe11d56cdeb82aaadedfdb3ad61dd7cc4c2f4d0 Gerrit-Change-Number: 18102 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 13 Jan 2022 11:09:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11051: Add support for 'void' Iceberg partition transform
Hello Tamas Mate, wangsheng, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18102 to look at the new patch set (#3). Change subject: IMPALA-11051: Add support for 'void' Iceberg partition transform .. IMPALA-11051: Add support for 'void' Iceberg partition transform Iceberg recently added a new partition transform called 'void': https://iceberg.apache.org/#spec/#partition-transforms This patch adds support for this transform. When the user wants to drop a column from the partition spec, the VOID transform should be used instead of just omitting the column. Simply omitting the column might cause problems when the metadata table is being queried (currently only supported by other engines). Testing * added SHOW CREATE TABLE test * added e2e test Change-Id: Icbe11d56cdeb82aaadedfdb3ad61dd7cc4c2f4d0 --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-partitioned-insert.test M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test 8 files changed, 155 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/18102/3 -- To view, visit http://gerrit.cloudera.org:8080/18102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icbe11d56cdeb82aaadedfdb3ad61dd7cc4c2f4d0 Gerrit-Change-Number: 18102 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-11051: Add support for 'void' Iceberg partition transform
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18102 ) Change subject: IMPALA-11051: Add support for 'void' Iceberg partition transform .. Patch Set 3: Code-Review+2 Carry +2 -- To view, visit http://gerrit.cloudera.org:8080/18102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icbe11d56cdeb82aaadedfdb3ad61dd7cc4c2f4d0 Gerrit-Change-Number: 18102 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 13 Jan 2022 11:08:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11051: Add support for 'void' Iceberg partition transform
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18102 ) Change subject: IMPALA-11051: Add support for 'void' Iceberg partition transform .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10008/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icbe11d56cdeb82aaadedfdb3ad61dd7cc4c2f4d0 Gerrit-Change-Number: 18102 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 13 Jan 2022 11:30:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11072: Deflake TestSpillingDebugActionDimensions.test spilling
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18138 ) Change subject: IMPALA-11072: Deflake TestSpillingDebugActionDimensions.test_spilling .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7745/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18138 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie8802505e0dcae1be5e855107436805bd10e0077 Gerrit-Change-Number: 18138 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 13 Jan 2022 11:59:03 + Gerrit-HasComments: No
[Impala-ASF-CR] [DO NOT MERGE]: Test patch to check unit test failures
Sourabh Goyal has abandoned this change. ( http://gerrit.cloudera.org:8080/18122 ) Change subject: [DO NOT MERGE]: Test patch to check unit test failures .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/18122 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I69566d37d437befb10e2555e81c4fafe1f38151f Gerrit-Change-Number: 18122 Gerrit-PatchSet: 1 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11070: Create TestSection class.
Steve Carlin has abandoned this change. ( http://gerrit.cloudera.org:8080/18125 ) Change subject: IMPALA-11070: Create TestSection class. .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/18125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I3fed7ef64ac263b49b5b6e4a237a296092e4dcb8 Gerrit-Change-Number: 18125 Gerrit-PatchSet: 4 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Steve Carlin
[Impala-ASF-CR] IMPALA-11051: Add support for 'void' Iceberg partition transform
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18102 ) Change subject: IMPALA-11051: Add support for 'void' Iceberg partition transform .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7744/ -- To view, visit http://gerrit.cloudera.org:8080/18102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icbe11d56cdeb82aaadedfdb3ad61dd7cc4c2f4d0 Gerrit-Change-Number: 18102 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 13 Jan 2022 17:46:23 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group is associated with distinct number of nodes and a threshold for estimated memory per host (in bytes). In the patch, attempts are made to estimate per host memory from a complete compilation of a query, select a suitable executor group to use and generate the distributed plan for it. A new query option 'enable_replan', default to true, is added. It can be set to false to disable this patch and to generate the distributed plan for the default executor group. The testing of the quality of the replanning is done by verifying that the planner tests generate identical plans in a two-executor group configuration with the last one being identical to the existing default executor environment. Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test 33 files changed, 432 insertions(+), 181 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/8 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 8 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 8: Handles the following 1. Reuse StmtTableCache across multiple auto-scale compilations; 2. Use the original version of query options in each auto-scale compilation; 3. Allows only QUERY or LOAD stmt to be auto-scale (to do: DML and CTAS as they involve transactions/locking); 4. Pass planner tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 8 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 13 Jan 2022 18:08:18 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10009/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 8 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 13 Jan 2022 18:27:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11072: Deflake TestSpillingDebugActionDimensions.test spilling
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18138 ) Change subject: IMPALA-11072: Deflake TestSpillingDebugActionDimensions.test_spilling .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18138 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie8802505e0dcae1be5e855107436805bd10e0077 Gerrit-Change-Number: 18138 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Thu, 13 Jan 2022 18:45:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11072: Deflake TestSpillingDebugActionDimensions.test spilling
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18138 ) Change subject: IMPALA-11072: Deflake TestSpillingDebugActionDimensions.test_spilling .. IMPALA-11072: Deflake TestSpillingDebugActionDimensions.test_spilling The first test case in TestSpillingDebugActionDimensions.test_spilling has been flaky for not spilling any partitions in its hash join node. This patch fixes the flakiness by reducing the buffer_pool_limit from 215 MB to 110 MB, which is around double of the query Per Host Min Memory Reservation. Testing: - Manually run the first test case of TestSpillingDebugActionDimensions.test_spilling. Verify that both of the hash joins are spilling and the test pass. Change-Id: Ie8802505e0dcae1be5e855107436805bd10e0077 Reviewed-on: http://gerrit.cloudera.org:8080/18138 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M testdata/workloads/functional-query/queries/QueryTest/spilling.test 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18138 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie8802505e0dcae1be5e855107436805bd10e0077 Gerrit-Change-Number: 18138 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-11051: Add support for 'void' Iceberg partition transform
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18102 ) Change subject: IMPALA-11051: Add support for 'void' Iceberg partition transform .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7746/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icbe11d56cdeb82aaadedfdb3ad61dd7cc4c2f4d0 Gerrit-Change-Number: 18102 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 13 Jan 2022 19:45:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11051: Add support for 'void' Iceberg partition transform
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18102 ) Change subject: IMPALA-11051: Add support for 'void' Iceberg partition transform .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icbe11d56cdeb82aaadedfdb3ad61dd7cc4c2f4d0 Gerrit-Change-Number: 18102 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 13 Jan 2022 19:45:29 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group is associated with distinct number of nodes and a threshold for estimated memory per host (in bytes). In the patch, attempts are made to estimate per host memory from a complete compilation of a query, select a suitable executor group to use and generate the distributed plan for it. A new query option 'enable_replan', default to true, is added. It can be set to false to disable this patch and to generate the distributed plan for the default executor group. The testing of the quality of the replanning is done by verifying that the planner tests generate identical plans in a two-executor group configuration with the last one being identical to the existing default executor environment. To do: 1. Enable auto-scaling for DMLs; Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test 33 files changed, 440 insertions(+), 181 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/9 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 9 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 9: Set the name prefix of the selected executor group in the returned query options and query context. Both are needed for BE to execute the query plan. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 9 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 13 Jan 2022 20:47:07 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10010/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 9 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 13 Jan 2022 21:11:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11063: Add metrics to expose state of each executor group set
Qifan Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/18142 )
Change subject: IMPALA-11063: Add metrics to expose state of each executor
group set
..
Patch Set 2: Code-Review+1
(2 comments)
Thanks!
http://gerrit.cloudera.org:8080/#/c/18142/1/be/src/scheduling/cluster-membership-mgr.cc
File be/src/scheduling/cluster-membership-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/18142/1/be/src/scheduling/cluster-membership-mgr.cc@89
PS1, Line 89: MetricGroup* metric_grp =
metrics->GetOrCreateChildGroup("cluster-membership");
: aggregated_group_set_metrics_.total_live_executor_groups_ =
: metric_grp->AddCounter(LIVE_EXEC_GROUP_KEY, 0);
: aggregated_group_set_metrics_.total_healthy_executor_groups_ =
: metric_grp->AddCounter(HEALTHY_EXEC_GROUP_KEY, 0);
> The keys and inputs to AddCounter() are different for 'overall' vs 'per_gro
Done
http://gerrit.cloudera.org:8080/#/c/18142/1/be/src/scheduling/cluster-membership-mgr.cc@606
PS1, Line 606: ngPiece name(grou
> group.NumHosts() > 0 is the only simple check we need unlike IsHealthy() wh
Done
--
To view, visit http://gerrit.cloudera.org:8080/18142
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib39f940de830ef6302785aee30eeb847fa5deeba
Gerrit-Change-Number: 18142
Gerrit-PatchSet: 2
Gerrit-Owner: Bikramjeet Vig
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Fri, 14 Jan 2022 00:09:02 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11051: Add support for 'void' Iceberg partition transform
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18102 ) Change subject: IMPALA-11051: Add support for 'void' Iceberg partition transform .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7746/ -- To view, visit http://gerrit.cloudera.org:8080/18102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icbe11d56cdeb82aaadedfdb3ad61dd7cc4c2f4d0 Gerrit-Change-Number: 18102 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 14 Jan 2022 02:16:41 + Gerrit-HasComments: No
