[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 23: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10041/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 23 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 25 Jan 2022 21:54:30 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 23: Also call openTransaction() after the analysis phase for the 1st executor group and keep the transaction info. Reuse the transaction info in the following compilations. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 23 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 25 Jan 2022 21:28:51 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 23: Fix the following: Class 2 (fixed, 1/25/2022) custom_cluster.test_coordinators.TestCoordinators.test_exclusive_coordinator_plan (from pytest) custom_cluster.test_coordinators.TestCoordinators.test_num_expected_executors_flag (from pytest) Class 9 (fixed, 1/25/2022) org.apache.impala.planner.ClusterSizeTest.testChangeClusterSize Stacktrace java.lang.AssertionError at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at org.apache.impala.planner.ClusterSizeTest.testChangeClusterSize(ClusterSizeTest.java:93) -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 23 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 25 Jan 2022 21:27:00 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#23). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group in the set is associated with a distinct number of nodes and a threshold for estimated memory per host in bytes. In the patch, a query of type EXPLAIN, QUERY, LOAD or DML can be compiled more than once. In each attempt, per host memory is estimated and compared with the threshold of an executor group. If the estimated memory is no more than the threshold, the iteration process terminates and the final plan is determined. The executor group with the threshold is selected to run against the query. A new query option 'enable_replan', default to 1 (enabled), is added. It can be set to 0 to disable this patch and to generate the distributed plan for the default executor group. To avoid long compilation time, the following enhancement is enabled. 1. Authorization is performed only for the 1st compilation; 2. Meta-data is fetched into a StmtTableCache in 1st compilation and reused in subsequent compilations; Testing: 1. The planner test passes in an artificial two-executor group setup with the last group being identical to a 3-node default group. To do: 1. BE testing of auto-scaled plans. Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M fe/src/test/java/org/apache/impala/planner/ClusterSizeTest.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test M tests/custom_cluster/test_coordinators.py M tests/custom_cluster/test_executor_groups.py 38 files changed, 617 insertions(+), 208 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/23 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 23 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 19: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10035/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 19 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 25 Jan 2022 01:00:07 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 19: Rebase and fix the following issue. Class 3 (fixed) custom_cluster.test_executor_groups.TestExecutorGroups.test_join_strategy_single_executor (from pytest) assert ':EXCHANGE [BROADCAST]' in 'Summary: Returned 26 rows\nSuccess: True\nTook: 0.026978969574(s)\nData:\nMax Per-Host Resource Reservation: Memory=8... HDFS partitions=24/24 files=24 size=478.45KB\n runtime filters: RF000 -> a.id\n row-size=89B cardinality=7.30K\n' + where 'Summary: Returned 26 rows\nSuccess: True\nTook: 0.026978969574(s)\nData:\nMax Per-Host Resource Reservation: Memory=8... HDFS partitions=24/24 files=24 size=478.45KB\n runtime filters: RF000 -> a.id\n row-size=89B cardinality=7.30K\n' = str() -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 19 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 25 Jan 2022 00:35:22 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#19). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group in the set is associated with a distinct number of nodes and a threshold for estimated memory per host in bytes. In the patch, a query of type EXPLAIN, QUERY, LOAD or DML can be compiled more than once. In each attempt, per host memory is estimated and compared with the threshold of an executor group. If the estimated memory is no more than the threshold, the iteration process terminates and the final plan is determined. The executor group with the threshold is selected to run against the query. A new query option 'enable_replan', default to 1 (enabled), is added. It can be set to 0 to disable this patch and to generate the distributed plan for the default executor group. To avoid long compilation time, the following enhancement is enabled. 1. Authorization is performed only for the 1st compilation; 2. Meta-data is fetched into a StmtTableCache in 1st compilation and reused in subsequent compilations; Testing: 1. The planner test passes in an artificial two-executor group setup with the last group being identical to a 3-node default group. To do: 1. BE testing of auto-scaled plans. Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test M tests/custom_cluster/test_executor_groups.py 36 files changed, 700 insertions(+), 294 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/19 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 19 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 17: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10034/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 17 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 24 Jan 2022 18:33:11 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 17: class 2 should class 4 in the above message. Class 4: custom_cluster.test_kudu.TestKuduTransaction.test_kudu_txn_not_implemented (from pytest) custom_cluster.test_kudu.TestKuduTransaction.test_kudu_txn_abort_dup_key (from pytest) custom_cluster.test_kudu.TestKuduTransaction.test_kudu_txn_ctas (from pytest) custom_cluster.test_kudu.TestKuduTransaction.test_kudu_txn_abort_partition_lock (from pytest) custom_cluster/test_kudu.py:413: in test_kudu_txn_not_implemented assert False, "query was expected to fail" E AssertionError: query was expected to fail E assert False -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 17 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 24 Jan 2022 18:12:44 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 17: Fixes the following two classes of issues seen with core tests. Class 2: custom_cluster.test_coordinators.TestCoordinators.test_exclusive_coordinator_plan (from pytest) custom_cluster.test_coordinators.TestCoordinators.test_num_expected_executors_flag (from pytest) Class 8 (fixed) Test Results observability.test_log_fragments TestLogFragments test_log_fragments E AssertionError: Expected 1 lines in file /data0/jenkins/workspace/impala-private-basic-parameterized/repos/Impala/logs/ee_tests/impalad.impala-ec2-centos74-m5-4xlarge-ondemand-0e46.vpc.cloudera.com.jenkins.log.INFO.20220119-204602.15077 matching regex 'e74ca65276d6a874:d3f0108b] Analysis and authorization finished.', but found 2 lines. Last line was: E 134: minmax_filter_partition_col -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 17 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 24 Jan 2022 18:12:00 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#17). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group in the set is associated with a distinct number of nodes and a threshold for estimated memory per host in bytes. In the patch, a query of type EXPLAIN, QUERY, LOAD or DML can be compiled more than once. In each attempt, per host memory is estimated and compared with the threshold of an executor group. If the estimated memory is no more than the threshold, the iteration process terminates and the final plan is determined. The executor group with the threshold is selected to run against the query. A new query option 'enable_replan', default to 1 (enabled), is added. It can be set to 0 to disable this patch and to generate the distributed plan for the default executor group. To avoid long compilation time, the following enhancement is enabled. 1. Authorization is performed only for the 1st compilation; 2. Meta-data is fetched into a StmtTableCache in 1st compilation and reused in subsequent compilations; Testing: 1. The planner test passes in an artificial two-executor group setup with the last group being identical to a 3-node default group. To do: 1. BE testing of auto-scaled plans. Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test 35 files changed, 677 insertions(+), 294 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/17 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 17 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 16: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10032/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 16 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 21 Jan 2022 22:22:54 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/18143 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
..
Patch Set 16:
Fix the failure when executing SQL create table insertonly_acid_ctas31
tblproperties("transactional"="true", "transactional_properties"="insert_only")
as select 1, 2, "name", due to writeId is not properly set in HdfsTableSink in
the auto-scaled plan.
--
To view, visit http://gerrit.cloudera.org:8080/18143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24
Gerrit-Change-Number: 18143
Gerrit-PatchSet: 16
Gerrit-Owner: Qifan Chen
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Fri, 21 Jan 2022 22:01:05 +
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group in the set is associated with a distinct number of nodes and a threshold for estimated memory per host in bytes. In the patch, a query of type EXPLAIN, QUERY, LOAD or DML can be compiled more than once. In each attempt, per host memory is estimated and compared with the threshold of an executor group. If the estimated memory is no more than the threshold, the iteration process terminates and the final plan is determined. The executor group with the threshold is selected to run against the query. A new query option 'enable_replan', default to 1 (enabled), is added. It can be set to 0 to disable this patch and to generate the distributed plan for the default executor group. To avoid long compilation time, the following enhancement is enabled. 1. Authorization is performed only for the 1st compilation; 2. Meta-data is fetched into a StmtTableCache in 1st compilation and reused in subsequent compilations; Testing: 1. The planner test passes in an artificial two-executor group setup with the last group being identical to a 3-node default group. To do: 1. BE testing of auto-scaled plans. Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test 35 files changed, 633 insertions(+), 255 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/16 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 16 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10025/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 15 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 20 Jan 2022 23:10:30 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 15: Resolve class 1 type of issues seen with core test: ImpalaBeeswaxException: ImpalaBeeswaxException: Query aborted:IllegalStateException: Invalid transaction id -1 for generating insert events on table test_self_events_f09a62c5.acid_no_part_tbl_zimla custom_cluster.test_events_custom_configs.TestEventProcessingCustomConfigs.test_event_batching (from pytest) -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 15 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 20 Jan 2022 22:48:05 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group in the set is associated with a distinct number of nodes and a threshold for estimated memory per host in bytes. In the patch, a query of type EXPLAIN, QUERY, LOAD or DML can be compiled more than once. In each attempt, per host memory is estimated and compared with the threshold of an executor group. If the estimated memory is no more than the threshold, the iteration process terminates and the final plan is determined. The executor group with the threshold is selected to run against the query. A new query option 'enable_replan', default to 1 (enabled), is added. It can be set to 0 to disable this patch and to generate the distributed plan for the default executor group. To avoid long compilation time, the following enhancement is enabled. 1. Authorization is performed only for the 1st compilation; 2. Meta-data is fetched into a StmtTableCache in 1st compilation and reused in subsequent compilations; Testing: 1. The planner test passes in an artificial two-executor group setup with the last group being identical to a 3-node default group. To do: 1. BE testing of auto-scaled plans. Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test 35 files changed, 609 insertions(+), 255 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/15 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 15 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10023/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 14 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 20 Jan 2022 18:26:51 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group in the set is associated with a distinct number of nodes and a threshold for estimated memory per host in bytes. In the patch, a query of type EXPLAIN, QUERY, LOAD or DML can be compiled more than once. In each attempt, per host memory is estimated and compared with the threshold of an executor group. If the estimated memory is no more than the threshold, the iteration process terminates and the final plan is determined. The executor group with the threshold is selected to run against the query. A new query option 'enable_replan', default to 1 (enabled), is added. It can be set to 0 to disable this patch and to generate the distributed plan for the default executor group. To avoid long compilation time, the following enhancement is enabled. 1. Authorization is performed only for the 1st compilation; 2. Meta-data is fetched into a StmtTableCache in 1st compilation and reused in subsequent compilations; Testing: 1. The planner test passes in an artificial two-executor group setup with the last group being identical to a 3-node default group. To do: 1. BE testing of auto-scaled plans. Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test 35 files changed, 606 insertions(+), 252 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/14 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 14 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10020/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 13 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 20 Jan 2022 02:26:46 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10019/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 12 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 20 Jan 2022 02:17:39 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 13: Small change to a comment in ImpalaService.thrift. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 13 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 20 Jan 2022 02:02:33 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group is associated with a distinct number of nodes and a threshold for estimated memory per host (in bytes). In the patch, a query of type EXPLAIN, QUERY, LOAD or DML can be compiled more than once. In each attempt, per host memory is estimated and compared with the thread of an executor group. If the estimated memory is no more than the threshold, auto-scaling process terminates and the plan is selected. The executor group with the threshold is selected. A new query option 'enable_replan', default to 1 (enabled), is added. It can be set to 0 to disable this patch and to generate the distributed plan for the default executor group. The testing of the quality of the replanning is done by verifying that the planner tests generate identical plans in a two-executor group configuration with the last one being identical to the existing default executor environment. To do: 1. BE testing of auto-scaled plans. Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test 35 files changed, 606 insertions(+), 252 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/13 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 13 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 12: Modified 'enable_replan' query option to take 0, 1 and 2 as follows. This is to facilitate BE testing. 0: turn off; 1: turn on; 2: turn on and treat a default group as [3, 3, regular, 64MB] [3, 3, large, 8PB]. Rewrote setupThresholdsForGroupSets() that will always return a new list of executor groups to apply. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 12 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 20 Jan 2022 01:58:36 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group is associated with a distinct number of nodes and a threshold for estimated memory per host (in bytes). In the patch, a query of type EXPLAIN, QUERY, LOAD or DML can be compiled more than once. In each attempt, per host memory is estimated and compared with the thread of an executor group. If the estimated memory is no more than the threshold, auto-scaling process terminates and the plan is selected. The executor group with the threshold is selected. A new query option 'enable_replan', default to 1 (enabled), is added. It can be set to 0 to disable this patch and to generate the distributed plan for the default executor group. The testing of the quality of the replanning is done by verifying that the planner tests generate identical plans in a two-executor group configuration with the last one being identical to the existing default executor environment. To do: 1. BE testing of auto-scaled plans. Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test 35 files changed, 606 insertions(+), 252 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/12 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 12 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10018/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 11 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 19 Jan 2022 17:53:48 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 11: The above work is done in patch 10. All planner tests pass. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 11 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 19 Jan 2022 17:32:22 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group is associated with a distinct number of nodes and a threshold for estimated memory per host (in bytes). In the patch, a query of type EXPLAIN, QUERY, LOAD or DML can be compiled more than once. In each attempt, per host memory is estimated and compared with the thread of an executor group. If the estimated memory is no more than the threshold, auto-scaling process terminates and the plan is selected. The executor group with the threshold is selected. A new query option 'enable_replan', default to true, is added. It can be set to false to disable this patch and to generate the distributed plan for the default executor group. The testing of the quality of the replanning is done by verifying that the planner tests generate identical plans in a two-executor group configuration with the last one being identical to the existing default executor environment. To do: 1. BE testing of auto-scaled plans. Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test 33 files changed, 569 insertions(+), 250 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/11 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 11 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10010/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 9 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 13 Jan 2022 21:11:49 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 9: Set the name prefix of the selected executor group in the returned query options and query context. Both are needed for BE to execute the query plan. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 9 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 13 Jan 2022 20:47:07 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group is associated with distinct number of nodes and a threshold for estimated memory per host (in bytes). In the patch, attempts are made to estimate per host memory from a complete compilation of a query, select a suitable executor group to use and generate the distributed plan for it. A new query option 'enable_replan', default to true, is added. It can be set to false to disable this patch and to generate the distributed plan for the default executor group. The testing of the quality of the replanning is done by verifying that the planner tests generate identical plans in a two-executor group configuration with the last one being identical to the existing default executor environment. To do: 1. Enable auto-scaling for DMLs; Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test 33 files changed, 440 insertions(+), 181 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/9 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 9 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10009/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 8 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 13 Jan 2022 18:27:34 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 8: Handles the following 1. Reuse StmtTableCache across multiple auto-scale compilations; 2. Use the original version of query options in each auto-scale compilation; 3. Allows only QUERY or LOAD stmt to be auto-scale (to do: DML and CTAS as they involve transactions/locking); 4. Pass planner tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 8 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 13 Jan 2022 18:08:18 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group is associated with distinct number of nodes and a threshold for estimated memory per host (in bytes). In the patch, attempts are made to estimate per host memory from a complete compilation of a query, select a suitable executor group to use and generate the distributed plan for it. A new query option 'enable_replan', default to true, is added. It can be set to false to disable this patch and to generate the distributed plan for the default executor group. The testing of the quality of the replanning is done by verifying that the planner tests generate identical plans in a two-executor group configuration with the last one being identical to the existing default executor environment. Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test 33 files changed, 432 insertions(+), 181 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/8 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 8 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10006/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 6 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 12 Jan 2022 23:10:52 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 6: Fix build error and address the bug that the threshold for regular group in planner test was set to 8PB. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 6 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 12 Jan 2022 22:49:57 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group is associated with distinct number of nodes and a threshold for estimated memory per host (in bytes). In the patch, attempts are made to estimate per host memory from a complete compilation of a query, select a suitable executor group to use and generate the distributed plan for it. A new query option 'enable_replan', default to true, is added. It can be set to false to disable this patch and to generate the distributed plan for the default executor group. The testing of the quality of the replanning is done by verifying that the planner tests generate identical plans in a two-executor group configuration with the last one being identical to the existing default executor environment. Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test 33 files changed, 401 insertions(+), 180 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/6 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 6 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/10005/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 5 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 12 Jan 2022 19:30:40 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 5: 1st draft of V1 approach that iterates over executor groups and compile the entire query in each iteration. 1. Handle query option service_pool; 2. Handle query option enable_replan; 3. Sort executor groups by thresholds; 4. Fix a bug in computing resource profile for UnionNode 5. Pass planner test -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 5 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Wed, 12 Jan 2022 19:19:13 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18143 Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 .. [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group is associated with distinct number of nodes and a threshold for estimated memory per host (in bytes). In the patch, attempts are made to estimate per host memory from a complete compilation of a query, select a suitable executor group to use and generate the distributed plan for it. A new query option 'enable_replan', default to true, is added. It can be set to false to disable this patch and to generate the distributed plan for the default executor group. The testing of the quality of the replanning is done by verifying that the planner tests generate identical plans in a two-executor group configuration with the last one being identical to the existing default executor environment. Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test 33 files changed, 399 insertions(+), 180 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/5 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 5 Gerrit-Owner: Qifan Chen
