[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has abandoned this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1 .. Abandoned This version is the draft version to https://gerrit.cloudera.org/#/c/18178/. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 29 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 29: Bikramjeet, the latest version of patch is at https://gerrit.cloudera.org/#/c/18178/ -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 29 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 25 Feb 2022 05:30:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 29: (20 comments) http://gerrit.cloudera.org:8080/#/c/18143/29//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18143/29//COMMIT_MSG@9 PS29, Line 9: a set of executor : groups nit: multiple executor group sets. Change mentions of executor group to executor group sets where applicable in the commit message. This is to stay consistent with the distinction between executor groups vs group sets http://gerrit.cloudera.org:8080/#/c/18143/29//COMMIT_MSG@25 PS29, Line 25: To facilitate testing, the patch imposes an artificial two executor : group setup in FE. This setup is enabled when 'enable_replan' is set : to 3 (testing) or RuntimeEnv.INSTANCE.isTestEnv() is true as in most : frontend tests. The artificial two executor groups are configured as : follows. Does this mean all tests are run in this configuration? All planner tests and end2end pytest are currently based on a minicluster or a custom cluster to emulate an actual cluster. Please see my comment in Frontend.java about how we can ensure re-planning code-path gets tested while ensuring the actual group set gets used. http://gerrit.cloudera.org:8080/#/c/18143/29/common/thrift/Frontend.thrift File common/thrift/Frontend.thrift: http://gerrit.cloudera.org:8080/#/c/18143/29/common/thrift/Frontend.thrift@729 PS29, Line 729: The optional threshold for queries to run in an executor group nit: can you provide more context here as to what this threshold signifies, like which resource is represents and how it helps the planner make decisions. http://gerrit.cloudera.org:8080/#/c/18143/29/common/thrift/Query.thrift File common/thrift/Query.thrift: http://gerrit.cloudera.org:8080/#/c/18143/29/common/thrift/Query.thrift@585 PS29, Line 585: TEnableReplanMode.TESTING shouldn't this be set to ON by default as per the comment? http://gerrit.cloudera.org:8080/#/c/18143/29/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/18143/29/fe/src/main/java/org/apache/impala/service/Frontend.java@267 PS29, Line 267: /// : // BEGIN: Data members used by auto-scaling nit: would it make sense to put this inside a separate State class? http://gerrit.cloudera.org:8080/#/c/18143/29/fe/src/main/java/org/apache/impala/service/Frontend.java@286 PS29, Line 286: // The stmt table cache to be used by all iterations of auto-scaling compilation. nit: mention when it is set and when can it be reset http://gerrit.cloudera.org:8080/#/c/18143/29/fe/src/main/java/org/apache/impala/service/Frontend.java@339 PS29, Line 339: auto-scaling nit: not sure auto-scaling is the right term here, since we are not scaling anything but rather assigning a query to the right group. Open to suggestions for more semantically appropriate terms. http://gerrit.cloudera.org:8080/#/c/18143/29/fe/src/main/java/org/apache/impala/service/Frontend.java@1763 PS29, Line 1763: setupThresholdsForGroupSets nit: add a comment about what it does http://gerrit.cloudera.org:8080/#/c/18143/29/fe/src/main/java/org/apache/impala/service/Frontend.java@1780 PS29, Line 1780: enable_replan == TEnableReplanMode.TESTING nit: also might be worth logging this while planning http://gerrit.cloudera.org:8080/#/c/18143/29/fe/src/main/java/org/apache/impala/service/Frontend.java@1787 PS29, Line 1787: TExecutorGroupSet l = new TExecutorGroupSet(3, 3, "large"); : l.setThreshold(Long.MAX_VALUE); : result.add(l); if we want to emulate a 2 exec group set configuration, what if we only add a dummy executor group set which will always be rejected (i think its the 'regular' one in this case) and the other one as the actual group set 'e = executorGroupSets.get(0)'. Otherwise this can get very confusing if the someone is running a test (and is not aware of this) on a different configuration but the plan always gets created for a 3 node group set. http://gerrit.cloudera.org:8080/#/c/18143/29/fe/src/main/java/org/apache/impala/service/Frontend.java@1792 PS29, Line 1792: executorGroupSets.get(0) nit: can just use 'e' here too. http://gerrit.cloudera.org:8080/#/c/18143/29/fe/src/main/java/org/apache/impala/service/Frontend.java@1801 PS29, Line 1801: request_pool if we use query_exec_request.query_ctx.request_pool instead of queryOptions.getRequest_pool(), that should have the resolved request pool that always has the "root." prefix http://gerrit.cloudera.org:8080/#/c/18143/29/fe/src/main/java/org/apache/impala/service/Frontend.java@1802 PS29, Line 1802: if (request_pool !=
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 29: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10064/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 29 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 28 Jan 2022 21:22:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 29: Reverse changes on UnionNode.java and its impact to tpcds queries with unions. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 29 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 28 Jan 2022 21:00:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#29). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1 .. IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group in the set is associated with a distinct number of nodes and a threshold for estimated memory per host in bytes that can be denoted as [:<#nodes>, ]. In the patch, a query of type EXPLAIN, QUERY or DML can be compiled more than once. In each attempt, per host memory is estimated and compared with the threshold of an executor group. If the estimated memory is no more than the threshold, the iteration process terminates and the final plan is determined. The executor group with the threshold is selected to run against the query. A new query option 'enable_replan', default to 1 (enabled), is added. It can be set to 0 to disable this patch and to generate the distributed plan for the default executor group. To facilitate testing, the patch imposes an artificial two executor group setup in FE. This setup is enabled when 'enable_replan' is set to 3 (testing) or RuntimeEnv.INSTANCE.isTestEnv() is true as in most frontend tests. The artificial two executor groups are configured as follows. 1. [regular:3, 64MB] 2. [large:3, 8PB] To avoid long compilation time, the following enhancement is enabled. Note 1) and 2) can be disabled when some meta-data change is detected. 1. Authorization is performed only for the 1st compilation; 2. The relevant meta-data is fetched into a StmtTableCache in 1st compilation and reused in subsequent compilations; 3. openTransaction() is called for transactional queries in 1st compilation and the saved transactional info is used in subsequent compilations. Similar logic is applied to Kudu transactional queries. Testing: 1. Ran core tests Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M fe/src/test/java/org/apache/impala/planner/ClusterSizeTest.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_coordinators.py M tests/custom_cluster/test_executor_groups.py M tests/query_test/test_observability.py 21 files changed, 499 insertions(+), 74 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/29 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 29 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 28: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10063/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 28 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 28 Jan 2022 19:12:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/18143 )
Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1
..
Patch Set 28:
C14 (fixed, by transfer userHasProfileAccess flag from analysisResult of 1st
iteration to the final one).
authorization.test_ranger.TestRanger.test_profile_protection[protocol: beeswax
| exec_option: {'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
text/none] (from pytest)
Failing for the past 1 build (Since Failed#332 )
Took 14 sec.
add description
Error Message
AssertionError: No failure encountered for query
--
To view, visit http://gerrit.cloudera.org:8080/18143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24
Gerrit-Change-Number: 18143
Gerrit-PatchSet: 28
Gerrit-Owner: Qifan Chen
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Fri, 28 Jan 2022 18:50:22 +
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#28). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1 .. IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group in the set is associated with a distinct number of nodes and a threshold for estimated memory per host in bytes that can be denoted as [:<#nodes>, ]. In the patch, a query of type EXPLAIN, QUERY or DML can be compiled more than once. In each attempt, per host memory is estimated and compared with the threshold of an executor group. If the estimated memory is no more than the threshold, the iteration process terminates and the final plan is determined. The executor group with the threshold is selected to run against the query. A new query option 'enable_replan', default to 1 (enabled), is added. It can be set to 0 to disable this patch and to generate the distributed plan for the default executor group. To facilitate testing, the patch imposes an artificial two executor group setup in FE. This setup is enabled when 'enable_replan' is set to 3 (testing) or RuntimeEnv.INSTANCE.isTestEnv() is true as in most frontend tests. The artificial two executor groups are configured as follows. 1. [regular:3, 64MB] 2. [large:3, 8PB] To avoid long compilation time, the following enhancement is enabled. Note 1) and 2) can be disabled when some meta-data change is detected. 1. Authorization is performed only for the 1st compilation; 2. The relevant meta-data is fetched into a StmtTableCache in 1st compilation and reused in subsequent compilations; 3. openTransaction() is called for transactional queries in 1st compilation and the saved transactional info is used in subsequent compilations. Similar logic is applied to Kudu transactional queries. Testing: 1. Ran core tests Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M fe/src/test/java/org/apache/impala/planner/ClusterSizeTest.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_coordinators.py M tests/custom_cluster/test_executor_groups.py M tests/query_test/test_observability.py 40 files changed, 664 insertions(+), 208 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/28 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType:
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 27: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10056/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 27 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 28 Jan 2022 00:32:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 27: rebase -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 27 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 28 Jan 2022 00:11:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#27). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1 .. IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group in the set is associated with a distinct number of nodes and a threshold for estimated memory per host in bytes that can be denoted as [:<#nodes>, ]. In the patch, a query of type EXPLAIN, QUERY or DML can be compiled more than once. In each attempt, per host memory is estimated and compared with the threshold of an executor group. If the estimated memory is no more than the threshold, the iteration process terminates and the final plan is determined. The executor group with the threshold is selected to run against the query. A new query option 'enable_replan', default to 1 (enabled), is added. It can be set to 0 to disable this patch and to generate the distributed plan for the default executor group. To facilitate testing, the patch imposes an artificial two executor group setup in FE. This setup is enabled when 'enable_replan' is set to 3 (testing) or RuntimeEnv.INSTANCE.isTestEnv() is true as in most frontend tests. The artificial two executor groups are configured as follows. 1. [regular:3, 64MB] 2. [large:3, 8PB] To avoid long compilation time, the following enhancement is enabled. Note 1) and 2) can be disabled when some meta-data change is detected. 1. Authorization is performed only for the 1st compilation; 2. The relevant meta-data is fetched into a StmtTableCache in 1st compilation and reused in subsequent compilations; 3. openTransaction() is called for transactional queries in 1st compilation and the saved transactional info is used in subsequent compilations. Similar logic is applied to Kudu transactional queries. Testing: 1. Ran core tests Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M fe/src/test/java/org/apache/impala/planner/ClusterSizeTest.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_coordinators.py M tests/custom_cluster/test_executor_groups.py M tests/query_test/test_observability.py 40 files changed, 653 insertions(+), 208 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/27 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType:
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 26: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10055/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 26 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 28 Jan 2022 00:11:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 26: Class 15 (fixed by disable replan in test_observability.py, as it can produce multiple event sequences in event_regexes): query_test.test_observability.TestObservability.test_query_profile_contains_query_compilation_lineage_event (from pytest) E AssertionError: Planning finished: not in- Analysis finished: 2.013ms (824.408us) E Query (id=184137c564236526:b64885ff): E DEBUG MODE WARNING: Query profile created while running a DEBUG build of Impala. Use RELEASE builds to measure query performance. E- Ina Class 5 (fixed by disable stmt cache and reauthorize when there is meta-data change: custom_cluster.test_metastore_service.TestMetastoreService.test_database_create_drop_seq (from pytest) Error Message ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: MESSAGE: InconsistentMetadataFetchException: Fetching DATABASE failed. Could not find TCatalogObject(type:DATABASE, catalog_version:0, db:TDatabase(db_name:test_database_create_drop_seq_db_sgurp)) -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 26 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 27 Jan 2022 23:47:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#26). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1 .. IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group in the set is associated with a distinct number of nodes and a threshold for estimated memory per host in bytes that can be denoted as [:<#nodes>, ]. In the patch, a query of type EXPLAIN, QUERY or DML can be compiled more than once. In each attempt, per host memory is estimated and compared with the threshold of an executor group. If the estimated memory is no more than the threshold, the iteration process terminates and the final plan is determined. The executor group with the threshold is selected to run against the query. A new query option 'enable_replan', default to 1 (enabled), is added. It can be set to 0 to disable this patch and to generate the distributed plan for the default executor group. To facilitate testing, the patch imposes an artificial two executor group setup in FE. This setup is enabled when 'enable_replan' is set to 3 (testing) or RuntimeEnv.INSTANCE.isTestEnv() is true as in most frontend tests. The artificial two executor groups are configured as follows. 1. [regular:3, 64MB] 2. [large:3, 8PB] To avoid long compilation time, the following enhancement is enabled. Note 1) and 2) can be disabled when some meta-data change is detected. 1. Authorization is performed only for the 1st compilation; 2. The relevant meta-data is fetched into a StmtTableCache in 1st compilation and reused in subsequent compilations; 3. openTransaction() is called for transactional queries in 1st compilation and the saved transactional info is used in subsequent compilations. Similar logic is applied to Kudu transactional queries. Testing: 1. Ran core tests Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M fe/src/test/java/org/apache/impala/planner/ClusterSizeTest.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_coordinators.py M tests/custom_cluster/test_executor_groups.py M tests/query_test/test_observability.py 40 files changed, 653 insertions(+), 208 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/26 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType:
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 25: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10048/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 25 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 26 Jan 2022 22:49:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 25: class 13 (fixed in Frontend.java, via endsWith() comparison). custom_cluster.test_executor_groups.TestExecutorGroups.test_admission_control_with_multiple_coords_and_exec_groups (from pytest) Stacktrace custom_cluster/test_executor_groups.py:645: in test_admission_control_with_multiple_coords_and_exec_groups handle_long_running_queue1 = self.execute_query_async(LONG_QUERY) common/impala_test_suite.py:836: in wrapper return function(*args, **kwargs) common/impala_test_suite.py:899: in execute_query_async return self.client.execute_async(query) common/impala_connection.py:217: in execute_async beeswax_handle = self.__beeswax_client.execute_query_async(sql_stmt, user=user) beeswax/impala_beeswax.py:359: in execute_query_async handle = self.__do_rpc(lambda: self.imp_service.query(query,)) beeswax/impala_beeswax.py:522: in __do_rpc raise ImpalaBeeswaxException(self.__build_error_message(b), b) E ImpalaBeeswaxException: ImpalaBeeswaxException: EINNER EXCEPTION: EMESSAGE: NullPointerException: null -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 25 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 26 Jan 2022 22:25:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#25). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1 .. IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group in the set is associated with a distinct number of nodes and a threshold for estimated memory per host in bytes. In the patch, a query of type EXPLAIN, QUERY, LOAD or DML can be compiled more than once. In each attempt, per host memory is estimated and compared with the threshold of an executor group. If the estimated memory is no more than the threshold, the iteration process terminates and the final plan is determined. The executor group with the threshold is selected to run against the query. A new query option 'enable_replan', default to 1 (enabled), is added. It can be set to 0 to disable this patch and to generate the distributed plan for the default executor group. To avoid long compilation time, the following enhancement is enabled. 1. Authorization is performed only for the 1st compilation; 2. Meta-data is fetched into a StmtTableCache in 1st compilation and reused in subsequent compilations; 3. openTransaction() is called for transactional queries in 1st 1st compilation and the saved transactional info is used in subsequent compilations. Similar logic is applied to Kudu transactional queries. Testing: 1. The planner test passes in an artificial two-executor group setup with the last group being identical to a 3-node default group. To do: 1. BE testing of auto-scaled plans. Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M fe/src/test/java/org/apache/impala/planner/ClusterSizeTest.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_coordinators.py M tests/custom_cluster/test_executor_groups.py 39 files changed, 636 insertions(+), 208 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/25 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 25 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18143 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1 .. Patch Set 24: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10047/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 24 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 26 Jan 2022 18:32:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/18143 )
Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1
..
Patch Set 24:
Fix the following:
class 10 (fixed, 1/26/2022)
custom_cluster.test_admission_controller.TestAdmissionController.test_dedicated_coordinator_planner_estimates[protocol:
beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
text/none] (from pytest)
class 11 (not seen after rework the main loop in Frontend.java, 1/26/2022)
custom_cluster.test_concurrent_ddls.TestConcurrentDdls.test_local_catalog_ddls_with_invalidate_metadata_sync_ddl
(from pytest)
Error Message
AssertionError: Timeout in thread run_ddls(13) assert False
Stacktrace
custom_cluster/test_concurrent_ddls.py:77: in
test_local_catalog_ddls_with_invalidate_metadata_sync_ddl
self._run_ddls_with_invalidation(unique_database, sync_ddl=True)
custom_cluster/test_concurrent_ddls.py:149: in _run_ddls_with_invalidation
assert False, "Timeout in thread run_ddls(%d)" % i
E AssertionError: Timeout in thread run_ddls(13)
E assert False
--
To view, visit http://gerrit.cloudera.org:8080/18143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24
Gerrit-Change-Number: 18143
Gerrit-PatchSet: 24
Gerrit-Owner: Qifan Chen
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Wed, 26 Jan 2022 18:08:45 +
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10992 Planner changes for estimate peak memory - v1
Qifan Chen has uploaded a new patch set (#24). ( http://gerrit.cloudera.org:8080/18143 ) Change subject: IMPALA-10992 Planner changes for estimate peak memory - v1 .. IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group in the set is associated with a distinct number of nodes and a threshold for estimated memory per host in bytes. In the patch, a query of type EXPLAIN, QUERY, LOAD or DML can be compiled more than once. In each attempt, per host memory is estimated and compared with the threshold of an executor group. If the estimated memory is no more than the threshold, the iteration process terminates and the final plan is determined. The executor group with the threshold is selected to run against the query. A new query option 'enable_replan', default to 1 (enabled), is added. It can be set to 0 to disable this patch and to generate the distributed plan for the default executor group. To avoid long compilation time, the following enhancement is enabled. 1. Authorization is performed only for the 1st compilation; 2. Meta-data is fetched into a StmtTableCache in 1st compilation and reused in subsequent compilations; 3. openTransaction() is called for transactional queries in 1st 1st compilation and the saved transactional info is used in subsequent compilations. Similar logic is applied to Kudu transactional queries. Testing: 1. The planner test passes in an artificial two-executor group setup with the last group being identical to a 3-node default group. To do: 1. BE testing of auto-scaled plans. Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M fe/src/test/java/org/apache/impala/planner/ClusterSizeTest.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_coordinators.py M tests/custom_cluster/test_executor_groups.py 39 files changed, 630 insertions(+), 208 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/24 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 24 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen
