Qifan Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18143
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 ...................................................................... [WIP] IMPALA-10992 Planner changes for estimate peak memory - v1 This patch provides replan support in planner for a set of executor groups. Each executor group is associated with distinct number of nodes and a threshold for estimated memory per host (in bytes). In the patch, attempts are made to estimate per host memory from a complete compilation of a query, select a suitable executor group to use and generate the distributed plan for it. A new query option 'enable_replan', default to true, is added. It can be set to false to disable this patch and to generate the distributed plan for the default executor group. The testing of the quality of the replanning is done by verifying that the planner tests generate identical plans in a two-executor group configuration with the last one being identical to the existing default executor environment. Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds/tpcds-q80.test 33 files changed, 399 insertions(+), 180 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18143/5 -- To view, visit http://gerrit.cloudera.org:8080/18143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ibe71f905d6a8c1e42cf951b3a69ff33b81277c24 Gerrit-Change-Number: 18143 Gerrit-PatchSet: 5 Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
