Qifan Chen has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/18123 )
Change subject: [WIP] IMPALA-10992 Planner changes for estimate peak memory - v3 ...................................................................... [WIP] IMPALA-10992 Planner changes for estimate peak memory - v3 This patch provides replan support in planner for a set of executor groups. Each executor group is associated with distinct number of nodes and a threshold for estimated memory per host (in bytes). In the patch, attempts are made to estimate per host memory from the serial plan, select a suitable executor group to use and generate the distributed plan for it. A new query option 'enable_replan', default to true, is added. It can be set to false to disable this patch and to generate the distributed plan for the default executor group. The testing of the quality of the replanning is done by verifying that the planner tests generate identical plans in a two-executor group configuration with the last one being identical to the existing default executor environment. Change-Id: If8e1776ffef3948c9d5fbf895fb4a19317e43502 --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/Frontend.thrift M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/common/IdGenerator.java M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/util/ClassUtil.java M fe/src/main/java/org/apache/impala/util/ExecutorMembershipSnapshot.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test 14 files changed, 198 insertions(+), 34 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18123/6 -- To view, visit http://gerrit.cloudera.org:8080/18123 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If8e1776ffef3948c9d5fbf895fb4a19317e43502 Gerrit-Change-Number: 18123 Gerrit-PatchSet: 6 Gerrit-Owner: Qifan Chen <qc...@cloudera.com> Gerrit-Reviewer: Aman Sinha <amsi...@cloudera.com> Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com> Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com> Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
