Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14690
to look at the new patch set (#3).
Change subject: IMPALA-9146: Add a configurable limit for the size of broadcast
input.
......................................................................
IMPALA-9146: Add a configurable limit for the size of broadcast input.
Impala's DistributedPlanner may sometimes accidentally choose broadcast
distribution for inputs that are larger than the destination executor's
total memory. This could potentially happen if the cluster membership is
not accurately known and the planner's cost computation of the
broadcastCost vs partitionCost happens to favor the broadcast
distribution. This causes spilling and severely affects performance.
Although the DistributedPlanner does a mem_limit check before picking
broadcast, the mem_limit is not an accurate reflection since it is
assigned during admission control.
As a safety here we introduce an explicit configurable limit:
broadcast_bytes_limit for the size of the broadcast input and set it
to default of 32GB. The default is chosen based on analysis of existing
benchmark queries and representative workloads. If the estimated input
size on the build side is greater than this threshold, the
DistributedPlanner will fall back to a partition distribution.
Testing:
- Ran all regression tests on Jenkins successfully
- Added a new unit test in PlannerTest that sets the
broadcast_bytes_limit to a small value and checks whether the
distributed plan does hash partitioning on the build side instead of
broadcast.
Change-Id: Ibe5639ca38acb72e0194aa80bc6ebb6cafb2acd9
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
A
testdata/workloads/functional-planner/queries/PlannerTest/broadcast-bytes-limit.test
9 files changed, 75 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/14690/3
--
To view, visit http://gerrit.cloudera.org:8080/14690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibe5639ca38acb72e0194aa80bc6ebb6cafb2acd9
Gerrit-Change-Number: 14690
Gerrit-PatchSet: 3
Gerrit-Owner: Aman Sinha <amsi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
