Hello Qifan Chen, Shant Hovsepian, David Rorke, Tim Armstrong, Impala Public
Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16219
to look at the new patch set (#11).
Change subject: IMPALA-9983 : Pushdown limit to analytic sort operator
......................................................................
IMPALA-9983 : Pushdown limit to analytic sort operator
This patch pushes the LIMIT from a top level Sort down to
the Sort below an Analytic operator when it is safe to do
so. There are several qualifying checks that are done. The
optimization is done at the time of creating the top level
Sort in the single node planner. When the pushdown is
applicable, the analytic sort is converted to a TopN sort.
Further, this is split into 2 TopN sorts separated by a
hash partition exchange. This ensures that the limit is
applied as early as possible before hash partitioning.
Fixed couple of additional related issues uncovered as a
result of limit pushdown:
- Changed the analytic sort's partition-by expr sort
semantic from NULLS FIRST to NULLS LAST to ensure
correctness in the presence of limit.
- The LIMIT on the analytic sort node was causing it to
be treated as a merging point in the distributed planner.
Fixed it by introducing an api allowPartitioned() in the
PlanNode.
Testing:
- Ran PlannerTest and updated several EXPLAIN plans.
- Added Planner tests for both positive and negative cases of
limit pushdown.
- Ran end-to-end TPC-DS queries. Specifically tested
TPC-DS q67 for limit pushdown and result correctness.
- Added targeted end-to-end tests (TODO: capture results)
Change-Id: Ib39f46a7bb75a34466eef7f91ddc25b6e6c99284
---
M fe/src/main/java/org/apache/impala/analysis/AnalyticExpr.java
M fe/src/main/java/org/apache/impala/analysis/AnalyticWindow.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
M fe/src/main/java/org/apache/impala/planner/AnalyticPlanner.java
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/planner/SortNode.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M
testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns-mt-dop.test
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M
testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test
M testdata/workloads/functional-planner/queries/PlannerTest/convert-to-cnf.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
M testdata/workloads/functional-planner/queries/PlannerTest/insert.test
A
testdata/workloads/functional-planner/queries/PlannerTest/limit-pushdown-analytic.test
M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test
M
testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test
M
testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test
M
testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M
testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test
M
testdata/workloads/functional-planner/queries/PlannerTest/semi-join-distinct.test
M
testdata/workloads/functional-planner/queries/PlannerTest/sort-expr-materialization.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test
A
testdata/workloads/functional-query/queries/QueryTest/limit-pushdown-analytic.test
M tests/query_test/test_queries.py
27 files changed, 1,297 insertions(+), 278 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/16219/11
--
To view, visit http://gerrit.cloudera.org:8080/16219
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib39f46a7bb75a34466eef7f91ddc25b6e6c99284
Gerrit-Change-Number: 16219
Gerrit-PatchSet: 11
Gerrit-Owner: Aman Sinha <[email protected]>
Gerrit-Reviewer: Aman Sinha <[email protected]>
Gerrit-Reviewer: David Rorke <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Qifan Chen <[email protected]>
Gerrit-Reviewer: Shant Hovsepian <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
