Qifan Chen has uploaded a new patch set (#21). (
http://gerrit.cloudera.org:8080/17706 )
Change subject: IMPALA-3430: Runtime filter : Extend runtime filter to support
Min/Max values for HDFS scans
......................................................................
IMPALA-3430: Runtime filter : Extend runtime filter to support Min/Max values
for HDFS scans
This patches enables min/max filtering for non-correlated subqueries
that return one row. In this case, the filters are built from the
results of the subqueries and the filtering target is the scan node to
be qualified by one of the subqueries. Shown below is one such query
that normally gets compiled into a nested loop join. The filtering
limits the values from column store_sales.ss_sales_price to be within
[-infinite, min(ss_wholesale_cost)].
select count(*) from store_sales
where ss_sales_price <=
(select min(ss_wholesale_cost) from store_sales);
In the patch, the min/max filtering infrastructure is integrated with
the nested loop join as follows.
1. Similar to hash join, NljBuilderConfig is populated with filter
descriptors from nested join plan node via
NljBuilder::CreateEmbeddedBuilder();
2. NljBuilder is populated with filter contexts utilizing the filter
descriptors in NljBuilderConfig. Filter contexts are the interface
to actual min/max filters;
3. New insertion methods InertFor<op>(), where <op> is LE, LT, GE and
GT, are added to MinMaxFilter class hierarcy. They are used for
join predicate target <op> src_expr;
4. RuntimeContext::InsertPerCompareOp() calls one of the new
insertion methods above based on the comparison op saved in the
filter descriptor;
5. NljBuilder::InsertRuntimeFilters() calls the new methods.
By default, the feature is turned on only for sorted or partitioned
join columns.
Testing:
1. Add single range insertion tests in min-max-filter-test.cc;
2. Add positive and negative plan tests in
overlap_min_max_filters.test;
3. Add tests in overlap_min_max_filters_on_partition_columns.test;
4. Add tests in overlap_min_max_filters_on_sorted_columns.test;
5. Run core tests.
Change-Id: I7c2bb5baad622051d1002c9c162c672d428e5446
---
M be/src/exec/filter-context.cc
M be/src/exec/filter-context.h
M be/src/exec/nested-loop-join-builder.cc
M be/src/exec/nested-loop-join-builder.h
M be/src/exec/nested-loop-join-node.cc
M be/src/runtime/coordinator.cc
M be/src/runtime/query-state.h
M be/src/runtime/runtime-filter.h
M be/src/runtime/timestamp-value.h
M be/src/util/min-max-filter-ir.cc
M be/src/util/min-max-filter-test.cc
M be/src/util/min-max-filter.cc
M be/src/util/min-max-filter.h
M common/thrift/PlanNodes.thrift
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java
M fe/src/main/java/org/apache/impala/analysis/Predicate.java
M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
M fe/src/main/java/org/apache/impala/catalog/ScalarType.java
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M
testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters.test
M
testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters_on_partition_columns.test
M
testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters_on_sorted_columns.test
29 files changed, 1,205 insertions(+), 52 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/17706/21
--
To view, visit http://gerrit.cloudera.org:8080/17706
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7c2bb5baad622051d1002c9c162c672d428e5446
Gerrit-Change-Number: 17706
Gerrit-PatchSet: 21
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Amogh Margoor <amarg...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
