Hello Aman Sinha, Gabor Kaszab, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16228
to look at the new patch set (#4).
Change subject: IMPALA-9859: Full ACID Milestone 4: Part 2 Reading modified
tables (complex types)
......................................................................
IMPALA-9859: Full ACID Milestone 4: Part 2 Reading modified tables (complex
types)
This implements scanning full ACID tables that contain complex types.
The same technique works that we use for primitive types. I.e. we add
a LEFT ANTI JOIN on top of the Hdfs scan node in order to subtract
the deleted rows from the inserted rows.
However, there were some types of queries where we couldn't do that.
These are the queries that scan the nested collection items directly.
E.g.: SELECT item FROM complextypestbl.int_array;
The above query only creates a single tuple descriptor that holds the
collection items. Since this tuple descriptor is not at the table-level,
we cannot add slot references to the hidden ACID column which are at the
top level of the table schema.
To resolve this I added a statement rewriter that rewrites the above
statement to the following:
SELECT item FROM complextypestbl $a$1, $a$1.int_array;
Now in this example we'll have two tuple descriptors, one for the
table-level, and one for the collection item. So we can add the ACID
slot refs to the table-level tuple descriptor. The rewrite is
implemented by the new AcidRewriter class.
Testing
* Added planner tests to PlannerTest/acid-scans.test
* E2E query tests to QueryTest/full-acid-complex-type-scans.test
* E2E tests for rowid-generation: QueryTest/full-acid-rowid.test
Change-Id: I8b2c6cd3d87c452c5b96a913b14c90ada78d4c6f
---
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/FromClause.java
M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
M fe/src/main/java/org/apache/impala/analysis/TableRef.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/workloads/functional-planner/queries/PlannerTest/acid-scans.test
M testdata/workloads/functional-query/queries/QueryTest/acid-negative.test
A
testdata/workloads/functional-query/queries/QueryTest/full-acid-complex-type-scans.test
M testdata/workloads/functional-query/queries/QueryTest/full-acid-rowid.test
M tests/query_test/test_acid.py
12 files changed, 924 insertions(+), 48 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/16228/4
--
To view, visit http://gerrit.cloudera.org:8080/16228
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8b2c6cd3d87c452c5b96a913b14c90ada78d4c6f
Gerrit-Change-Number: 16228
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <borokna...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <amsi...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <gaborkas...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
