Hello Tamas Mate, Gabor Kaszab, Zoltan Borok-Nagy, Gergely Fürnstáhl, Impala
Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19380
to look at the new patch set (#6).
Change subject: IMPALA-10893: Use old schema during iceberg time travel.
......................................................................
IMPALA-10893: Use old schema during iceberg time travel.
Before this change the schema used during Iceberg Time Travel was the
current schema of the table. With this change we will use the schema
from the point specified by the Time Travel parameters.
The parameters used by an Iceberg Time Travel query are part of the FROM
clause of the query. Previously analysis of the Time Travel parameters
took place after the table Path was resolved, at which point some
schema information is cached. In order to use the old schema during
iceberg time travel however we need to ensure that the version of the
Table that is used is always the version specified by the Time Travel
parameters. To do this we have to move the analysis of the Time Travel
parameters inside the code that resolves the Path.
Add a new implementation of FeIcebergTable that represents an Iceberg
table involved in Time Travel. This is implemented by embedding a
reference to the base Iceberg Table. All methods that are not Time
Travel related are delegated to the base table. The Time Travel related
methods use the historic Iceberg schema.
TESTING:
- Add a new file iceberg_util.py to hold the snapshot utility code that
was developed for the in-progress IMPALA-11482.
- Extend the existing Iceberg Time Travel tests to check the schema.
- Add a test that shows time travel working with columns masking.
The column masking configuration is not tightly coupled to the schema
so it is possible to mask historical columns.
Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java
M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java
M fe/src/main/java/org/apache/impala/analysis/TableRef.java
M fe/src/main/java/org/apache/impala/analysis/TimeTravelSpec.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
A fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M tests/authorization/test_ranger.py
M tests/common/iceberg_test_suite.py
M tests/query_test/test_iceberg.py
A tests/util/iceberg_util.py
16 files changed, 1,036 insertions(+), 98 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/19380/6
--
To view, visit http://gerrit.cloudera.org:8080/19380
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab
Gerrit-Change-Number: 19380
Gerrit-PatchSet: 6
Gerrit-Owner: Andrew Sherman <asher...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <asher...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <gaborkas...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gfurnst...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tma...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
