Hello Vihang Karajgaonkar, Sudhanshu Arora, Zoltan Borok-Nagy, Todd Lipcon,
Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13215
to look at the new patch set (#11).
Change subject: IMPALA-8438: Store WriteId and ValidWriteId list for table and
partition
......................................................................
IMPALA-8438: Store WriteId and ValidWriteId list for table and partition
This happens when tables load metadata from HMS.
Add MetastoreShim functions to support HMS3 only functions.
Add validwriteIdlists to query profile through timeline.
Tests:
Manually tests HMS2 and HMS3, using log files to check
Unit tests against HMS3
ToDo:
WriteId and valid writeIds can be fetched in other time, need
more study on that.
Profile example:
Query Compilation: 5s057ms
- Metadata load started: 63.006ms (63.006ms)
- Metadata load finished. loaded-tables=2/2...: 4s801ms (4s738ms)
- Loaded ValidWriteIdLists:
acid.insert_only_no_partitions:6:9223372036854775807::
acid.insert_only_with_partitions:3:9223372036854775807::
: 4s921ms (120.580ms)
- Analysis finished: 4s929ms (8.013ms)
Change-Id: I6edbd64424edf0ba88af110ab8b958a1966b8b54
---
M common/thrift/CatalogObjects.thrift
A fe/src/compat-hive-2/java/org/apache/hadoop/hive/common/ValidWriteIdList.java
M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
M fe/src/main/java/org/apache/impala/catalog/FeFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/FeTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java
15 files changed, 346 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/13215/11
--
To view, visit http://gerrit.cloudera.org:8080/13215
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6edbd64424edf0ba88af110ab8b958a1966b8b54
Gerrit-Change-Number: 13215
Gerrit-PatchSet: 11
Gerrit-Owner: Yongzhi Chen <yc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Sudhanshu Arora <sudhan...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com>
Gerrit-Reviewer: Yongzhi Chen <yc...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
