Hello Bharath Vissapragada, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13874
to look at the new patch set (#11).
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local
catalog mode
......................................................................
IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
In LocalCatalog implementation, LocalDb#getTable will always return a
completely loaded table containing all the meta of columns, partitions,
files, etc. It's time consuming if we implement the GET_TABLES
HiveServer2 operation based on this interface, since GET_TABLES only
requires table names, table types and table comments, while this
interface will trigger catalogd to fully load the table meta. It becomes
worse when we do this for all the tables.
This patch introduces a new interface, getTableIfCached, to return a
LocalIncompleteTable object if the corresponding table is unloaded,
which requires no round trips to the catalogd. It's used to boost the
GET_TABLES performance in LocalCatalog mode.
Tests
- Testing in a HMS with 100 dbs and 3000 tables, without this patch it
takes ~2mins in GET_TABLES for all tables on a cold started cluster. With
this patch, the time reduces to ~1s.
- Testing in HUE-4.4.0 with a db with 3000 tables, the performance is the
same as using legacy catalog implementation.
Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
---
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
A fe/src/main/java/org/apache/impala/catalog/local/LocalIncompleteTable.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M fe/src/test/java/org/apache/impala/service/JdbcTest.java
M tests/hs2/test_hs2.py
11 files changed, 156 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/13874/11
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 11
Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
