Hello Bharath Vissapragada, Tianyi Wang, Vuk Ercegovac, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11208
to look at the new patch set (#3).
Change subject: IMPALA-7437. LRU caching of partitions in impalad
......................................................................
IMPALA-7437. LRU caching of partitions in impalad
This changes the CatalogdMetaProvider to use a Guava-based LRU cache.
The eviction strategy is currently time-based (1 hour), and it only
performs caching of partition information and table column statistics.
It does not cache the table entries themselves, which means that we
don't need to do any invalidation propagation via the statestore quite
yet. Instead, every query will do an initial fetch of the table metadata
in order to know the current version number. That version number is then
used as part of the cache key for all further metadata, so when the
version number changes, all of the prior cache entries become
"unreachable" and effectively evicted.
Initially, I attempted to implement this by adding a new MetaProvider
implementation that would transparently wrap another MetaProvider
implementation (either catalogd-based or direct-from-source). However, I
found that I wanted to use catalogd-based implementation details like
the version number in the cache key, and trying to abstract this behind
an interface wasn't very clear. So, I elected to just embed the caching
logic into the CatalogdMetaProvider itself.
This includes some basic unit testing and I also verified that some
query tests like TPCH pass.
Change-Id: I9a57521ad851da605604a1e7c48d3d6627da5df5
---
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
A fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java
4 files changed, 435 insertions(+), 37 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/11208/3
--
To view, visit http://gerrit.cloudera.org:8080/11208
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9a57521ad851da605604a1e7c48d3d6627da5df5
Gerrit-Change-Number: 11208
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <t...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <vercego...@cloudera.com>
