Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11208 )
Change subject: IMPALA-7437. LRU caching of partitions in impalad ...................................................................... IMPALA-7437. LRU caching of partitions in impalad This changes the CatalogdMetaProvider to use a Guava-based LRU cache. The eviction strategy is currently time-based (1 hour), and it only performs caching of some basic items like partition information, the null-partition-key-value, and table column statistics. It does not cache the table entries themselves, which means that we don't need to do any invalidation propagation via the statestore quite yet. Instead, every query will do an initial fetch of the table metadata in order to know the current version number. That version number is then used as part of the cache key for all further metadata, so when the version number changes, all of the prior cache entries become "unreachable" and effectively evicted. Initially, I attempted to implement this by adding a new MetaProvider implementation that would transparently wrap another MetaProvider implementation (either catalogd-based or direct-from-source). However, I found that I wanted to use catalogd-based implementation details like the version number in the cache key, and trying to abstract this behind an interface wasn't very clear. So, I elected to just embed the caching logic into the CatalogdMetaProvider itself. Note that this patch upgrades the Guava reference in the pom from 11.0.2 to 14.0.1. In fact, I found that Guava 14.0.1 was already leaking onto the classpath by being included in hive-exec.jar, so it was ending up picking one or the other in a somewhat unpredictable fashion. The CacheBuilder class had a small API change between v11 and v14 so I needed to ensure a specific version so that Eclipse and Maven agreed on which version to build against. This includes some basic unit testing and I also verified that some query tests like TPCH pass. Change-Id: I9a57521ad851da605604a1e7c48d3d6627da5df5 Reviewed-on: http://gerrit.cloudera.org:8080/11208 Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Reviewed-by: Vuk Ercegovac <vercego...@cloudera.com> --- M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java A fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java M impala-parent/pom.xml 6 files changed, 538 insertions(+), 38 deletions(-) Approvals: Impala Public Jenkins: Verified Vuk Ercegovac: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/11208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9a57521ad851da605604a1e7c48d3d6627da5df5 Gerrit-Change-Number: 11208 Gerrit-PatchSet: 8 Gerrit-Owner: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com> Gerrit-Reviewer: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Vuk Ercegovac <vercego...@cloudera.com>
