[
https://issues.apache.org/jira/browse/IMPALA-14856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Smith updated IMPALA-14856:
-----------------------------------
Summary: Unnecessary invalidation for just loaded table meta (was:
Unneccessary invalidation for just loaded table meta)
> Unnecessary invalidation for just loaded table meta
> ---------------------------------------------------
>
> Key: IMPALA-14856
> URL: https://issues.apache.org/jira/browse/IMPALA-14856
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog, Frontend
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> In local catalog mode, running a query on an unloaded table can't make sure
> its metadata is loaded in coordinator side. This is due to a race between two
> metadata flows:
> # Coordinator uses getPartialCatalogObject RPC to load the metadata from
> catalogd. Once coordinator gets the response, the table meta is cached.
> # getPartialCatalogObject RPC on the unloaded table actually triggers
> catalogd to load metadata of it. When the loading is done, catalogd bumps the
> table version. The new table version will then be collected in the statestore
> update, and send to all coordinators.
> For the coordinator that triggers the request, it will invalidate the table
> meta when receiving the statestore update, regradless what the table version
> is.
> {code:java}
> private void invalidateCacheForTable(String dbName, String tblName,
> List<String> invalidated) {
> TableCacheKey key = new TableCacheKey(dbName.toLowerCase(),
> tblName.toLowerCase());
> if (cache_.asMap().remove(key) != null) {
> invalidated.add("table " + dbName + "." + tblName);
> }
> }{code}
> [https://github.com/apache/impala/blob/20220fb9232b94d228383fe693a383d2c71a4733/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java#L1843-L1844]
> Ideally the invalidation should check the cached table version and only
> invalidate the item when it's older than the statestore update. However,
> there are two problems:
> * Checking the cached version using getIfPresent() and removing the item is
> not atomic. Another thread could put a newer version in between and got
> invalidated unintentionally.
> * The value could be a CompletableFuture loading the metadata from catalogd.
> We should remove it only when it's loading an older version. How to get the
> version it's loading is also a problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
