Hi Alan, To add to Daniel’s response, as part of https://issues.apache.org/jira/browse/HIVE-18264 and https://issues.apache.org/jira/browse/HIVE-18661 (I’m actively working on these), we plan to remove the current mechanism of updating the cache (which is very inefficient anyway) and instead use the NOTIFICATION_LOG table to update the cache incrementally. The code that you pointed was meant to not let the background update thread block the metastore client calls for a long time, but with the plan to update the cache incrementally we may not need to worry about that, as applying the notification incrementally will not be a long blocking execution.
Thanks, --Vaibhav On 2/8/18, 11:41 AM, "Daniel Dai" <[email protected]> wrote: Hi, Alan, If database cache is changed locally, we don’t want to bring remote copy to overwrite it as the remote copy doesn’t carry local changes (ideally, we shall also apply local changes to the remote copy images we bring in from db, but we are not there yet). That’s why we skip the update if there’s local changes, and wait for the next iteration to sync with remote. isDatabaseCacheDirty is initially set to false unless there’s local update, and will be reset during cache swap, thus give a chance for the next iteration to update the cache if there’s no local changes. Thanks, Daniel On 2/6/18, 11:57 AM, "Alan Gates" <[email protected]> wrote: I’m confused by the following code in the CachedStore. This in in the CacheUpdateMasterWork thread, in the updateDatabases method (which is called by update()): *// Skip background updates if we detect change* *if *(*isDatabaseCacheDirty*.compareAndSet(*true*, *false*)) { *LOG*.debug(*"Skipping database cache update; the database list we have is dirty."*); *return*; } Why are we not updating the cache if we’ve dirtied it? Also, AFAICT no one ever sets isDatabaseCacheDirty to false, meaning once one database is created the cache will never be updated. Am I missing something? Alan.
