Vuk Ercegovac has uploaded a new patch set (#4) to the change originally created by Todd Lipcon. ( http://gerrit.cloudera.org:8080/11403 )
Change subject: IMPALA-7530, IMPALA-7509. Re-plan queries if they fetch inconsistent metadata ...................................................................... IMPALA-7530, IMPALA-7509. Re-plan queries if they fetch inconsistent metadata Given the granular catching in CatalogdMetaProvider, it's possible to have a situation like the following: - impalad has cached the table list including some table "t" - some other daemon has issued deletion of "t" - we try to access "t" before the invalidation has reached us via the statestore In this case, we'll get an error back when we try to fetch the table "t". This can confuse the planning process since it previously determined that the table exists. Note that this may occur either when "t" is first referenced or later during planning (eg when fetching a specific partition post-pruning), so it wouldn't be possible to simply convert it into a 'table does not exist' result. The solution here is to throw InconsistentMetadataFetchException after invalidating the table list and associated object. This then triggers a re-plan of the query. This patch implements such re-planning up to 10 times before eventually giving up. Given that each attempt invalidates the inconsistent cache we would normally expect it to succeed on the first such retry. The limit of 10 retries is just to avoid infinite loops in the case of a bug. Change-Id: I17389954780fa22d7866224c17e128458fffa545 --- M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/custom_cluster/test_local_catalog.py 5 files changed, 192 insertions(+), 68 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/11403/4 -- To view, visit http://gerrit.cloudera.org:8080/11403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I17389954780fa22d7866224c17e128458fffa545 Gerrit-Change-Number: 11403 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Vuk Ercegovac <vercego...@cloudera.com>
