Yu-Wen Lai has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/18043 )
Change subject: IMPALA-11032: Automatic Refresh of Metadata for Local Catalog after Compaction ...................................................................... IMPALA-11032: Automatic Refresh of Metadata for Local Catalog after Compaction After compaction happened in Hive(HIVE ACID table), queries made in Impala possibly fail with a FileNotFoundException if files already removed by the Hive cleaner. In IMPALA-10801, catalogd checks the latest compaction id before serving metadata. However, coordinators don't take advantage of that. Coordinators have their own local cache, so we will have to do the same check for coordinators as well. Besides, we also need to attach writeIdList to requests that need to fetch file metadata. Since this checking brings additional overhead for queries, we introduce a flag auto_check_compaction and set it as false by default for now. We will find some other efficient ways to do compaction checking in the future. Tests: Added unit tests to CatalogdMetaProviderTest Change-Id: I173ea848917b6a41139b25b80677111463bfdc4b --- M be/src/service/impala-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CompactionInfoLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java 12 files changed, 337 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18043/4 -- To view, visit http://gerrit.cloudera.org:8080/18043 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I173ea848917b6a41139b25b80677111463bfdc4b Gerrit-Change-Number: 18043 Gerrit-PatchSet: 4 Gerrit-Owner: Yu-Wen Lai <yu-wen....@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Sourabh Goyal <soura...@cloudera.com> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com> Gerrit-Reviewer: Yu-Wen Lai <yu-wen....@cloudera.com>
