Vihang Karajgaonkar has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17697 )
Change subject: IMPALA-10801: Check the latest compaction Id before serving ACID table ...................................................................... IMPALA-10801: Check the latest compaction Id before serving ACID table Since compactions don't advance write id, we don't know if a table/partition is compacted by comparing writeIdList. A possible issue is that CatalogD provides obsolete file metadata and causes a runtime error. In order to fix this issue, we introduced a HMS API that can get the latest compaction record for a table/partition (HIVE-24828). In CatalogD, we compare the cached id with the latest compaction id before serving. If there is a newer compaction happened, we will cache the latest compaction id and refresh the file metadata. Besides, this patch also change how to replace the existing table after a table full reloading. The current way is to replace the table if the catalog version is not changed. For transactional tables, things get additional complexity given that file metadata refreshing and full table reloading can happen together. We can actually use writeIdList to determine whether we should replace the table for transactional tables. As long as the updated table has more recent writeIdList than the existing one, we are safe to replace the table. For non-transactional tables, we still keep the original behavior. Testing: - Add several tests in PartialCatalogInfoWriteIdTest Change-Id: I86a112a77980fef7f6238978bc9668a65262101e Reviewed-on: http://gerrit.cloudera.org:8080/17697 Reviewed-by: Vihang Karajgaonkar <vih...@cloudera.com> Tested-by: Vihang Karajgaonkar <vih...@cloudera.com> --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java A fe/src/main/java/org/apache/impala/catalog/CompactionInfoLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java 6 files changed, 369 insertions(+), 29 deletions(-) Approvals: Vihang Karajgaonkar: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I86a112a77980fef7f6238978bc9668a65262101e Gerrit-Change-Number: 17697 Gerrit-PatchSet: 17 Gerrit-Owner: Yu-Wen Lai <yu-wen....@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Sourabh Goyal <soura...@cloudera.com> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com> Gerrit-Reviewer: Yu-Wen Lai <yu-wen....@cloudera.com>
