Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/18043 )
Change subject: IMPALA-11032: Automatic Refresh of Metadata for Local Catalog after Compaction ...................................................................... Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/18043/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18043/2//COMMIT_MSG@10 PS2, Line 10: After compaction happened in Hive(HIVE ACID table), queries made in : Impala possibly fail with a FileNotFoundException if files already : removed by the Hive cleaner. Can you confirm if Impala open's a transaction for select queries for ACID tables? My understanding is that this approach would only work as long as the select query opens a transaction so that a compaction which runs immediately after we check does not remove the files after we mark them as not stale. http://gerrit.cloudera.org:8080/#/c/18043/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/18043/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@898 PS2, Line 898: List<PartitionRef> stalePartitions = directProvider_.checkLatestCompaction( : refImpl.dbName_, refImpl.tableName_, refImpl, refToMeta); looks like this is going to be called during each query's compilation when the query references a ACID table. Do you have a rough estimate of how much overhead per query are we introducing here? http://gerrit.cloudera.org:8080/#/c/18043/2/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java File fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java: http://gerrit.cloudera.org:8080/#/c/18043/2/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java@525 PS2, Line 525: /** : * Fetches the latest compaction id from HMS and compares with partition metadata in : * cache. If a partition is stale due to compaction, removes it from metas. : */ please also mention what is it that this method returns. Looks like we are returning all the PartitionRef which are deemed to be stale due to compaction. -- To view, visit http://gerrit.cloudera.org:8080/18043 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I173ea848917b6a41139b25b80677111463bfdc4b Gerrit-Change-Number: 18043 Gerrit-PatchSet: 2 Gerrit-Owner: Yu-Wen Lai <yu-wen....@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Sourabh Goyal <soura...@cloudera.com> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com> Gerrit-Reviewer: Yu-Wen Lai <yu-wen....@cloudera.com> Gerrit-Comment-Date: Wed, 24 Nov 2021 20:00:44 +0000 Gerrit-HasComments: Yes
