Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21326 )
Change subject: IMPALA-13009: Fix catalogd not sending deletion updates for some dropped partitions ...................................................................... Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/21326/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/21326/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1048 PS3, Line 1048: TCatalogObject catalog = Extracted this part into a method. http://gerrit.cloudera.org:8080/#/c/21326/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1082 PS3, Line 1082: return; We should only add the deletion if it's not in the updates (same as L1065), i.e. if there is a new version of the partition collected, don't add the deletion. When a partition is updated, the old version is added to the droppedPartitions set, while the new version is in partitionsMap. When the table is invalidated and then reloaded, the old HdfsTable instance will be added to the deleteLog. If these all happen in a catalog update cycle, catalogd will collect updates of the partition. We should ignore its old version which is inside the droppedPartitions of the old HdfsTable instance. This causes tests like metadata/test_recover_partitions.py::TestRecoverPartitions::test_post_invalidate hanging forever. -- To view, visit http://gerrit.cloudera.org:8080/21326 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I12a68158dca18ee48c9564ea16b7484c9f5b5d21 Gerrit-Change-Number: 21326 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Fang-Yu Rao <fangyu....@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Comment-Date: Fri, 19 Apr 2024 11:53:46 +0000 Gerrit-HasComments: Yes
