Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17848 )
Change subject: IMPALA-9857: Batching of consecutive partition events ...................................................................... Patch Set 5: (3 comments) http://gerrit.cloudera.org:8080/#/c/17848/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/17848/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2713 PS5, Line 2713: return hmsPartitions.size(); Shouldn't we return hmsPartToHdfsPart.size() since those are the actual partitions we are reloading in the cache? http://gerrit.cloudera.org:8080/#/c/17848/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/17848/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@279 PS2, Line 279: if (!current.canBeBatched(next)) { Currently one of the conditions of batching two events together is - event ids should be consecutive. This means that if there are events intermingled from multiple tables, then this batching logic won't be very effective. Do we see intermingled events in production? If yes, should we improve the batching logic here? http://gerrit.cloudera.org:8080/#/c/17848/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1641 PS2, Line 1641: MetastoreEventPropertyKey.CATALOG_VERSION.getKey(), "-1")); > Since the default value of polling is 1s and we fetch the events in the bat I understand that there may not be significant performance gain. I still feel that we should batch alter partition events from other clusters as well (and ignore self events when actually processing a batched event) considering that we would be syncing table till latest event id in future where self event logic will not be applicable anymore. -- To view, visit http://gerrit.cloudera.org:8080/17848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5d27a68a64436d31731e9a219b1efd6fc842de73 Gerrit-Change-Number: 17848 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar <vih...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Sourabh Goyal <soura...@cloudera.com> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com> Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com> Gerrit-Comment-Date: Mon, 20 Sep 2021 20:31:43 +0000 Gerrit-HasComments: Yes
