Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18937 )
Change subject: IMPALA-11490: Add more metrics for event processor ...................................................................... Patch Set 2: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/18937/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/18937/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@878 PS2, Line 878: CurrentNotificationEventId currentNotificationEventId = : msClient.getHiveClient().getCurrentNotificationEventId(); Can't this make a slow HMS even slower? If I understand correctly these calls become database queries in HMS. https://github.com/apache/hive/blob/4e4e39c471094567dcdfd9840edbd99d7eafc230/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L11846 https://github.com/apache/hive/blob/4e4e39c471094567dcdfd9840edbd99d7eafc230/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java#L2603 The idea would be to poll the new events as we are doing it now and keep a queue in Impala, but do the processing on a different thread. This way we could get an update every pollingFrequencyInSec_ without increasing HMS load when the processing is up to date. -- To view, visit http://gerrit.cloudera.org:8080/18937 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0e7d40a0d8e140e6b0698936e97b454cb9abdc1b Gerrit-Change-Number: 18937 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Yu-Wen Lai <yu-wen....@cloudera.com> Gerrit-Comment-Date: Wed, 07 Sep 2022 16:11:10 +0000 Gerrit-HasComments: Yes
