Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20507 )
Change subject: IMPALA-12460: Add lag and histogram of event processing in the log ...................................................................... Patch Set 6: (7 comments) http://gerrit.cloudera.org:8080/#/c/20507/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20507/5//COMMIT_MSG@12 PS5, Line 12: mins > Nit: line too long. Done http://gerrit.cloudera.org:8080/#/c/20507/5//COMMIT_MSG@12 PS5, Line 12: the top-10 targets that c > Nit: _the_ top-10 targets _that_ contribute Done http://gerrit.cloudera.org:8080/#/c/20507/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/20507/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@503 PS5, Line 503: if (tblName_ == null) return dbName_; > Could add a Precondition check that 'dbName_' is not null in this case. If 'dbName_' is also null in this case, it should be captured at the previous line. http://gerrit.cloudera.org:8080/#/c/20507/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/20507/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@948 PS5, Line 948: {} > We could indicate in the log that it's in milliseconds. PrintUtils.printTimeMs() returns a human readable time string, e.g. 43s961ms. I think it's clear enough. http://gerrit.cloudera.org:8080/#/c/20507/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1102 PS5, Line 1102: eventList.sort(Map.Entry.<MetastoreEvent, Long>comparingByValue().reversed()); > Can there be too many events and sorting all map entries consumes more memo That's a good point. The event batch size is currently hard coded as 1000 so we are good currently: https://github.com/apache/impala/blob/2d3289027c2ffdd245d13b60e6fa3f9b3e7bf833/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L218 We can add this optimization when we make the event batch configurable in the future. http://gerrit.cloudera.org:8080/#/c/20507/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1106 PS5, Line 1106: Map.Entry<MetastoreEvent, Long> entry = eventList.get(i); > Wouldn't it be more convenient to order it in descending order, for example Yeah, I tried reversed() and found I need to cast the result to Comparator<? super Map.Entry<MetastoreEvent, Long>> which looks verbose. Finally figure the correct syntax to use reverse(): Map.Entry.<MetastoreEvent, Long>comparingByValue().reversed() http://gerrit.cloudera.org:8080/#/c/20507/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1128 PS5, Line 1128: long durationMs = entry.getValue(); > See L1106. Done -- To view, visit http://gerrit.cloudera.org:8080/20507 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib9421b5e26bfa2324217ec9695fbd81636727d22 Gerrit-Change-Number: 20507 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Anonymous Coward <k.venureddy2...@gmail.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Daniel Becker <daniel.bec...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Sai Hemanth Gantasala <saihema...@cloudera.com> Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com> Gerrit-Comment-Date: Sat, 30 Sep 2023 05:52:12 +0000 Gerrit-HasComments: Yes
