Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20986 )
Change subject: IMPALA-12782: Show info of the event processing in /events webUI ...................................................................... Patch Set 8: (5 comments) http://gerrit.cloudera.org:8080/#/c/20986/8//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20986/8//COMMIT_MSG@12 PS8, Line 12: processing > nit: processed Done http://gerrit.cloudera.org:8080/#/c/20986/8/be/src/catalog/catalog-server.cc File be/src/catalog/catalog-server.cc: http://gerrit.cloudera.org:8080/#/c/20986/8/be/src/catalog/catalog-server.cc@980 PS8, Line 980: last_synced_event_id > idea to reduce the chattiness of the code: Good point! Added such a wrapper in be/src/util/json-util.h http://gerrit.cloudera.org:8080/#/c/20986/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/20986/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1192 PS3, Line 1192: * @throws MetastoreNotificationException > One member when this seems problematic is currentEventIndex_ - if there is Starts from PS6, resetProgressInfo() is called in start() so I think the metrics are good when the event processor was restarted. I also thought about adding a new field to track the last failed event and always show it in the WebUI. But it's a bit confusing when the event processor is actually active (after restart). The current section of "Error Message" is moved to the top of the /events page to indicate the status of the event processor. So I thought it would be better to add a new section about the event processing history. In that section we can show the top-10 expensive events, top-10 expensive tables, and the recent 10 failures, etc. Do you think it's ok to do it in a new JIRA? http://gerrit.cloudera.org:8080/#/c/20986/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/20986/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1146 PS8, Line 1146: in case they are replaced : // when being used below > "in case they are replaced concurrently"? Done http://gerrit.cloudera.org:8080/#/c/20986/8/tests/custom_cluster/test_web_pages.py File tests/custom_cluster/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/20986/8/tests/custom_cluster/test_web_pages.py@444 PS8, Line 444: catalogd_event_processing_delay > Probably out of the scope of this patch, but a potential alternative would Good point! Filed IMPALA-12901 for this. -- To view, visit http://gerrit.cloudera.org:8080/20986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e7d4952c7fd04ae89b6751204499bf9dd99f57c Gerrit-Change-Number: 20986 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Anonymous Coward <k.venureddy2...@gmail.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Sai Hemanth Gantasala <saihema...@cloudera.com> Gerrit-Comment-Date: Thu, 14 Mar 2024 03:54:40 +0000 Gerrit-HasComments: Yes
