Vihang Karajgaonkar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12549
Change subject: IMPALA-7975 : Improve supportability of the automatic invalidate feature ...................................................................... IMPALA-7975 : Improve supportability of the automatic invalidate feature This patch adds supportability metrics to make it easier to find the state of Metastore events processor. It adds the following metrics 1. Total number of events received so far This is not super useful in itself but is helpful to see if there are events being received or not. A more usable metric which publishes the rate of such events received is also published. See details for it below. 2. 1min, 5min and 15min rate of events received Exponential moving average of number of events received in last 1min, 5min and 15min. This can be used to determine if there are spikes in event processor activity during certain hours of the day 3. Total number of events skipped so far Events can be skipped based on certain flags are table and database level. This metric keeps a count of total number of events skipped so far. Useful to make decisions like - If most of the events are being skipped, may be just turn off the event processing. If most of the events are not skipped, do we need to add flags on certain databases? 4. Status of events processor Easy way to determine what is the current state of metastore events processor. Possible states could be STOPPED, ACTIVE, ERROR, NEEDS_INVALIDATE 5. Events fetch and process durations in seconds Average duration to fetch a batch of events and process it. Additionally, it also registers a url handler specific for events at /events. The /events page provides a more detailed view of the metrics of the event processor which include (min, max, mean, median) of the durations and rate metrics for all the counters. In the future this page can be enhanced to show the top-n tables with most number of invalidates due to the events, etc. Patch adds few tests to make sure the value of these event metrics are as expected. Change-Id: I23cb3aa866879eca03c64ab881796eaa9caa0337 --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/catalog/catalogd-main.cc M be/src/util/CMakeLists.txt A be/src/util/event-metrics.cc A be/src/util/event-metrics.h M common/thrift/JniCatalog.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java M fe/src/main/java/org/apache/impala/common/Metrics.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A www/events.tmpl 19 files changed, 621 insertions(+), 66 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/12549/2 -- To view, visit http://gerrit.cloudera.org:8080/12549 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I23cb3aa866879eca03c64ab881796eaa9caa0337 Gerrit-Change-Number: 12549 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar <vih...@cloudera.com>
