[
https://issues.apache.org/jira/browse/IMPALA-11588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17726398#comment-17726398
]
ASF subversion and git services commented on IMPALA-11588:
----------------------------------------------------------
Commit 1cf8f5065acb79e4377492c6e67c312249e42a78 in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=1cf8f5065 ]
IMPALA-12053: Expose event-processor error message in WebUI
When the event-processor goes into the ERROR/NEEDS_INVALIDATE state, we
can only check logs to get the detailed information. This is
inconvenient in triaging failures. This patch exposes the error message
in the /events WebUI. It includes the timestamp string and the
stacktrace of the exception.
This patch makes the /events page visable. Also modifies the test code
of EventProcessorUtils.wait_for_synced_event_id() to print the error
message if the event processor is down.
A trivial bug of lastProcessedEvent is not updated (IMPALA-11588) is
also fixed in this patch. Refactored the variable to be a member of the
class so internal methods can update it before processing each event.
Some new metrics are not added in the /events page, e.g.
latest-event-id, latest-event-time-ms, last-synced-event-time-ms. This
patch addes them and also add a metric of event-processing-delay-ms
which is latest-event-time-ms minors last-synced-event-time-ms.
Tests:
- Manually inject codes to fail the event processor and verified the
WebUI.
- Ran metadata/test_event_processing.py when the event processor is in
ERROR state. Verified the error message is shown up in test output.
Change-Id: I077375422bc3d24eed57c95c6b05ac408228f083
Reviewed-on: http://gerrit.cloudera.org:8080/19916
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
> lastProcessedEvent is not updated in MetastoreEventsProcessor.processEvents()
> -----------------------------------------------------------------------------
>
> Key: IMPALA-11588
> URL: https://issues.apache.org/jira/browse/IMPALA-11588
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Major
>
> Code snipper:
> {code:java}
> public void processEvents() {
> NotificationEvent lastProcessedEvent = null;
> try {
> EventProcessorStatus currentStatus = eventProcessorStatus_;
> if (currentStatus != EventProcessorStatus.ACTIVE) {
> LOG.warn(String.format(
> "Event processing is skipped since status is %s. Last synced
> event id is %d",
> currentStatus, lastSyncedEventId_.get()));
> return;
> }
> List<NotificationEvent> events = getNextMetastoreEvents();
> processEvents(events);
> } catch (MetastoreNotificationFetchException ex) {
> // No need to change the EventProcessor state to error since we want the
> // EventProcessor to continue getting new events after HMS is back up.
> LOG.error("Unable to fetch the next batch of metastore events. Hive
> Metastore " +
> "may be unavailable. Will retry.", ex);
> } catch(MetastoreNotificationNeedsInvalidateException ex) {
> updateStatus(EventProcessorStatus.NEEDS_INVALIDATE);
> LOG.error("Event processing needs a invalidate command to resolve the
> state", ex);
> } catch (Exception ex) {
> // There are lot of Preconditions which can throw RuntimeExceptions
> when we
> // process events this catch all exception block is needed so that the
> scheduler
> // thread does not die silently
> updateStatus(EventProcessorStatus.ERROR);
> LOG.error("Unexpected exception received while processing event", ex);
> dumpEventInfoToLog(lastProcessedEvent);
> }
> }
> {code}
> https://github.com/apache/impala/blob/cfd79b40beab86f08ad72e0bea41eabf736d0a99/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L838-L865
> "lastProcessedEvent" is a local variable and never updated so it's always
> null.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
