[ https://issues.apache.org/jira/browse/IMPALA-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17812214#comment-17812214 ]
Maxwell Guo commented on IMPALA-12771: -------------------------------------- with alter parition event, we may found that database/table is not found , or table is not loaded see [here |https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1057]. then we just skipped the and made a debug log , the log is : " Ignoring the event since the table is not found ". But actually, we skipped the handling of these events when table is not found .So I think we can also +1 on the events-skipped metric if table is not found or table is IncompleteTable or table was remove in catalog. Besides, we just mark the events-skipped metric for event process with isOlderEvent method and isSelfEvent see [isSelfEvent|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L857] and [isOlderEvent|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1198] But as for [canBeSkipped|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1703] method, the metric is not +1. I think we can also add here. > Impala catalogd events-skipped may mark the wrong number > -------------------------------------------------------- > > Key: IMPALA-12771 > URL: https://issues.apache.org/jira/browse/IMPALA-12771 > Project: IMPALA > Issue Type: Bug > Components: Catalog > Reporter: Maxwell Guo > Assignee: Maxwell Guo > Priority: Minor > > See the description of [event-skipped > metric|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L237] > > {code:java} > // total number of events which are skipped because of the flag setting or > // in case of [CREATE|DROP] events on [DATABASE|TABLE|PARTITION] which were > ignored > // because the [DATABASE|TABLE|PARTITION] was already [PRESENT|ABSENT] in > the catalogd. > {code} > > As for CREATE and DROP event on Database/Table/Partition (Also AddPartition > is inclued) when we found that the table/database when the database or table > is not found in the cache then we will skip the event process and make the > event-skipped metric +1. > But I found that there is some question here for alter table and Reload event: > * For alter table if renaming a table , the events-skipped metric will also > +1 ,see [oldTblRemoved to be false > |https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1653] > * For Reload event that is not describe in the description of events-skipped, > but the value is +1 when is oldevent; > * Besides if the table is in blacklist the metric will also +1 > In summary, I think this description is inconsistent with the actual > implementation. > So can we also mark the events-skipped metric for alter partition events and > modify the > description to be all the events skipped -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org