[ https://issues.apache.org/jira/browse/IMPALA-12468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maxwell Guo updated IMPALA-12468: --------------------------------- Description: Once the impala and hive's status is missmatched , and the EventProcessorStatus become NEED_INVALIDATE, we usually use invalidate metadata to reset the catalog instance. And then impala will update the status to ACTIVE . But if impala contains many tables , the cost of invalidate is a bit high for a global invalidate. So we may invalidate metadata for tables one by one for these incremental changed table. For example , we have 1000,000,000,000 tables but only some of the table event process occurs CatalogException and MetastoreNotificationNeedsInvalidateException was thrown. I think there is no need to invalidate all table caches in order to reset the catalog instance see [here |https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2088]. MetaStoresProcessor 's async update process will not update the currentStatus when the status is not ACTIVE, see [here|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L876] So what about add a new SQL grammar : SET EVENT STATUS ${status} ? was: Once the impala and hive's status is missmatched , and the EventProcessorStatus become NEED_INVALIDATE, we usually use invalidate metadata to reset the catalog instance. And then impala will update the status to ACTIVE . But if impala contains many tables , the cost of invalidate is a bit high for a global invalidate. So we may invalidate metadata for tables one by one for these incremental changed table. For example , we have 1000,000,000,000 tables but only some of the table event process occurs CatalogException and MetastoreNotificationNeedsInvalidateException was thrown. I think there is no need to invalidate all table caches in order to reset the catalog instance see [here |https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2088]. MetaStoresProcessor 's async update process will not update the currentStatus when the status is not ACTIVE, see [here|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L876] So what about add a new SQL grammar : RESET STATUS status ? > Add the ability to update EventProcessorStatus > ---------------------------------------------- > > Key: IMPALA-12468 > URL: https://issues.apache.org/jira/browse/IMPALA-12468 > Project: IMPALA > Issue Type: Improvement > Components: be, Catalog, fe > Reporter: Maxwell Guo > Assignee: Maxwell Guo > Priority: Minor > > Once the impala and hive's status is missmatched , and the > EventProcessorStatus become NEED_INVALIDATE, we usually use invalidate > metadata to reset the catalog instance. And then impala will update the > status to ACTIVE . > But if impala contains many tables , the cost of invalidate is a bit high for > a global invalidate. So we may invalidate metadata for tables one by one for > these incremental changed table. For example , we have 1000,000,000,000 > tables but only some of the table event process occurs CatalogException and > MetastoreNotificationNeedsInvalidateException was thrown. I think there is no > need to invalidate all table caches in order to reset the catalog instance > see [here > |https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2088]. > > MetaStoresProcessor 's async update process will not update the currentStatus > when the status is not ACTIVE, see > [here|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L876] > So what about add a new SQL grammar : SET EVENT STATUS ${status} ? -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org