[email protected] has posted comments on this change. ( http://gerrit.cloudera.org:8080/22997 )
Change subject: IMPALA-13801: Support greatest synced event with hierarchical metastore event processing ...................................................................... Patch Set 13: (2 comments) http://gerrit.cloudera.org:8080/#/c/22997/12/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java: http://gerrit.cloudera.org:8080/#/c/22997/12/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@1156 PS12, Line 1156: new PseudoCommitTxnEvent(context, tbl.getDbName(), tbl.getTableName(), : tbl.getPartitionKeysSize() > 0, : MetaStoreUtils > Please compare between Catalog.getTable() vs Catalog.getTableNoThrow(). Correct. Need to use getTableNoThrow(). Have modified it. getCatalog().getTable() itself may return null here in the following cases: 1. If the create database or table events are in the same batch as the committxn event and those create events are not processed yet. 2. If the create database or table events are skipped. With IMPALA-12785, we can skip some events. For example, with following steps, we can skip create database/table events: 1. :event_processor('pause') 2. create database or table from hive beeline. 3. :event_processor('start', -1) i.e., start from the latest event id http://gerrit.cloudera.org:8080/#/c/22997/11/fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java File fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java: http://gerrit.cloudera.org:8080/#/c/22997/11/fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java@323 PS11, Line 323: // Throwing EventProcessException triggers global invalidate > Can you double check this statement please? MetastoreEventsProcessor#handleEventProcessException() has no relation to eventExecutorService_.cleanup(); EventExecutorService#cleanup() is called from MetastoreEventsProcessor#processEvents() as a regular cleanup process to remove the stale DbProcessors if any after processing events received in the batch. A DbProcessor is removed iff DbEventExecutor.DbProcessor#canBeRemoved() returns true. This happens from events dispatch thread. Upon event process exception, catalog reset happens. It calls MetastoreEventsProcessor#clear() which internally calls EventExecutorService#clear() to clear all the DbProcessors and TableProcessors on all the DbEventExecutors and TableEventExecutors. -- To view, visit http://gerrit.cloudera.org:8080/22997 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26240f36aaf85125428dc39a66a2a1e4d3197e85 Gerrit-Change-Number: 22997 Gerrit-PatchSet: 13 Gerrit-Owner: Anonymous Coward <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Comment-Date: Wed, 20 Aug 2025 19:19:36 +0000 Gerrit-HasComments: Yes
