Vihang Karajgaonkar has uploaded a new patch set (#20). ( http://gerrit.cloudera.org:8080/17308 )
Change subject: IMPALA-10502: Handle CREATE/DROP events correctly ...................................................................... IMPALA-10502: Handle CREATE/DROP events correctly The current way to detect self-events in case of CREATE/DROP events on database, table and partition is problematic when the same object is created and dropped repeatedly in quick succession. This happens mainly due to couple of reasons. For example if we have the below sequence of DDLs in Impala: 1. create table foo; --> catalogd creates table foo 2. drop table foo; --> catalogd drops table foo ... Events processor receives the CREATE_TABLE event pertaining to (1) above. Now it cannot determine whether the table needs to be created or not. Similarly, if we interchange the order of DROP and CREATE statements above, the DROP_TABLE event received by the events processor will unnecessarily remove the table when it should not. This can cause problems for queries which expect the table to exist or not exist. E.g create table query fails with a table already exists or a drop table query fails with table does not exist error. In order to fix this issue, catalogd now keeps track of dropped objects in a deleteLog which are garbage collected as the events come in. Every time a database, table or partition is dropped, the deleteLog is populated with the drop event id generated due to the drop operation. This deleteLog is looked up when the event is received to determine if the event can be ignored. Additionally, catalogd keeps track of the create event id at the Database, Table or Partition level during the create DDL execution so that the event can be ignored later by events processor. Testing: 1. Added test_create_drop_events and test_local_catalog_create_drop_events test which loops to create create/drop events for database, table and partitions. 2. Added new metrics which the test verifies to ensure that events don't create or drop the object. Change-Id: Ia2c5e96b48abac015240f20295b3ec3b1d71f24a --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/TableLoader.java M fe/src/main/java/org/apache/impala/catalog/TableLoadingMgr.java A fe/src/main/java/org/apache/impala/catalog/events/DeleteEventLog.java A fe/src/main/java/org/apache/impala/catalog/events/EventFactory.java M fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/events/InFlightEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServer.java M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java A fe/src/test/java/org/apache/impala/catalog/metastore/AbstractCatalogMetastoreTest.java M fe/src/test/java/org/apache/impala/catalog/metastore/CatalogHmsFileMetadataTest.java M fe/src/test/java/org/apache/impala/catalog/metastore/EnableCatalogdHmsCacheFlagTest.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java A fe/src/test/java/org/apache/impala/testutil/CatalogTestMetastoreServer.java M tests/custom_cluster/test_event_processing.py 29 files changed, 2,648 insertions(+), 1,180 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/17308/20 -- To view, visit http://gerrit.cloudera.org:8080/17308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia2c5e96b48abac015240f20295b3ec3b1d71f24a Gerrit-Change-Number: 17308 Gerrit-PatchSet: 20 Gerrit-Owner: Vihang Karajgaonkar <vih...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Sourabh Goyal <soura...@cloudera.com> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com> Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
