Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID ...................................................................... Patch Set 12: (5 comments) http://gerrit.cloudera.org:8080/#/c/20022/12/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/20022/12/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3229 PS12, Line 3229: Long nit: long BTW, why we don't verify 'lastSyncEventIdBefore' now? http://gerrit.cloudera.org:8080/#/c/20022/12/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3239 PS12, Line 3239: assertTrue nit: use assertEquals() http://gerrit.cloudera.org:8080/#/c/20022/11/tests/custom_cluster/test_events_custom_configs.py File tests/custom_cluster/test_events_custom_configs.py: http://gerrit.cloudera.org:8080/#/c/20022/11/tests/custom_cluster/test_events_custom_configs.py@345 PS11, Line 345: self.run_stmt_in_hive( : "insert into {db}.{tbl} partition (year=2024) values (10),(20),(30)" : .format(db=unique_database, tbl=test_old_table_1)) : self.run_stmt_in_hive( : "insert into {db}.{tbl} partition (year=2024) values (10),(20),(30)" : .format(db=unique_database, tbl=test_old_table_1)) : self.run_stmt_in_hive( : "insert into {db}.{tbl} partition (year=2024) values (10),(20),(30)" : .format(db=unique_database, tbl=test_old_table_1)) : self.client.execute( : "refresh {}.{}".format(unique_database, test_old_table_1)) > The test is running fine. I ran the test 10 times consecutively and it pass Ack http://gerrit.cloudera.org:8080/#/c/20022/12/tests/custom_cluster/test_events_custom_configs.py File tests/custom_cluster/test_events_custom_configs.py: http://gerrit.cloudera.org:8080/#/c/20022/12/tests/custom_cluster/test_events_custom_configs.py@374 PS12, Line 374: "create transactional table {}.{} (i int) partitioned by (year int)" Can we also test for non-partitioned tables? We can extract a common method to test on the 4 cases of transactional or not and partitioned or not. The method creates the table, then alters and inserts it in Hive and refreshes it in Impala, verify no reloads happen. http://gerrit.cloudera.org:8080/#/c/20022/12/tests/custom_cluster/test_events_custom_configs.py@384 PS12, Line 384: assert tbl_events_skipped_after > tbl_events_skipped_before Let's verify no reloads happen by comparing the old and new metrics of "tables-refreshed" and "partitions-refreshed". -- To view, visit http://gerrit.cloudera.org:8080/20022 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1 Gerrit-Change-Number: 20022 Gerrit-PatchSet: 12 Gerrit-Owner: Sai Hemanth Gantasala <saihema...@cloudera.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Sai Hemanth Gantasala <saihema...@cloudera.com> Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com> Gerrit-Comment-Date: Wed, 23 Aug 2023 10:54:50 +0000 Gerrit-HasComments: Yes
