Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/22554 )
Change subject: IMPALA-13593: Enable event processor to consume ALTER_PARTITIONS events from metastore ...................................................................... Patch Set 16: (8 comments) http://gerrit.cloudera.org:8080/#/c/22554/11//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/22554/11//COMMIT_MSG@19 PS11, Line 19: . This can cause partitions to be refreshed even on : trivial changes to them. > Thanks for the explanation? Created https://issues.apache.org/jira/browse/HIVE-29141 to track this issue. http://gerrit.cloudera.org:8080/#/c/22554/15/bin/impala-config.sh File bin/impala-config.sh: http://gerrit.cloudera.org:8080/#/c/22554/15/bin/impala-config.sh@243 PS15, Line 243: export CDP_BUILD_NUMBER=66846208 > It looks like we should be good to use CDP_BUILD_NUMBER=66846208 now (7.3.1 Ack http://gerrit.cloudera.org:8080/#/c/22554/15/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java File fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java: http://gerrit.cloudera.org:8080/#/c/22554/15/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@476 PS15, Line 476: getPartitions > getPartitionsAfter()? This method is for TableReload class, it make sense to keep getPartitions() as the name of the method: https://gerrit.cloudera.org/c/22554/15/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java#1158 http://gerrit.cloudera.org:8080/#/c/22554/15/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java: http://gerrit.cloudera.org:8080/#/c/22554/15/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@611 PS15, Line 611: returns them in a Alte > this should be updated Ack http://gerrit.cloudera.org:8080/#/c/22554/15/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@1132 PS15, Line 1132: > I don't get the naming - this class seems specific to ALTER_PARTITIONS, but I initially wanted to use to for ReloadEvent as well but it seems it is not useful as the fields are different. I'll rename the class appropriately. http://gerrit.cloudera.org:8080/#/c/22554/15/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@1150 PS15, Line 1150: > Maybe getMsTableAfter would be more exact? There'll be only one table object for the alter partition event. That's the naming convention we have followed in other places. http://gerrit.cloudera.org:8080/#/c/22554/15/tests/custom_cluster/test_events_custom_configs.py File tests/custom_cluster/test_events_custom_configs.py: http://gerrit.cloudera.org:8080/#/c/22554/15/tests/custom_cluster/test_events_custom_configs.py@1677 PS15, Line 1677: > Can we also verify to the total number of events generated by ANALYZE TABLE The number of events for managed and external tables varies. Also, it might cause some flakiness in the future if some transactional events don't make it in time. I think verifying no batches were formed and the generation of ALTER_PARTITIONS event is good enough for this test. Yeah, analyze table compute statistics will call alter-table api in hive metastore, which generates alter partitions events as they can update stats in metastore. http://gerrit.cloudera.org:8080/#/c/22554/15/tests/custom_cluster/test_events_custom_configs.py@1704 PS15, Line 1704: events = EventProcessorUtils.get_next_notification(self.hive_client, last_event_id) : events_skipped_after = EventProcessorUtils.get_int_metric('events-skipped', 0) : assert _verify_alter_partitions_event(events) : assert events_skipped_after > events_skipped_before : : # Case-IV: Truncate table from Hive is currently generating single alter_partition : # events. HIVE-286 > The same is done twice, there could be a function for it Ack -- To view, visit http://gerrit.cloudera.org:8080/22554 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I009a87ef5e2c331272f9e2d7a6342cc860e64737 Gerrit-Change-Number: 22554 Gerrit-PatchSet: 16 Gerrit-Owner: Sai Hemanth Gantasala <saihema...@cloudera.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Daniel Becker <daniel.bec...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Michael Smith <michael.sm...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Sai Hemanth Gantasala <saihema...@cloudera.com> Gerrit-Comment-Date: Tue, 12 Aug 2025 22:56:33 +0000 Gerrit-HasComments: Yes
