Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/23942 )
Change subject: IMPALA-14230: Add catch-up mode for event processing ...................................................................... Patch Set 5: (5 comments) http://gerrit.cloudera.org:8080/#/c/23942/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/23942/4//COMMIT_MSG@20 PS4, Line 20: trigger heavy : operations, > This always mean loading/refreshing tables/partitions, or there are also ot Yes, I think these are operations we want to avoid, and maybe the local operations that involve lock stuff. Added commit message here. http://gerrit.cloudera.org:8080/#/c/23942/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/23942/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1242 PS4, Line 1242: long lag = evaluateCatchUpLag(); : if (lag > 0) { : if (catalog_.i > I don't understand this "if". Yeah, I was hesitated for this because maybe it is not that wrong to log when the table is already invalidated even though it is not invalidated by this action. I've updated invalidateTableIfExists() to return null when skipIfIncomplete is true and the table is already incomplete. This will make the logging more correct. http://gerrit.cloudera.org:8080/#/c/23942/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1978 PS4, Line 1978: > Is it a good idea to evaluate this before isSelfEvent() and isOlderEvent()? Added some comments. http://gerrit.cloudera.org:8080/#/c/23942/4/tests/custom_cluster/test_events_custom_configs.py File tests/custom_cluster/test_events_custom_configs.py: http://gerrit.cloudera.org:8080/#/c/23942/4/tests/custom_cluster/test_events_custom_configs.py@728 PS4, Line 728: onfigured thresh > Is there a reason for creating the table in Hive? Thought transactional table needs hive, but it seems not necessary. Changed it to impala. http://gerrit.cloudera.org:8080/#/c/23942/4/tests/custom_cluster/test_events_custom_configs.py@760 PS4, Line 760: "--debug_actions=catalogd_event_processing_delay:SLEEP@3 > Shouldn't the test also check if the given event type invalidates the table That makes sense. Made the change to catch invalidated logs. -- To view, visit http://gerrit.cloudera.org:8080/23942 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib906c06346d5d3159999eeac632e1318bc060065 Gerrit-Change-Number: 23942 Gerrit-PatchSet: 5 Gerrit-Owner: Yida Wu <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Yida Wu <[email protected]> Gerrit-Comment-Date: Tue, 24 Feb 2026 21:02:12 +0000 Gerrit-HasComments: Yes
