Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23942 )
Change subject: IMPALA-14230: Add catch-up mode for event processing ...................................................................... Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/23942/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/23942/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1978 PS4, Line 1978: > a. my understanding is that each coordinator request will start to load the > table if is an IncompleteTable so always replacing it with an IncompleteTable > during events can lead to parallel reloads on the table (not 100% sure here) Concurrent requests on the same unloaded table will just trigger one reload. TableLoadingMgr has some logics to handle this, e.g. https://github.com/apache/impala/blob/ff8bb33b91cd689bfaf2a51a19d0f615edd33d54/fe/src/main/java/org/apache/impala/catalog/TableLoadingMgr.java#L260-L266 https://github.com/apache/impala/blob/ff8bb33b91cd689bfaf2a51a19d0f615edd33d54/fe/src/main/java/org/apache/impala/catalog/TableLoadingMgr.java#L193-L196 (only for legacy catalog mode) > b. once the catch up threshold is reached, we start processing the events > again, and if those are slow, can easily go above the threshold again, and > the work done will be thrown out on the next event ... It seems inevitable if a large table is frequently used by queries and if event processing on it always take long, e.g. a read-heavy and write-heavy large table. The table will flip between two states (unloaded and loaded). But at least for other tables, users won't see stale metadata on them. For checking createEventId and lastRefreshEventId first, I think it's a good optimization. We can evaluate if it requires large changes. BTW, skipping events older than lastRefreshEventId is still disabled by default (IMPALA-14415). I think it's time to turn it on now. -- To view, visit http://gerrit.cloudera.org:8080/23942 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib906c06346d5d3159999eeac632e1318bc060065 Gerrit-Change-Number: 23942 Gerrit-PatchSet: 6 Gerrit-Owner: Yida Wu <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Yida Wu <[email protected]> Gerrit-Comment-Date: Wed, 25 Feb 2026 13:47:10 +0000 Gerrit-HasComments: Yes
