Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21031 )
Change subject: IMPALA-12709: Add support for hierarchical metastore event processing ...................................................................... Patch Set 38: (2 comments) http://gerrit.cloudera.org:8080/#/c/21031/38/common/thrift/metrics.json File common/thrift/metrics.json: http://gerrit.cloudera.org:8080/#/c/21031/38/common/thrift/metrics.json@3656 PS38, Line 3656: "key" : "events-processor.outstanding-event-count" > nit: Will be interesting to add a 2 more granular counter: Please add documentation for new metrics in docs/topics/impala_metadata.xml http://gerrit.cloudera.org:8080/#/c/21031/38/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/21031/38/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@1463 PS38, Line 1463: lastSyncedEventId_.set(event.getEventId()); I understand that eventExecutorService_ enable us from processing events in the backgroud and get out of this synchronized block early. However, for this initial implementation, is it make sense if we spin loop to only exit the synchronized block only after getOutstandingEventCount() == 0? This will give us initial guarantee that processEvents fully finished processing one batch of events before moving on to the next batch. Processing events within single batch will still be parallelized. If it works well, then we can remove such spin loop in next patch. -- To view, visit http://gerrit.cloudera.org:8080/21031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I76d8a739f9db6d40f01028bfd786a85d83f9e5d6 Gerrit-Change-Number: 21031 Gerrit-PatchSet: 38 Gerrit-Owner: Anonymous Coward <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Comment-Date: Sat, 01 Mar 2025 22:02:36 +0000 Gerrit-HasComments: Yes
