Yida Wu has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/23942 )
Change subject: IMPALA-14230: Add catch-up mode for event processing ...................................................................... IMPALA-14230: Add catch-up mode for event processing When HMS events fall far behind (e.g. during HA failover), event handling can become a serious bottleneck especially in the single threaded legacy mode, heavy operations such as table reload could take minutes that blocks the whole event processing. This patch introduces a configurable catch-up mode. If event lag exceeds the new added flag hms_event_catchup_threshold_s (default 1800s), the event processor switches to a faster way by invalidating the table instead of performing every event to avoid unnecessary heavy table reload to speed up the event processing. Catch-up mode is applied only to table events that may trigger heavy operations, like reloading or refreshing table and partition metadata, or operations that may involve severe lock contention. Lightweight events such as create/drop table and database events are excluded since they are already fast. Testing: Added test_catchup_mode_* tests. Passed exhaustive tests. Change-Id: Ib906c06346d5d3159999eeac632e1318bc060065 --- M be/src/catalog/catalog-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M tests/custom_cluster/test_events_custom_configs.py 8 files changed, 286 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/23942/5 -- To view, visit http://gerrit.cloudera.org:8080/23942 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib906c06346d5d3159999eeac632e1318bc060065 Gerrit-Change-Number: 23942 Gerrit-PatchSet: 5 Gerrit-Owner: Yida Wu <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Yida Wu <[email protected]>
