Sai Hemanth Gantasala has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20022
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID ...................................................................... IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID Summary: If the table has been manually refreshed, all its events happen before the manual REFRESH can be skipped. This happens when catalogd is lagging behind in processing events. When processing an event, we can check whether there are manual REFRESH executed after its eventTime. In such case, we don't need to process the event to refresh anything. This helps catalogd to catch up HMS events quickly. Implementation details: Updated the lastRefreshEventId on the table or partition whenever there is table or partition level refresh/load. By comparing the lastRefreshEventId to current event id in the event processor the older events can be skipped. Testing: - Unit end-to-end test and unit test to test the functionality. Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1 --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M tests/custom_cluster/test_events_custom_configs.py 5 files changed, 128 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/20022/1 -- To view, visit http://gerrit.cloudera.org:8080/20022 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1 Gerrit-Change-Number: 20022 Gerrit-PatchSet: 1 Gerrit-Owner: Sai Hemanth Gantasala <saihema...@cloudera.com>
