[
https://issues.apache.org/jira/browse/IMPALA-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17760244#comment-17760244
]
ASF subversion and git services commented on IMPALA-11535:
----------------------------------------------------------
Commit b718d63860356a04814e07d91711c3c748b3e769 in impala's branch
refs/heads/master from Sai Hemanth Gantasala
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=b718d6386 ]
IMPALA-11535: Skip older events in the event processor based on the
latestRefreshEventID
Summary: If the table has been manually refreshed, all its events
happen before the manual REFRESH can be skipped. This happens when
catalogd is lagging behind in processing events. When processing an
event, we can check whether there are manual REFRESH executed after
its eventTime. In such case, we don't need to process the event to
refresh anything. This helps catalogd to catch up HMS events quickly.
Implementation details: Updated the lastRefreshEventId on the table or
partition whenever there is table or partition level refresh/load.
By comparing the lastRefreshEventId to current event id in the event
processor the older events can be skipped.
set enable_skipping_older_events to true to enable this optimization
Testing:
- Unit end-to-end test and unit test to test the functionality.
Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Reviewed-on: http://gerrit.cloudera.org:8080/20022
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
> Skip events happen before manual REFRESH
> ----------------------------------------
>
> Key: IMPALA-11535
> URL: https://issues.apache.org/jira/browse/IMPALA-11535
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Quanlong Huang
> Assignee: Sai Hemanth Gantasala
> Priority: Critical
>
> If the table has been manually refreshed, all its events happen before the
> manual REFRESH can be skipped.
>
> This happens when catalogd is lagging behind in processing events. When
> processing an event, we can check whether there are manual REFRESH executed
> after its eventTime. In such case, we don't need to process the event to
> refresh anything. This helps catalogd to catch up HMS events quickly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
