Hello Quanlong Huang, k.venureddy2...@gmail.com, Csaba Ringhofer, Impala Public
Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20367
to look at the new patch set (#3).
Change subject: IMPALA-10976: Sync db/table in catalogD to latest HMS event id
for all DDLS from Impala clients
......................................................................
IMPALA-10976: Sync db/table in catalogD to latest
HMS event id for all DDLS from Impala clients
The idea is that when any DDL operation is performed by Impala, it also
syncs the db/table to its latest event ID as per HMS. This way updates
to a db/table's are applied in the same order as they appear in the
Notification log table in HMS which ensures consistency. Currently
catalogD applies any updates received from Impala clients in-place.
Instead it should perform an HMS operation first and then replay all
the HMS events since the last synced event id.
Implementation: when the enable_sync_to_latest_event_on_ddls flag is
set to true, we fetch the latest snapshot of db/table and update it in
the catalogD's cache. Once HIVE-27499 is implemented, we can update the
snapshot only if there are any pending events. Currently, there is no
efficient way to identify if there are pending events for a db/table.
Set enable_sync_to_latest_event_on_ddls to true to use this feature.
Testing:
Added few tests in the MetaStoreEventProcessorForTest to verify this
feature.
Change-Id: Ia250d0a943838086c187e5cb7c60035e5a564bbf
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
4 files changed, 150 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/20367/3
--
To view, visit http://gerrit.cloudera.org:8080/20367
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia250d0a943838086c187e5cb7c60035e5a564bbf
Gerrit-Change-Number: 20367
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <saihema...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k.venureddy2...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <saihema...@cloudera.com>
