Hello k.venureddy2...@gmail.com, Sai Hemanth Gantasala, Csaba Ringhofer, Impala
Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21186
to look at the new patch set (#11).
Change subject: IMPALA-12933: Avoid fetching unneccessary events of unwanted
types
......................................................................
IMPALA-12933: Avoid fetching unneccessary events of unwanted types
There are several places where catalogd will fetch all events of a
specific type on a table. E.g. in TableLoader#load(), if the table has
an old createEventId, catalogd will fetch all CREATE_TABLE events after
that createEventId on the table.
Fetching the list of events is expensive since the filtering is done on
client side, i.e. catalogd fetches all events and filter them locally
based on the event type and table name. This could take hours if there
are lots of events (e.g 1M) in HMS.
This patch sets the eventTypeSkipList with the complement set of the
wanted type. So the get_next_notification RPC can filter out some events
on HMS side. To avoid bringing too much computation overhead to HMS's
underlying RDBMS in evaluating predicates of EVENT_TYPE != 'xxx', rare
event types (e.g. DROP_ISCHEMA) are not added in the list. A new flag,
common_hms_event_types, is added to specify the common HMS event types.
Once HIVE-28146 is resolved, we can set the wanted types directly in the
HMS RPC. This approach can be simplified.
UPDATE_TBL_COL_STAT_EVENT, UPDATE_PART_COL_STAT_EVENT are the most
common unused events for Impala. They are also added to the default skip
list. A new flag, common_hms_event_types, is added to configure this
list.
This patch also fixes an issue that events of the non-default catalog
are not filtered out.
In a local perf test, I generated 100K RELOAD events after creating a
table in Hive. Then use the table in Impala to trigger metadata loading
on it which will fetch the latest CREATE_TABLE event by polling all
events after the last known CREATE_TABLE event. Before this patch,
fetching the events takes 1s779ms. Now it takes only 395.377ms. Note
that in prod env, the event messages are usually larger, we could have
a larger speedup.
Tests:
- Added an FE test
- Ran CORE tests
Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M
fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java
M
fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
15 files changed, 285 insertions(+), 115 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/21186/11
--
To view, visit http://gerrit.cloudera.org:8080/21186
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ieabe714328aa2cc605cb62b85ae8aa4bd537dbe9
Gerrit-Change-Number: 21186
Gerrit-PatchSet: 11
Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k.venureddy2...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <saihema...@cloudera.com>
