Hello k.venureddy2...@gmail.com, Sai Hemanth Gantasala, Csaba Ringhofer, Impala
Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21096
to look at the new patch set (#3).
Change subject: IMPALA-12855: Fix NPE in firing RELOAD events when the
partition doesn't exist
......................................................................
IMPALA-12855: Fix NPE in firing RELOAD events when the partition doesn't exist
When --enable_reload_events is set to true, catalogd will fire RELOAD
events for INVALIDATE/REFRESH statements. When the RELOAD event is fired
successfully for a REFRESH statement, we also update lastRefreshEventId
of the table/partition. This part could hit NullPointerException when
the partition is dropped by concurrent DDLs.
This patch ignores updating lastRefreshEventId if the partition doesn't
exists. Note that ideally we should hold the table lock of REFRESH until
finish firing the RELOAD events and updating lastRefreshEventId. So no
concurrent operations can drop the partition. However, when the table is
loaded from scratch, we don't actually hold the table write lock. We
just load the table and take a read lock to get the thrift object. The
partition could still be dropped concurrently after the load and before
taking the read lock. So ignoring missing partitions is a simpler
solution.
Refactors some codes of fireReloadEventAndUpdateRefreshEventId to save
some indention and avoid acquiring table lock if no events are fired.
Adds error messages in some Precondition checks in methods used by this
feature. Also refactors Table.getFullName() to not always constructing
the result. Improves logs of not reloading a partition for an event.
Tests:
- Add e2e test
Change-Id: I01af3624bf7cf5cd69935cffa28d54f6a6807504
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
6 files changed, 86 insertions(+), 46 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/21096/3
--
To view, visit http://gerrit.cloudera.org:8080/21096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I01af3624bf7cf5cd69935cffa28d54f6a6807504
Gerrit-Change-Number: 21096
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k.venureddy2...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <saihema...@cloudera.com>
