Anurag Mantripragada has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events. ...................................................................... IMPALA-7973: Add support for fine grained events processing for partition level HMS events. This patch adds support for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be expensive for large tables. Here, we refresh affected partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction, which means there may be multiple partitions affected in a single add/drop event. We try to refresh all these partitions in a loop. If any of the partition refresh fails, we throw MetastoreNotificationNeedsInvalidateException to mandate a manual invalidate for event processing to continue. Testing: Modified pre-existing tests for partition events to instead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 212 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/7 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 7 Gerrit-Owner: Anurag Mantripragada <anu...@cloudera.com> Gerrit-Reviewer: Anurag Mantripragada <anu...@cloudera.com> Gerrit-Reviewer: Bharath Krishna <bhar...@cloudera.com> Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Fredy Wijaya <fwij...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com>
