Anurag Mantripragada has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12889 )
Change subject: IMPALA-7971: Add support for insert events in event processor. ...................................................................... IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. The event-processor will refresh tables using this event. Both insert into and overwrite are supported for tables/partitions. Also, renamed TableInvalidatingEvent class to TableInvalidatingOrRefreshingEvent to reflect new behaviour. Known Issues: 1. There is an unnecessary table invalidate when insert is done in Hive as the insert operation creates an ALTER and an INSERT notification event. Currently there is no way for the Event Processor to identify and prevent the unnecessary invalidate. IMPALA-7973 may potentially solve this issue. 2. Detection of self-events does not work for inserts currently because of the way the self-event checks are implemented. The flags added to test for self events have no way to persist in HMS with just an insert operation. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: Wrote new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A tests/custom_cluster/test_event_processing.py 7 files changed, 299 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/3 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada <anu...@cloudera.com> Gerrit-Reviewer: Bharath Krishna <bhar...@cloudera.com> Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Paul Rogers <prog...@cloudera.com> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com>
