[ https://issues.apache.org/jira/browse/IMPALA-7971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829851#comment-16829851 ]
ASF subversion and git services commented on IMPALA-7971: --------------------------------------------------------- Commit 15a33d1baaf4723794e52825a5c4082ff7487507 in impala's branch refs/heads/master from Anurag Mantripragada [ https://gitbox.apache.org/repos/asf?p=impala.git;h=15a33d1 ] IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Reviewed-on: http://gerrit.cloudera.org:8080/12889 Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Reviewed-by: Bharath Vissapragada <bhara...@cloudera.com> > Add support to detect insert events from Impala > ----------------------------------------------- > > Key: IMPALA-7971 > URL: https://issues.apache.org/jira/browse/IMPALA-7971 > Project: IMPALA > Issue Type: Sub-task > Reporter: Vihang Karajgaonkar > Assignee: Anurag Mantripragada > Priority: Major > > When data is inserted into existing tables and partitions, Catalog does not > issue any metastore API calls. Metastore provides a API called > {{fire_listener_event}} which can be used to add a {{INSERT_EVENT}} to the > metastore notification log. This event can be used by other Impala instances > to invalidate or update the filemetada information when data is inserted or > overrwriten on a given table or partition. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org