[jira] [Commented] (IMPALA-7971) Add support to detect insert events from Impala
[
https://issues.apache.org/jira/browse/IMPALA-7971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832016#comment-16832016
]
Tim Armstrong commented on IMPALA-7971:
---
[~anuragmantri] can you set the fix version please?
> Add support to detect insert events from Impala
> ---
>
> Key: IMPALA-7971
> URL: https://issues.apache.org/jira/browse/IMPALA-7971
> Project: IMPALA
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Anurag Mantripragada
>Priority: Major
>
> When data is inserted into existing tables and partitions, Catalog does not
> issue any metastore API calls. Metastore provides a API called
> {{fire_listener_event}} which can be used to add a {{INSERT_EVENT}} to the
> metastore notification log. This event can be used by other Impala instances
> to invalidate or update the filemetada information when data is inserted or
> overrwriten on a given table or partition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7971) Add support to detect insert events from Impala
[
https://issues.apache.org/jira/browse/IMPALA-7971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830405#comment-16830405
]
ASF subversion and git services commented on IMPALA-7971:
-
Commit 095d609260ba8957c13e61c539892ab37c5da10b in impala's branch
refs/heads/master from Todd Lipcon
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=095d609 ]
IMPALA-7971 (follow-up). Fix compilation error
15a33d1ba was committed at about the same time as 5ced9160bd6, which
renamed one of the methods of the FileDescriptor class, causing a
compilation error. This follow-up mixes the semantic conflict.
Tested tests/custom_cluster/test_event_processing.py manually.
Change-Id: I8071b421f0dcbae2d303bd9f1e1f6f64657c49c0
Reviewed-on: http://gerrit.cloudera.org:8080/13184
Reviewed-by: Todd Lipcon
Tested-by: Todd Lipcon
> Add support to detect insert events from Impala
> ---
>
> Key: IMPALA-7971
> URL: https://issues.apache.org/jira/browse/IMPALA-7971
> Project: IMPALA
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Anurag Mantripragada
>Priority: Major
>
> When data is inserted into existing tables and partitions, Catalog does not
> issue any metastore API calls. Metastore provides a API called
> {{fire_listener_event}} which can be used to add a {{INSERT_EVENT}} to the
> metastore notification log. This event can be used by other Impala instances
> to invalidate or update the filemetada information when data is inserted or
> overrwriten on a given table or partition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7971) Add support to detect insert events from Impala
[
https://issues.apache.org/jira/browse/IMPALA-7971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829851#comment-16829851
]
ASF subversion and git services commented on IMPALA-7971:
-
Commit 15a33d1baaf4723794e52825a5c4082ff7487507 in impala's branch
refs/heads/master from Anurag Mantripragada
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=15a33d1 ]
IMPALA-7971: Add support for insert events in event processor.
This patch adds support for detecting and processing insert events
triggered by impala as well as external engines (eg.Hive).
Inserts from Impala will fire an insert event notification.
Using this event, event-processor will refresh table/partition.
Both insert into and overwrite are supported for tables/partitions.
Known Issues:
1. Inserts into tables from Hive are ignored by the event processor
as these inserts create an ALTER event first followed by an
INSERT event. The alter will invalidate table making the refresh
a no-op. Insert into partitions from hive will create an INSERT
event first followed by an ALTER event. In this case, there is
an unnecessary table invalidate after a refresh.
2. Existing self-events logic cannot be used for insert events since
firing insert event does not allow us to modify table parameters in
HMS. This means we cannot get the CatalogServiceIdentifiers in insert
events. Therefore, the event-processor will also refresh the tables
for which insert operation is performed through Impala.
Testing:
1. Added new custom cluster tests to run different insert commands from
hive and verified new data is available in Impala without invalidate
metadata.
2. Added a test in MetastoreEventsProcessor for testing insert events.
Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52
Reviewed-on: http://gerrit.cloudera.org:8080/12889
Tested-by: Impala Public Jenkins
Reviewed-by: Bharath Vissapragada
> Add support to detect insert events from Impala
> ---
>
> Key: IMPALA-7971
> URL: https://issues.apache.org/jira/browse/IMPALA-7971
> Project: IMPALA
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Anurag Mantripragada
>Priority: Major
>
> When data is inserted into existing tables and partitions, Catalog does not
> issue any metastore API calls. Metastore provides a API called
> {{fire_listener_event}} which can be used to add a {{INSERT_EVENT}} to the
> metastore notification log. This event can be used by other Impala instances
> to invalidate or update the filemetada information when data is inserted or
> overrwriten on a given table or partition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7971) Add support to detect insert events from Impala
[
https://issues.apache.org/jira/browse/IMPALA-7971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799303#comment-16799303
]
Vihang Karajgaonkar commented on IMPALA-7971:
-
Changing the status to in-progress since I believe Anurag has already started
working on this.
> Add support to detect insert events from Impala
> ---
>
> Key: IMPALA-7971
> URL: https://issues.apache.org/jira/browse/IMPALA-7971
> Project: IMPALA
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Anurag Mantripragada
>Priority: Major
>
> When data is inserted into existing tables and partitions, Catalog does not
> issue any metastore API calls. Metastore provides a API called
> {{fire_listener_event}} which can be used to add a {{INSERT_EVENT}} to the
> metastore notification log. This event can be used by other Impala instances
> to invalidate or update the filemetada information when data is inserted or
> overrwriten on a given table or partition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
