[
https://issues.apache.org/jira/browse/IMPALA-9369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061001#comment-17061001
]
Norbert Luksa commented on IMPALA-9369:
---------------------------------------
Looks like ASF Jira bot failed to copy the commit message, so here it is for
reference:
IMPALA-9369: Make createInsertEvents() async.
This patch makes the createInsertEvents() method async to avoid
blocking the insert code path for long periods for tables with
large number of partitions and files.
Currently the createInsertEvents() method fires the HMS insert
event one partition at a time. This makes insert statements
with thousands of new files significantly slower. This change
makes the createInsertEvent() call asynchronous by making it
run in a separate thread.
Testing:
- Ran MetastoreEventsProcessorTest#testInsertEvents.
- Ran test_events_processing::test_insert_events.
Change-Id: I97802a5c03abc067fccf9e3a9d0047324626706e
Reviewed-on: http://gerrit.cloudera.org:8080/15263
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
> Inserts on large tables could be very slow when event processing it turned on
> -----------------------------------------------------------------------------
>
> Key: IMPALA-9369
> URL: https://issues.apache.org/jira/browse/IMPALA-9369
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Vihang Karajgaonkar
> Assignee: Anurag Mantripragada
> Priority: Critical
>
> In case where large number files are being inserted into a table, the
> {{createInsertEvents}} method fires insert events to HMS for each partition
> one take a time. This could be very slow for a insert statement which is
> added hundreds or thousands of files.
> We should see if we can fire the insert events asynchronously instead of
> blocking the query from returning to the user until all the insert events are
> fired.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
