[
https://issues.apache.org/jira/browse/IMPALA-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16830765#comment-16830765
]
ASF subversion and git services commented on IMPALA-7973:
---------------------------------------------------------
Commit 3ad5b3fba202bf6809986a86f5e041da656f0a88 in impala's branch
refs/heads/master from Anurag Mantripragada
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=3ad5b3f ]
IMPALA-7973: Add support for fine grained events processing for
partition level HMS events.
This patch adds support for fine grained updates for add/drop/alter
partition events.
Currently, partition events invalidate the table. This can be
expensive for large tables. Here, we refresh affected partitions
in case of add/drop/alter partition events. HMS processes add/drop
partitions in a transaction, which means there may be multiple
partitions affected in a single add/drop event. We try to refresh all
these partitions in a loop. If any of the partition refresh fails,
we throw MetastoreNotificationNeedsInvalidateException to mandate a
manual invalidate for event processing to continue.
Testing:
Modified pre-existing tests for partition events to instead test if
partitions are added/dropped/altered when event processing is enabled.
Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Reviewed-on: http://gerrit.cloudera.org:8080/13111
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
> Add support for fine-grained updates at partition level
> -------------------------------------------------------
>
> Key: IMPALA-7973
> URL: https://issues.apache.org/jira/browse/IMPALA-7973
> Project: IMPALA
> Issue Type: Sub-task
> Reporter: Vihang Karajgaonkar
> Assignee: Anurag Mantripragada
> Priority: Major
>
> When data is inserted into a partition or a new partition is created in a
> large table, we should not be invalidating the whole table. Instead it should
> be possible to refresh/add/drop certain partitions on the table directly
> based on the event information. This would help with the performance of
> subsequent access to the table by avoiding reloading the large table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
