[
https://issues.apache.org/jira/browse/SENTRY-1687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15962432#comment-15962432
]
Alexander Kolbasov commented on SENTRY-1687:
--------------------------------------------
As part of the change I also reworked the test for the FullUpdateInitializer to
simplify specifications of hive snapshots and to test
medium-sized snapshot.
> FullUpdateInitializer can be more efficient
> -------------------------------------------
>
> Key: SENTRY-1687
> URL: https://issues.apache.org/jira/browse/SENTRY-1687
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
> Affects Versions: sentry-ha-redesign
> Reporter: Alexander Kolbasov
> Assignee: Alexander Kolbasov
> Priority: Minor
> Attachments: SENTRY-1687.001-sentry-ha-redesign.patch
>
>
> The FullUpdateInitializer follows the {{MetastoreCacheInitializer}}. It reads
> a bunch of information from HMS and uses Thrift structures to pass around,
> but in the end it just constructs {{Map<String, Set<String>>}}. It uses
> concurrent fetches from HMS, but synchronizes a lot on common data structures
> to update them.
> I think that we can refactor all this code to make it faster and consume less
> memory. The idea is the following:
> Use background threads to collect Thrift results from HMS calls (database,
> table and partition data). Then we can use a single thread to construct the
> resulting update and return it without using intermediate Thrift methods.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
