[jira] [Commented] (RANGER-2651) Improve performance of building and querying RangerResourceTrie
[ https://issues.apache.org/jira/browse/RANGER-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982451#comment-16982451 ] Velmurugan Periasamy commented on RANGER-2651: -- Thank you [~starphin] for the contribution and [~abhayk] for merging. > Improve performance of building and querying RangerResourceTrie > > > Key: RANGER-2651 > URL: https://issues.apache.org/jira/browse/RANGER-2651 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.0.0, 1.2.0 >Reporter: star >Assignee: star >Priority: Major > Fix For: 2.1.0 > > Attachments: RANGER-2651.patch, building resource trie.png, > getMostLikelyEvaluators.png, ranger-2651.path > > > When we have 100,000 policies, it takes a long time to initialize hive plugin > (more than 1min) and evaluate access request(more than 1s). Digging into the > process, we found java stack as above images. Obviously it is the > List.IndexOf method which makes it taking too much time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2651) Improve performance of building and querying RangerResourceTrie
[ https://issues.apache.org/jira/browse/RANGER-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982211#comment-16982211 ] Junping Du commented on RANGER-2651: shall we consider to assign this jira to star given he report the issue and contribute most of the code here? [~abhayk] > Improve performance of building and querying RangerResourceTrie > > > Key: RANGER-2651 > URL: https://issues.apache.org/jira/browse/RANGER-2651 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.0.0, 1.2.0 >Reporter: star >Assignee: Abhay Kulkarni >Priority: Major > Attachments: RANGER-2651.patch, building resource trie.png, > getMostLikelyEvaluators.png, ranger-2651.path > > > When we have 100,000 policies, it takes a long time to initialize hive plugin > (more than 1min) and evaluate access request(more than 1s). Digging into the > process, we found java stack as above images. Obviously it is the > List.IndexOf method which makes it taking too much time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2651) Improve performance of building and querying RangerResourceTrie
[ https://issues.apache.org/jira/browse/RANGER-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16981929#comment-16981929 ] Abhay Kulkarni commented on RANGER-2651: A patch (RANGER-2651.patch) is attached here and is also available at the review board ( at [https://reviews.apache.org/r/71817/]). It is based on the original patch (ranger-2651.path) provided by the Reporter. > Improve performance of building and querying RangerResourceTrie > > > Key: RANGER-2651 > URL: https://issues.apache.org/jira/browse/RANGER-2651 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.0.0, 1.2.0 >Reporter: star >Assignee: Abhay Kulkarni >Priority: Major > Attachments: RANGER-2651.patch, building resource trie.png, > getMostLikelyEvaluators.png, ranger-2651.path > > > When we have 100,000 policies, it takes a long time to initialize hive plugin > (more than 1min) and evaluate access request(more than 1s). Digging into the > process, we found java stack as above images. Obviously it is the > List.IndexOf method which makes it taking too much time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2651) Improve performance of building and querying RangerResourceTrie
[
https://issues.apache.org/jira/browse/RANGER-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979918#comment-16979918
]
star commented on RANGER-2651:
--
In hive or hbase plugin, resource 'column' is always wildcard '*'. It will be
O(n^2) complexity to build RangerResourceTrie.
{code:java}
// O(n^2) in RangerResourceTrie
void addWildcardEvaluator(U evaluator) {
...
if (!wildcardEvaluators.contains(evaluator)) {
wildcardEvaluators.add(evaluator);
}
}
// O(n*m) in RangerPolicyRepository
private List getLikelyMatchPolicyEvaluators(Map resourceTrie, RangerAccessResource resource) {
...
ret = new HashSet<>(smallestSet);
for (List resourceEvaluators :
resourceEvaluatorsSet) {
if (resourceEvaluators != smallestSet) {
// remove policies from ret that are not in resourceEvaluators
ret.retainAll(resourceEvaluators);
}
}
}{code}
> Improve performance of building and querying RangerResourceTrie
>
>
> Key: RANGER-2651
> URL: https://issues.apache.org/jira/browse/RANGER-2651
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Affects Versions: 2.0.0, 1.2.0
>Reporter: star
>Priority: Major
> Attachments: building resource trie.png, getMostLikelyEvaluators.png,
> ranger-2651.path
>
>
> When we have 100,000 policies, it takes a long time to initialize hive plugin
> (more than 1min) and evaluate access request(more than 1s). Digging into the
> process, we found java stack as above images. Obviously it is the
> List.IndexOf method which makes it taking too much time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-2651) Improve performance of building and querying RangerResourceTrie
[ https://issues.apache.org/jira/browse/RANGER-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979447#comment-16979447 ] Abhay Kulkarni commented on RANGER-2651: Can you please provide details about the Java Runtime version used? Thanks! > Improve performance of building and querying RangerResourceTrie > > > Key: RANGER-2651 > URL: https://issues.apache.org/jira/browse/RANGER-2651 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.0.0, 1.2.0 >Reporter: star >Priority: Major > Attachments: building resource trie.png, getMostLikelyEvaluators.png > > > When we have 100,000 policies, it takes a long time to initialize hive plugin > (more than 1min) and evaluate access request(more than 1s). Digging into the > process, we found java stack as above images. Obviously it is the > List.IndexOf method which makes it taking too much time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2651) Improve performance of building and querying RangerResourceTrie
[ https://issues.apache.org/jira/browse/RANGER-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979426#comment-16979426 ] Velmurugan Periasamy commented on RANGER-2651: -- If you could share more details, that would be helpful. Do you have a patch to address the issue? CC [~abhay] > Improve performance of building and querying RangerResourceTrie > > > Key: RANGER-2651 > URL: https://issues.apache.org/jira/browse/RANGER-2651 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.0.0, 1.2.0 >Reporter: star >Priority: Major > Attachments: building resource trie.png, getMostLikelyEvaluators.png > > > When we have 100,000 policies, it takes a long time to initialize hive plugin > (more than 1min) and evaluate access request(more than 1s). Digging into the > process, we found java stack as above images. Obviously it is the > List.IndexOf method which makes it taking too much time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
