[jira] [Commented] (RANGER-2149) REST search queries make Ranger incredibly slow
[ https://issues.apache.org/jira/browse/RANGER-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17001927#comment-17001927 ] sdhalex commented on RANGER-2149: - Excuse me, is it really a mysql 5.1 issue? has this bug been fixed in the newest version? Or need we still do the same workaround to avoid this bug now ? Thx. [~pradeep] [~aposledov] > REST search queries make Ranger incredibly slow > > > Key: RANGER-2149 > URL: https://issues.apache.org/jira/browse/RANGER-2149 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 0.7.0 >Reporter: Alexander Posledov >Assignee: Pradeep Agrawal >Priority: Major > Attachments: bulkpolicytest.png, totalPolicy.png > > > If amount of Ranger policies is relatively big (3000-4000), then REST search > queries (e.g. search policy by name) make Ranger very slow. Example: > curl -u admin:admin -X GET > http://host:6080/service/plugins/policies?policyName=somePolicyName > 2-3 such queries (if we have ~3k policies) can cause too long delays even for > simple requests, while these searches aren't completed. In some cases, when > we ran several search queries simultaneously, this Ranger behavior caused 9 > min delay for simple adding a new policy. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2149) REST search queries make Ranger incredibly slow
[ https://issues.apache.org/jira/browse/RANGER-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16556879#comment-16556879 ] Pradeep Agrawal commented on RANGER-2149: - [~aposledov] : Closing this issue now, please reopen if required. > REST search queries make Ranger incredibly slow > > > Key: RANGER-2149 > URL: https://issues.apache.org/jira/browse/RANGER-2149 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 0.7.0 >Reporter: Alexander Posledov >Assignee: Pradeep Agrawal >Priority: Major > Attachments: bulkpolicytest.png, totalPolicy.png > > > If amount of Ranger policies is relatively big (3000-4000), then REST search > queries (e.g. search policy by name) make Ranger very slow. Example: > curl -u admin:admin -X GET > http://host:6080/service/plugins/policies?policyName=somePolicyName > 2-3 such queries (if we have ~3k policies) can cause too long delays even for > simple requests, while these searches aren't completed. In some cases, when > we ran several search queries simultaneously, this Ranger behavior caused 9 > min delay for simple adding a new policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2149) REST search queries make Ranger incredibly slow
[ https://issues.apache.org/jira/browse/RANGER-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1654#comment-1654 ] Alexander Posledov commented on RANGER-2149: I'll check if it is a mysql 5.1 issue. Let me try the same with mysql 5.6, I'll get back with the results during the next week. > REST search queries make Ranger incredibly slow > > > Key: RANGER-2149 > URL: https://issues.apache.org/jira/browse/RANGER-2149 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 0.7.0 >Reporter: Alexander Posledov >Assignee: Pradeep Agrawal >Priority: Major > Attachments: bulkpolicytest.png, totalPolicy.png > > > If amount of Ranger policies is relatively big (3000-4000), then REST search > queries (e.g. search policy by name) make Ranger very slow. Example: > curl -u admin:admin -X GET > http://host:6080/service/plugins/policies?policyName=somePolicyName > 2-3 such queries (if we have ~3k policies) can cause too long delays even for > simple requests, while these searches aren't completed. In some cases, when > we ran several search queries simultaneously, this Ranger behavior caused 9 > min delay for simple adding a new policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2149) REST search queries make Ranger incredibly slow
[ https://issues.apache.org/jira/browse/RANGER-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16539694#comment-16539694 ] Pradeep Agrawal commented on RANGER-2149: - [~aposledov] Ranger had total 8000 policies and that particular service had 4000 policies. I used the same REST that you have provided earlier in the post. Response time through UI and curl is almost same. > REST search queries make Ranger incredibly slow > > > Key: RANGER-2149 > URL: https://issues.apache.org/jira/browse/RANGER-2149 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 0.7.0 >Reporter: Alexander Posledov >Assignee: Pradeep Agrawal >Priority: Major > Attachments: bulkpolicytest.png, totalPolicy.png > > > If amount of Ranger policies is relatively big (3000-4000), then REST search > queries (e.g. search policy by name) make Ranger very slow. Example: > curl -u admin:admin -X GET > http://host:6080/service/plugins/policies?policyName=somePolicyName > 2-3 such queries (if we have ~3k policies) can cause too long delays even for > simple requests, while these searches aren't completed. In some cases, when > we ran several search queries simultaneously, this Ranger behavior caused 9 > min delay for simple adding a new policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2149) REST search queries make Ranger incredibly slow
[ https://issues.apache.org/jira/browse/RANGER-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16538490#comment-16538490 ] Alexander Posledov commented on RANGER-2149: [~pradeep.agrawal] how many policies did your ranger contain at the moment of running query, and what type of rest request did you use? Thanks.. > REST search queries make Ranger incredibly slow > > > Key: RANGER-2149 > URL: https://issues.apache.org/jira/browse/RANGER-2149 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 0.7.0 >Reporter: Alexander Posledov >Assignee: Pradeep Agrawal >Priority: Major > Attachments: bulkpolicytest.png, totalPolicy.png > > > If amount of Ranger policies is relatively big (3000-4000), then REST search > queries (e.g. search policy by name) make Ranger very slow. Example: > curl -u admin:admin -X GET > http://host:6080/service/plugins/policies?policyName=somePolicyName > 2-3 such queries (if we have ~3k policies) can cause too long delays even for > simple requests, while these searches aren't completed. In some cases, when > we ran several search queries simultaneously, this Ranger behavior caused 9 > min delay for simple adding a new policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2149) REST search queries make Ranger incredibly slow
[
https://issues.apache.org/jira/browse/RANGER-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16538332#comment-16538332
]
Pradeep Agrawal commented on RANGER-2149:
-
MySQL-5.6
RAM - 20GB
{code:java}
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 42
Stepping: 1
CPU MHz: 2399.998
BogoMIPS: 4799.99
Virtualization: VT-x
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 4096K
NUMA node0 CPU(s): 0,1
{code}
!bulkpolicytest.png!
--
> REST search queries make Ranger incredibly slow
>
>
> Key: RANGER-2149
> URL: https://issues.apache.org/jira/browse/RANGER-2149
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 0.7.0
>Reporter: Alexander Posledov
>Assignee: Pradeep Agrawal
>Priority: Major
> Attachments: bulkpolicytest.png, totalPolicy.png
>
>
> If amount of Ranger policies is relatively big (3000-4000), then REST search
> queries (e.g. search policy by name) make Ranger very slow. Example:
> curl -u admin:admin -X GET
> http://host:6080/service/plugins/policies?policyName=somePolicyName
> 2-3 such queries (if we have ~3k policies) can cause too long delays even for
> simple requests, while these searches aren't completed. In some cases, when
> we ran several search queries simultaneously, this Ranger behavior caused 9
> min delay for simple adding a new policy.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2149) REST search queries make Ranger incredibly slow
[ https://issues.apache.org/jira/browse/RANGER-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534110#comment-16534110 ] Pradeep Agrawal commented on RANGER-2149: - [~aposledov] MySQL 5.1 is not recommended for Ranger, if possible use 5.6 I will try this exercise at my end and shall post the stats. It should not take more than a second or two. > REST search queries make Ranger incredibly slow > > > Key: RANGER-2149 > URL: https://issues.apache.org/jira/browse/RANGER-2149 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 0.7.0 >Reporter: Alexander Posledov >Assignee: Pradeep Agrawal >Priority: Major > > If amount of Ranger policies is relatively big (3000-4000), then REST search > queries (e.g. search policy by name) make Ranger very slow. Example: > curl -u admin:admin -X GET > http://host:6080/service/plugins/policies?policyName=somePolicyName > 2-3 such queries (if we have ~3k policies) can cause too long delays even for > simple requests, while these searches aren't completed. In some cases, when > we ran several search queries simultaneously, this Ranger behavior caused 9 > min delay for simple adding a new policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2149) REST search queries make Ranger incredibly slow
[ https://issues.apache.org/jira/browse/RANGER-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534107#comment-16534107 ] Pradeep Agrawal commented on RANGER-2149: - [~bosco] : We don't have an index on the name column of x_policy table. Still, It should not take this much time so will try this at my end and shall post the stats. > REST search queries make Ranger incredibly slow > > > Key: RANGER-2149 > URL: https://issues.apache.org/jira/browse/RANGER-2149 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 0.7.0 >Reporter: Alexander Posledov >Assignee: Pradeep Agrawal >Priority: Major > > If amount of Ranger policies is relatively big (3000-4000), then REST search > queries (e.g. search policy by name) make Ranger very slow. Example: > curl -u admin:admin -X GET > http://host:6080/service/plugins/policies?policyName=somePolicyName > 2-3 such queries (if we have ~3k policies) can cause too long delays even for > simple requests, while these searches aren't completed. In some cases, when > we ran several search queries simultaneously, this Ranger behavior caused 9 > min delay for simple adding a new policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2149) REST search queries make Ranger incredibly slow
[ https://issues.apache.org/jira/browse/RANGER-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533282#comment-16533282 ] Don Bosco Durai commented on RANGER-2149: - [~pradeep.agrawal] do we have index in the DB table for policy name? In the DB world, 3000 rows is not a lot of data. Even full table scan should complete in seconds and not minutes. > REST search queries make Ranger incredibly slow > > > Key: RANGER-2149 > URL: https://issues.apache.org/jira/browse/RANGER-2149 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 0.7.0 >Reporter: Alexander Posledov >Assignee: Pradeep Agrawal >Priority: Major > > If amount of Ranger policies is relatively big (3000-4000), then REST search > queries (e.g. search policy by name) make Ranger very slow. Example: > curl -u admin:admin -X GET > http://host:6080/service/plugins/policies?policyName=somePolicyName > 2-3 such queries (if we have ~3k policies) can cause too long delays even for > simple requests, while these searches aren't completed. In some cases, when > we ran several search queries simultaneously, this Ranger behavior caused 9 > min delay for simple adding a new policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2149) REST search queries make Ranger incredibly slow
[ https://issues.apache.org/jira/browse/RANGER-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532675#comment-16532675 ] Alexander Posledov commented on RANGER-2149: DB: MySQL OS: CentOS 6 [root@qerpx02-vm1 log]# mysql --version mysql Ver 14.14 Distrib 5.1.73, for redhat-linux-gnu (x86_64) using readline 5.1 [root@qerpx02-vm1 log]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 6 On-line CPU(s) list: 0-5 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 6 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 94 Model name: Intel Core Processor (Skylake) Stepping: 3 CPU MHz: 2098.782 BogoMIPS: 4197.56 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 4096K NUMA node0 CPU(s): 0-5 > REST search queries make Ranger incredibly slow > > > Key: RANGER-2149 > URL: https://issues.apache.org/jira/browse/RANGER-2149 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 0.7.0 >Reporter: Alexander Posledov >Assignee: Pradeep Agrawal >Priority: Major > > If amount of Ranger policies is relatively big (3000-4000), then REST search > queries (e.g. search policy by name) make Ranger very slow. Example: > curl -u admin:admin -X GET > http://host:6080/service/plugins/policies?policyName=somePolicyName > 2-3 such queries (if we have ~3k policies) can cause too long delays even for > simple requests, while these searches aren't completed. In some cases, when > we ran several search queries simultaneously, this Ranger behavior caused 9 > min delay for simple adding a new policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2149) REST search queries make Ranger incredibly slow
[ https://issues.apache.org/jira/browse/RANGER-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532652#comment-16532652 ] Pradeep Agrawal commented on RANGER-2149: - [~aposledov] : Can you provide configuration of machine used. Please provide DB version details also. > REST search queries make Ranger incredibly slow > > > Key: RANGER-2149 > URL: https://issues.apache.org/jira/browse/RANGER-2149 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 0.7.0 >Reporter: Alexander Posledov >Assignee: Pradeep Agrawal >Priority: Major > > If amount of Ranger policies is relatively big (3000-4000), then REST search > queries (e.g. search policy by name) make Ranger very slow. Example: > curl -u admin:admin -X GET > http://host:6080/service/plugins/policies?policyName=somePolicyName > 2-3 such queries (if we have ~3k policies) can cause too long delays even for > simple requests, while these searches aren't completed. In some cases, when > we ran several search queries simultaneously, this Ranger behavior caused 9 > min delay for simple adding a new policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
