[jira] [Updated] (RANGER-1826) Import of bulk policies is causing OOM and Apparent Deadlock

2017-10-09 Thread Pradeep Agrawal (JIRA) 

 [ 
https://issues.apache.org/jira/browse/RANGER-1826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pradeep Agrawal updated RANGER-1826:

Attachment: RANGER-1826-master-4.patch

> Import of bulk policies is causing OOM and Apparent Deadlock
> 
>
> Key: RANGER-1826
> URL: https://issues.apache.org/jira/browse/RANGER-1826
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 0.7.0, 1.0.0
>Reporter: Deepak Sharma
>Assignee: Pradeep Agrawal
> Fix For: 1.0.0, 0.7.2
>
> Attachments: RANGER-1826-master-4.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (RANGER-1826) Import of bulk policies is causing OOM and Apparent Deadlock

2017-10-09 Thread Pradeep Agrawal (JIRA) 

 [ 
https://issues.apache.org/jira/browse/RANGER-1826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pradeep Agrawal updated RANGER-1826:

Description: 
while importing bulk polices around 1000, OOM error is thrown:

{code}
2017-09-05 10:19:53,856 [http-bio-6080-exec-13] ERROR 
org.apache.ranger.rest.ServiceREST (ServiceREST.java:1603) - deletePolicy(282) 
failed
java.lang.OutOfMemoryError: Java heap space
2017-09-05 10:19:53,857 [http-bio-6080-exec-13] INFO  
org.apache.ranger.common.RESTErrorUtil (RESTErrorUtil.java:63) - Request 
failed. loginId=admin, logMessage=Java heap space
2017-09-05 10:19:54,481 [http-bio-6080-exec-13] ERROR 
org.apache.ranger.rest.ServiceREST (ServiceREST.java:2117) - Error while 
importing policy from file!!
java.lang.OutOfMemoryError: GC overhead limit exceeded
2017-09-05 10:19:54,483 [http-bio-6080-exec-13] INFO  
org.apache.ranger.common.RESTErrorUtil (RESTErrorUtil.java:63) - Request 
failed. loginId=admin, logMessage=GC overhead limit exceeded
javax.ws.rs.WebApplicationException
at 
org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:56)
at 
org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:311)
at 
org.apache.ranger.rest.ServiceREST.importPoliciesFromFile(ServiceREST.java:2125)
at 
org.apache.ranger.rest.ServiceREST$$FastClassBySpringCGLIB$$92dab672.invoke()
at 
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at 
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:700)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at 
org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:64)
{code}

then Increased the heap size:
${JAVA_OPTS} -XX:MaxPermSize=256m -Xmx2048m -Xms1024m 

then restarted the ranger service and imported the policies again:

it causes apparent deadlock
{code}
2017-09-05 13:38:22,966 [Timer-0] WARN  
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector 
(ThreadPoolAsynchronousRunner.java:608) - 
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6726e60e -- 
APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
{code}

and after sometime it starts giving OOM again:

{code}
2017-09-05 13:50:54,670 [http-bio-6080-exec-4] ERROR 
org.apache.ranger.rest.ServiceREST (ServiceREST.java:1603) - deletePolicy(256) 
failed
java.lang.OutOfMemoryError: Java heap space
2017-09-05 13:52:08,355 [zkCallback-2-thread-5-EventThread] WARN  
org.apache.solr.common.cloud.ConnectionManager (ConnectionManager.java:183) - 
zkClient received AuthFailed
2017-09-05 13:52:20,662 [http-bio-6080-exec-4] INFO  ? (?:?) - Request failed. 
loginId=admin, logMessage=Java heap space
javax.ws.rs.WebApplicationException
at 
org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:56)
at 
org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:311)
at 
org.apache.ranger.rest.ServiceREST.deletePolicy(ServiceREST.java:1605)
at 
org.apache.ranger.rest.ServiceREST.deletePoliciesProvidedInServiceMap(ServiceREST.java:224


{code}

> Import of bulk policies is causing OOM and Apparent Deadlock
> 
>
> Key: RANGER-1826
> URL: https://issues.apache.org/jira/browse/RANGER-1826
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 0.7.0, 1.0.0
>Reporter: Deepak Sharma
>Assignee: Pradeep Agrawal
> Fix For: 1.0.0, 0.7.2
>
> Attachments: RANGER-1826-master-4.patch
>
>
> while importing bulk polices around 1000, OOM error is thrown:
> {code}
> 2017-09-05 10:19:53,856 [http-bio-6080-exec-13] ERROR 
> org.apache.ranger.rest.ServiceREST (ServiceREST.java:1603) - 
> deletePolicy(282) failed
> java.lang.OutOfMemoryError: Java heap space
> 2017-09-05 10:19:53,857 [http-bio-6080-exec-13] INFO  
> org.apache.ranger.common.RESTErrorUtil (RESTErrorUtil.java:63) - Request 
> failed. loginId=admin, logMessage=Java heap space
> 2017-09-05 10:19:54,481 [http-bio-6080-exec-13] ERROR 
> org.apache.ranger.rest.ServiceREST (ServiceREST.java:2117) - Error while 
> importing policy from file!!
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> 2017-09-05 10:19:54,483 [http-bio-6080-exec-13] INFO  
> org.apache.ranger.common.RESTErrorUtil (RESTErrorUtil.java:63) - Request 
> failed. loginId=admin, logMessage=GC overhead limit exceeded
> javax.ws.rs.WebApplicationException
> at 
> org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:56)
> at 
> org.apache.ranger.common.RESTErrorUtil.createR

Re: Review Request 62686: RANGER-1816:When the error occurs, the system does not record the error message

2017-10-09 Thread Alejandro Fernandez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62686/#review187428
---


Ship it!




Ship It!

- Alejandro Fernandez


On Oct. 9, 2017, 2:49 a.m., Qiang Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62686/
> ---
> 
> (Updated Oct. 9, 2017, 2:49 a.m.)
> 
> 
> Review request for ranger, Don Bosco Durai, Colm O hEigeartaigh, pengjianhua, 
> and Ramesh Mani.
> 
> 
> Bugs: RANGER-1816
> https://issues.apache.org/jira/browse/RANGER-1816
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> these java files are as follows:
> RangerPolicyService.java
> XGroupService.java
> XUserService.java
> 
> 
> Diffs
> -
> 
>   
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyService.java
>  f55a103 
>   security-admin/src/main/java/org/apache/ranger/service/XGroupService.java 
> fd57430 
>   security-admin/src/main/java/org/apache/ranger/service/XUserService.java 
> 9be833c 
> 
> 
> Diff: https://reviews.apache.org/r/62686/diff/2/
> 
> 
> Testing
> ---
> 
> Tested!
> 
> 
> Thanks,
> 
> Qiang Zhang
> 
>

[jira] [Created] (RANGER-1827) micro benchmark for policy evaluation

2017-10-09 Thread Endre Kovacs (JIRA) 
Endre Kovacs created RANGER-1827:


 Summary: micro benchmark for policy evaluation
 Key: RANGER-1827
 URL: https://issues.apache.org/jira/browse/RANGER-1827
 Project: Ranger
  Issue Type: Test
  Components: Ranger
Affects Versions: master
Reporter: Endre Kovacs
Priority: Minor
 Fix For: 1.0.0


implement micro benchmark testing the performance of RangerPolicyEngine at 
different load of # of policies and # of concurrent users



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (RANGER-1827) micro benchmark for policy evaluation

2017-10-09 Thread Endre Kovacs (JIRA) 

 [ 
https://issues.apache.org/jira/browse/RANGER-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Endre Kovacs updated RANGER-1827:
-
Attachment: 0001-RANGER-1827-microbenchmark-for-RangerPolicyEngine.patch

microbenchmark for RangerPolicyEngine

> micro benchmark for policy evaluation
> -
>
> Key: RANGER-1827
> URL: https://issues.apache.org/jira/browse/RANGER-1827
> Project: Ranger
>  Issue Type: Test
>  Components: Ranger
>Affects Versions: master
>Reporter: Endre Kovacs
>Priority: Minor
>  Labels: performance, test
> Fix For: 1.0.0
>
> Attachments: 
> 0001-RANGER-1827-microbenchmark-for-RangerPolicyEngine.patch
>
>
> implement micro benchmark testing the performance of RangerPolicyEngine at 
> different load of # of policies and # of concurrent users



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 62850: RANGER-1827: microbenchmark for policy evaluation

2017-10-09 Thread Endre Zoltan Kovacs via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62850/
---

Review request for ranger.


Bugs: RANGER-1827
https://issues.apache.org/jira/browse/RANGER-1827


Repository: ranger


Description
---

created a microbenchmark for the policy evaluation engine


Diffs
-

  
agents-common/src/main/java/org/apache/ranger/plugin/util/PerfDataRecorder.java 
25f533476 
  ranger-tools/src/main/java/org/apache/ranger/policyengine/PerfTestClient.java 
e6095cba2 
  
ranger-tools/src/main/java/org/apache/ranger/policyengine/RangerAccessRequestDeserializer.java
 PRE-CREATION 
  
ranger-tools/src/main/java/org/apache/ranger/policyengine/RangerResourceDeserializer.java
 PRE-CREATION 
  
ranger-tools/src/main/java/org/apache/ranger/policyengine/perftest/v2/RangerPolicyFactory.java
 PRE-CREATION 
  
ranger-tools/src/test/java/org/apache/ranger/policyengine/RangerPolicyEnginePerformanceTest.java
 PRE-CREATION 
  ranger-tools/src/test/resources/log4j.properties 4ea9d854e 
  ranger-tools/src/test/resources/testdata/single-policy-template.json 
PRE-CREATION 
  ranger-tools/src/test/resources/testdata/single-request-template.json 
PRE-CREATION 


Diff: https://reviews.apache.org/r/62850/diff/1/


Testing
---

A parameterized JUnit test that tests the performance of RangerPolicyEngine 
under increasing load of number of policies and concurrent calls.
a cross product of the input parameters are generated and fed into the test 
method.

This microbenchmark includes a warm-up phase so that any of the JIT performance 
optimizations happen before the measurement of the policy engine's performance.


Thanks,

Endre Zoltan Kovacs

Re: Review Request 62850: RANGER-1827: microbenchmark for policy evaluation

2017-10-09 Thread Alejandro Fernandez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62850/#review187462
---


Fix it, then Ship it!





agents-common/src/main/java/org/apache/ranger/plugin/util/PerfDataRecorder.java
Line 88 (original), 90 (patched)


May be good to include "(microsec)"



ranger-tools/src/main/java/org/apache/ranger/policyengine/perftest/v2/RangerPolicyFactory.java
Lines 62 (patched)


Please add JavaDoc to all new classes and important methods


- Alejandro Fernandez


On Oct. 9, 2017, 9:19 p.m., Endre Zoltan Kovacs wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62850/
> ---
> 
> (Updated Oct. 9, 2017, 9:19 p.m.)
> 
> 
> Review request for ranger.
> 
> 
> Bugs: RANGER-1827
> https://issues.apache.org/jira/browse/RANGER-1827
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> created a microbenchmark for the policy evaluation engine
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/util/PerfDataRecorder.java
>  25f533476 
>   
> ranger-tools/src/main/java/org/apache/ranger/policyengine/PerfTestClient.java 
> e6095cba2 
>   
> ranger-tools/src/main/java/org/apache/ranger/policyengine/RangerAccessRequestDeserializer.java
>  PRE-CREATION 
>   
> ranger-tools/src/main/java/org/apache/ranger/policyengine/RangerResourceDeserializer.java
>  PRE-CREATION 
>   
> ranger-tools/src/main/java/org/apache/ranger/policyengine/perftest/v2/RangerPolicyFactory.java
>  PRE-CREATION 
>   
> ranger-tools/src/test/java/org/apache/ranger/policyengine/RangerPolicyEnginePerformanceTest.java
>  PRE-CREATION 
>   ranger-tools/src/test/resources/log4j.properties 4ea9d854e 
>   ranger-tools/src/test/resources/testdata/single-policy-template.json 
> PRE-CREATION 
>   ranger-tools/src/test/resources/testdata/single-request-template.json 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62850/diff/1/
> 
> 
> Testing
> ---
> 
> A parameterized JUnit test that tests the performance of RangerPolicyEngine 
> under increasing load of number of policies and concurrent calls.
> a cross product of the input parameters are generated and fed into the test 
> method.
> 
> This microbenchmark includes a warm-up phase so that any of the JIT 
> performance optimizations happen before the measurement of the policy 
> engine's performance.
> 
> 
> Thanks,
> 
> Endre Zoltan Kovacs
> 
>

Re: Review Request 62686: RANGER-1816:When the error occurs, the system does not record the error message

2017-10-09 Thread pengjianhua 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62686/#review187482
---


Ship it!




Ship It!

- pengjianhua


On Oct. 9, 2017, 2:49 a.m., Qiang Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62686/
> ---
> 
> (Updated Oct. 9, 2017, 2:49 a.m.)
> 
> 
> Review request for ranger, Don Bosco Durai, Colm O hEigeartaigh, pengjianhua, 
> and Ramesh Mani.
> 
> 
> Bugs: RANGER-1816
> https://issues.apache.org/jira/browse/RANGER-1816
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> these java files are as follows:
> RangerPolicyService.java
> XGroupService.java
> XUserService.java
> 
> 
> Diffs
> -
> 
>   
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyService.java
>  f55a103 
>   security-admin/src/main/java/org/apache/ranger/service/XGroupService.java 
> fd57430 
>   security-admin/src/main/java/org/apache/ranger/service/XUserService.java 
> 9be833c 
> 
> 
> Diff: https://reviews.apache.org/r/62686/diff/2/
> 
> 
> Testing
> ---
> 
> Tested!
> 
> 
> Thanks,
> 
> Qiang Zhang
> 
>

[jira] [Resolved] (RANGER-1816) When the error occurs, the system does not record the error message

2017-10-09 Thread peng.jianhua (JIRA) 

 [ 
https://issues.apache.org/jira/browse/RANGER-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

peng.jianhua resolved RANGER-1816.
--
   Resolution: Fixed
Fix Version/s: master
   1.0.0

> When the error occurs, the system does not record the error message
> ---
>
> Key: RANGER-1816
> URL: https://issues.apache.org/jira/browse/RANGER-1816
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Reporter: Qiang Zhang
>Assignee: Qiang Zhang
>Priority: Trivial
>  Labels: patch
> Fix For: 1.0.0, master
>
> Attachments: 
> 0001-RANGER-1816-When-the-error-occurs-the-system-does-no.patch
>
>
> these java files are as follows:
> RangerPolicyService.java
> XGroupService.java
> XUserService.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (RANGER-1813) The drop-down box name "database" is not showing full when edit hive policy

2017-10-09 Thread peng.jianhua (JIRA) 

 [ 
https://issues.apache.org/jira/browse/RANGER-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

peng.jianhua resolved RANGER-1813.
--
   Resolution: Fixed
Fix Version/s: master
   1.0.0

> The drop-down box name "database" is not showing full when edit hive policy
> ---
>
> Key: RANGER-1813
> URL: https://issues.apache.org/jira/browse/RANGER-1813
> Project: Ranger
>  Issue Type: Bug
>  Components: admin
>Reporter: Qiang Zhang
>Assignee: Qiang Zhang
>Priority: Minor
> Fix For: 1.0.0, master
>
> Attachments: 
> 0001-RANGER-1813-The-drop-down-box-name-database-is-not-s.patch, database.PNG
>
>
> The drop-down box name "database" is not showing full when edit hive policy.
> Please see picture (database.PNG)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (RANGER-1815) Fix invalid code and error logic for the BaseDao class

2017-10-09 Thread Qiang Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/RANGER-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qiang Zhang resolved RANGER-1815.
-
   Resolution: Fixed
Fix Version/s: master
   1.0.0

> Fix invalid code and error logic for the BaseDao class
> --
>
> Key: RANGER-1815
> URL: https://issues.apache.org/jira/browse/RANGER-1815
> Project: Ranger
>  Issue Type: Bug
>  Components: audit
>Affects Versions: 1.0.0, master
>Reporter: peng.jianhua
>Assignee: peng.jianhua
>Priority: Minor
>  Labels: patch
> Fix For: 1.0.0, master
>
> Attachments: 
> 0001-RANGER-1815-Fix-invalid-code-and-logic-for-BaseDao-c.patch
>
>
> Fix invalid code and error logic for the BaseDao class



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1796) Updated masking policy for hive to support for deny/allowException/denyExceptions

2017-10-09 Thread peng.jianhua (JIRA) 

[ 
https://issues.apache.org/jira/browse/RANGER-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16198030#comment-16198030
 ] 

peng.jianhua commented on RANGER-1796:
--

Hi [~madhan.neethiraj], do you have further advice?

> Updated masking policy for hive  to support for 
> deny/allowException/denyExceptions
> --
>
> Key: RANGER-1796
> URL: https://issues.apache.org/jira/browse/RANGER-1796
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>Affects Versions: 1.0.0, master
>Reporter: peng.jianhua
>Assignee: peng.jianhua
>  Labels: newbie, patch
> Attachments: 
> 0001-RANGER-1796-Updated-masking-policy-for-hive-to-suppo.patch, 
> masking-03.png, masking-04.png, masking2.png, usecase-01.png, usecase-02.png
>
>
> Masking policy for hive  should support for 
> deny/allowException/denyExceptions to meet further business needs. Such as 
> masking policy for hive should support as following scene and so on:
> USER1, USER2 and USER3 belong to the user group GROUPA. Select GROUPA group 
> when created masking policy. The USER1 does not use masking and USER2, USER3 
> need masking.
> We rigorously tested this issue. The test result shows that the feature is ok.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1796) Updated masking policy for hive to support for deny/allowException/denyExceptions

2017-10-09 Thread Madhan Neethiraj (JIRA) 

[ 
https://issues.apache.org/jira/browse/RANGER-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16198061#comment-16198061
 ] 

Madhan Neethiraj commented on RANGER-1796:
--

bq. your case was not supported by ranger now. According to your configuration 
the ranger will execute error. 
[~peng.jianhua] - did you try this policy for USER1, other users in 
GROUPA/GROUPB and rest of the users? What kind of masking was applied in each 
case? If you got error, it will help if you can add error details.

> Updated masking policy for hive  to support for 
> deny/allowException/denyExceptions
> --
>
> Key: RANGER-1796
> URL: https://issues.apache.org/jira/browse/RANGER-1796
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>Affects Versions: 1.0.0, master
>Reporter: peng.jianhua
>Assignee: peng.jianhua
>  Labels: newbie, patch
> Attachments: 
> 0001-RANGER-1796-Updated-masking-policy-for-hive-to-suppo.patch, 
> masking-03.png, masking-04.png, masking2.png, usecase-01.png, usecase-02.png
>
>
> Masking policy for hive  should support for 
> deny/allowException/denyExceptions to meet further business needs. Such as 
> masking policy for hive should support as following scene and so on:
> USER1, USER2 and USER3 belong to the user group GROUPA. Select GROUPA group 
> when created masking policy. The USER1 does not use masking and USER2, USER3 
> need masking.
> We rigorously tested this issue. The test result shows that the feature is ok.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1796) Updated masking policy for hive to support for deny/allowException/denyExceptions

2017-10-09 Thread peng.jianhua (JIRA) 

[ 
https://issues.apache.org/jira/browse/RANGER-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16198116#comment-16198116
 ] 

peng.jianhua commented on RANGER-1796:
--

[~mad...@apache.org]  1.USER1 and USER2 in GROUPA,   USER3 in GROUPB , hong is 
a general user , you can see detail in user-05.png.
2.Configuration: the policy configuration detail in masking-04.png. 
3.Result: in each case   Partital mask:show  first 4  was applied.   you can 
see detail in masking-06.png ,masking-07.png,masking-08.png,masking-09.png.

> Updated masking policy for hive  to support for 
> deny/allowException/denyExceptions
> --
>
> Key: RANGER-1796
> URL: https://issues.apache.org/jira/browse/RANGER-1796
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>Affects Versions: 1.0.0, master
>Reporter: peng.jianhua
>Assignee: peng.jianhua
>  Labels: newbie, patch
> Attachments: 
> 0001-RANGER-1796-Updated-masking-policy-for-hive-to-suppo.patch, 
> masking-03.png, masking-04.png, masking2.png, usecase-01.png, usecase-02.png
>
>
> Masking policy for hive  should support for 
> deny/allowException/denyExceptions to meet further business needs. Such as 
> masking policy for hive should support as following scene and so on:
> USER1, USER2 and USER3 belong to the user group GROUPA. Select GROUPA group 
> when created masking policy. The USER1 does not use masking and USER2, USER3 
> need masking.
> We rigorously tested this issue. The test result shows that the feature is ok.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62841: RANGER-1826: Import of bulk policies is causing OutOfMemoryError and Apparent Deadlock

2017-10-09 Thread Pradeep Agrawal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62841/
---

(Updated Oct. 10, 2017, 3:47 a.m.)


Review request for ranger, Ankita Sinha, Don Bosco Durai, bhavik patel, Gautam 
Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Selvamohan 
Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.


Changes
---

Updated reviewer list


Bugs: RANGER-1826
https://issues.apache.org/jira/browse/RANGER-1826


Repository: ranger


Description
---

**Problem Statement:**
While importing bulk policies around 1000, OOM error is thrown, increasing the 
heap size is also not working as it seems EclipseLink holds all the objects and 
states of SQL operations for each delete policy request.

**Proposed Solution:**
To delete policies and its references rather relying on EclipseLink/JPA we can 
send delete request using native query so that EclipseLink object won't be 
created, and EclipseLink shall not hold too many objects in memory, which shall 
remove the cause of java.lang.OutOfMemoryError.

**Note:** Use of native query statement is discouraged so please suggest a 
better solution or provide a working solution.


Diffs
-

  security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 
bb43c53 
  
security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java
 7d1f28c 
  security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java c2832ea 
  security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java d8f217d 
  security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 
976fd0c 


Diff: https://reviews.apache.org/r/62841/diff/1/


Testing
---

**Steps Performed (with patch):**
1. After mvn Build; untar the Ranger module and updated install.properties for 
MySQL DB flavor. 
2. Called setup.sh to execute Ranger setup script. 
3. Started Ranger admin and created a hdfs service 'hadoopdev'.
4. Tried to import Policies.

**Attempt-1:** When there are no policies except the default one in 'hadoopdev'.
Tried Import operation with 1000 policies.

**Expected Behavior:**
Policy import process should finish successfully.

**Actual Behavior:**
Policy import process finished successfully.


**Attempt-2:** There are already 1000 policies in the hdfs service 
'hadoopdev'(policies created during the previous attempt)
Tried Import operation again with 1000 policies.

**Expected Behavior:**
Policy import process should finish successfully.

**Actual Behavior:**
Policy import process finished successfully.

**Note:** 
Patch has been tested on all supported DB Flavors.


Thanks,

Pradeep Agrawal

[jira] [Updated] (RANGER-1796) Updated masking policy for hive to support for deny/allowException/denyExceptions

2017-10-09 Thread peng.jianhua (JIRA) 

 [ 
https://issues.apache.org/jira/browse/RANGER-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

peng.jianhua updated RANGER-1796:
-
Attachment: user5.png
masking-06.png
masking-07.png
masking-08.png
masking-09.png

> Updated masking policy for hive  to support for 
> deny/allowException/denyExceptions
> --
>
> Key: RANGER-1796
> URL: https://issues.apache.org/jira/browse/RANGER-1796
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>Affects Versions: 1.0.0, master
>Reporter: peng.jianhua
>Assignee: peng.jianhua
>  Labels: newbie, patch
> Attachments: 
> 0001-RANGER-1796-Updated-masking-policy-for-hive-to-suppo.patch, 
> masking-03.png, masking-04.png, masking-06.png, masking-07.png, 
> masking-08.png, masking-09.png, masking2.png, usecase-01.png, usecase-02.png, 
> user5.png
>
>
> Masking policy for hive  should support for 
> deny/allowException/denyExceptions to meet further business needs. Such as 
> masking policy for hive should support as following scene and so on:
> USER1, USER2 and USER3 belong to the user group GROUPA. Select GROUPA group 
> when created masking policy. The USER1 does not use masking and USER2, USER3 
> need masking.
> We rigorously tested this issue. The test result shows that the feature is ok.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1796) Updated masking policy for hive to support for deny/allowException/denyExceptions

2017-10-09 Thread Madhan Neethiraj (JIRA) 

[ 
https://issues.apache.org/jira/browse/RANGER-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16198128#comment-16198128
 ] 

Madhan Neethiraj commented on RANGER-1796:
--

[~peng.jianhua] - can you also attach a screen shot of audit logs that show the 
maskType applied for each user?

> Updated masking policy for hive  to support for 
> deny/allowException/denyExceptions
> --
>
> Key: RANGER-1796
> URL: https://issues.apache.org/jira/browse/RANGER-1796
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>Affects Versions: 1.0.0, master
>Reporter: peng.jianhua
>Assignee: peng.jianhua
>  Labels: newbie, patch
> Attachments: 
> 0001-RANGER-1796-Updated-masking-policy-for-hive-to-suppo.patch, 
> masking-03.png, masking-04.png, masking-06.png, masking-07.png, 
> masking-08.png, masking-09.png, masking2.png, usecase-01.png, usecase-02.png, 
> user5.png
>
>
> Masking policy for hive  should support for 
> deny/allowException/denyExceptions to meet further business needs. Such as 
> masking policy for hive should support as following scene and so on:
> USER1, USER2 and USER3 belong to the user group GROUPA. Select GROUPA group 
> when created masking policy. The USER1 does not use masking and USER2, USER3 
> need masking.
> We rigorously tested this issue. The test result shows that the feature is ok.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62841: RANGER-1826: Import of bulk policies is causing OutOfMemoryError and Apparent Deadlock

2017-10-09 Thread Velmurugan Periasamy 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62841/#review187492
---


Ship it!




Ship It!

- Velmurugan Periasamy


On Oct. 10, 2017, 3:47 a.m., Pradeep Agrawal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62841/
> ---
> 
> (Updated Oct. 10, 2017, 3:47 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Don Bosco Durai, bhavik patel, 
> Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, 
> Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-1826
> https://issues.apache.org/jira/browse/RANGER-1826
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> **Problem Statement:**
> While importing bulk policies around 1000, OOM error is thrown, increasing 
> the heap size is also not working as it seems EclipseLink holds all the 
> objects and states of SQL operations for each delete policy request.
> 
> **Proposed Solution:**
> To delete policies and its references rather relying on EclipseLink/JPA we 
> can send delete request using native query so that EclipseLink object won't 
> be created, and EclipseLink shall not hold too many objects in memory, which 
> shall remove the cause of java.lang.OutOfMemoryError.
> 
> **Note:** Use of native query statement is discouraged so please suggest a 
> better solution or provide a working solution.
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 
> bb43c53 
>   
> security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java
>  7d1f28c 
>   security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 
> c2832ea 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 
> d8f217d 
>   security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 
> 976fd0c 
> 
> 
> Diff: https://reviews.apache.org/r/62841/diff/1/
> 
> 
> Testing
> ---
> 
> **Steps Performed (with patch):**
> 1. After mvn Build; untar the Ranger module and updated install.properties 
> for MySQL DB flavor. 
> 2. Called setup.sh to execute Ranger setup script. 
> 3. Started Ranger admin and created a hdfs service 'hadoopdev'.
> 4. Tried to import Policies.
> 
> **Attempt-1:** When there are no policies except the default one in 
> 'hadoopdev'.
> Tried Import operation with 1000 policies.
> 
> **Expected Behavior:**
> Policy import process should finish successfully.
> 
> **Actual Behavior:**
> Policy import process finished successfully.
> 
> 
> **Attempt-2:** There are already 1000 policies in the hdfs service 
> 'hadoopdev'(policies created during the previous attempt)
> Tried Import operation again with 1000 policies.
> 
> **Expected Behavior:**
> Policy import process should finish successfully.
> 
> **Actual Behavior:**
> Policy import process finished successfully.
> 
> **Note:** 
> Patch has been tested on all supported DB Flavors.
> 
> 
> Thanks,
> 
> Pradeep Agrawal
> 
>

Re: Review Request 62495: RANGER-1797:Tomcat Security Vulnerability Alert. The version of the tomcat for ranger should upgrade to 7.0.81.

2017-10-09 Thread bhavik patel 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62495/#review187494
---



@pengjianhua : Any updates on this?

- bhavik patel


On Sept. 22, 2017, 8:35 a.m., pengjianhua wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62495/
> ---
> 
> (Updated Sept. 22, 2017, 8:35 a.m.)
> 
> 
> Review request for ranger, Alok Lal, Ankita Sinha, Don Bosco Durai, Colm O 
> hEigeartaigh, Gautam Borad, Madhan Neethiraj, Ramesh Mani, Selvamohan 
> Neethiraj, Velmurugan Periasamy, and Qiang Zhang.
> 
> 
> Bugs: RANGER-1797
> https://issues.apache.org/jira/browse/RANGER-1797
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> [Security Vulnerability Alert] Tomcat Information leakage and remote code 
> execution vulnerabilities.
> 
> CVE ID:
> CVE-2017-12615\CVE-2017-12616
> 
> Description
> CVE-2017-12615:When running Apache Tomcat 7.0.0 to 7.0.79 on Windows with 
> HTTP PUTs enabled, it was possible to upload a JSP file to the server via a 
> specially crafted request. This JSP could then be requested and any code it 
> contained would be executed by the server.
> CVE-2017-12616:When using a VirtualDirContext with Apache Tomcat 7.0.0 to 
> 7.0.80, it was possible to use a specially crafted request, bypass security 
> constraints, or get the source code of JSPs for resources served by the 
> VirtualDirContext, thereby cased code disclosure.
> 
> Scope
> CVE-2017-12615:Apache Tomcat 7.0.0 - 7.0.79
> CVE-2017-12616:Apache Tomcat 7.0.0 - 7.0.80
> 
> Solution
> The official release of the Apache Tomcat 7.0.81 version has fixed the two 
> vulnerabilities and recommends upgrading to the latest version.
> 
> Reference
> https://tomcat.apache.org/security-7.html
> http://tomcat.apache.org/security-7.html#Fixed_in_Apache_Tomcat_7.0.81
> 
> 
> Diffs
> -
> 
>   pom.xml 3958014c 
> 
> 
> Diff: https://reviews.apache.org/r/62495/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengjianhua
> 
>

[jira] [Assigned] (RANGER-1176) Ranger admin does not allow to create / update a policy with only delegate admin permission

2017-10-09 Thread Nitin Galave (JIRA) 

 [ 
https://issues.apache.org/jira/browse/RANGER-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nitin Galave reassigned RANGER-1176:


Assignee: Nitin Galave

> Ranger admin does not allow to create / update a policy with only delegate 
> admin permission
> ---
>
> Key: RANGER-1176
> URL: https://issues.apache.org/jira/browse/RANGER-1176
> Project: Ranger
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 0.5.0
>Reporter: Vipin Rathor
>Assignee: Nitin Galave
>Priority: Minor
>
> As of RANGER-436, a policy with empty access list is valid if delegated admin 
> is true. But the Ranger Admin UI doesn't allow user to create / update a 
> policy with only the 'delegate admin' permission for a user / group.
> Expected Result:
> Create / update policy with only 'delegate admin' permission should be 
> allowed via Ranger Admin UI 
> Actual Result:
> Upon saving, the create page throws this error for user:
> "Please add permission(s) for the selected User, else User will not be added."
> And similar error for group:
> "Please add permission(s) for the selected Group, else Group will not be 
> added."
> Workaround:
> As of now, the workaround is to create / update the policy via REST API. But 
> the only caveat is user will not be able to edit the policy again via Admin 
> UI.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (RANGER-1176) Ranger admin does not allow to create / update a policy with only delegate admin permission

2017-10-09 Thread Nitin Galave (JIRA) 

 [ 
https://issues.apache.org/jira/browse/RANGER-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nitin Galave updated RANGER-1176:
-
Fix Version/s: 0.7.2
   1.0.0

> Ranger admin does not allow to create / update a policy with only delegate 
> admin permission
> ---
>
> Key: RANGER-1176
> URL: https://issues.apache.org/jira/browse/RANGER-1176
> Project: Ranger
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 0.5.0
>Reporter: Vipin Rathor
>Assignee: Nitin Galave
>Priority: Minor
> Fix For: 1.0.0, 0.7.2
>
>
> As of RANGER-436, a policy with empty access list is valid if delegated admin 
> is true. But the Ranger Admin UI doesn't allow user to create / update a 
> policy with only the 'delegate admin' permission for a user / group.
> Expected Result:
> Create / update policy with only 'delegate admin' permission should be 
> allowed via Ranger Admin UI 
> Actual Result:
> Upon saving, the create page throws this error for user:
> "Please add permission(s) for the selected User, else User will not be added."
> And similar error for group:
> "Please add permission(s) for the selected Group, else Group will not be 
> added."
> Workaround:
> As of now, the workaround is to create / update the policy via REST API. But 
> the only caveat is user will not be able to edit the policy again via Admin 
> UI.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62495: RANGER-1797:Tomcat Security Vulnerability Alert. The version of the tomcat for ranger should upgrade to 7.0.81.

2017-10-09 Thread pengjianhua 


> On Oct. 4, 2017, 2:16 p.m., Velmurugan Periasamy wrote:
> > pom.xml
> > Line 212 (original), 212 (patched)
> > 
> >
> > See 
> > https://tomcat.apache.org/security-7.html#Fixed_in_Apache_Tomcat_7.0.82 for 
> > additional CVE fixed, hence it is better to upgrade to 7.0.82 with this 
> > effort. 
> > 
> > +1 to Bhavik's suggestions to cover SSL/Kerberos/Knox tests for Ranger 
> > Admin and SSL/Kerberos for Ranger KMS

ok. Thanks.


- pengjianhua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62495/#review187093
---


On Sept. 22, 2017, 8:35 a.m., pengjianhua wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62495/
> ---
> 
> (Updated Sept. 22, 2017, 8:35 a.m.)
> 
> 
> Review request for ranger, Alok Lal, Ankita Sinha, Don Bosco Durai, Colm O 
> hEigeartaigh, Gautam Borad, Madhan Neethiraj, Ramesh Mani, Selvamohan 
> Neethiraj, Velmurugan Periasamy, and Qiang Zhang.
> 
> 
> Bugs: RANGER-1797
> https://issues.apache.org/jira/browse/RANGER-1797
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> [Security Vulnerability Alert] Tomcat Information leakage and remote code 
> execution vulnerabilities.
> 
> CVE ID:
> CVE-2017-12615\CVE-2017-12616
> 
> Description
> CVE-2017-12615:When running Apache Tomcat 7.0.0 to 7.0.79 on Windows with 
> HTTP PUTs enabled, it was possible to upload a JSP file to the server via a 
> specially crafted request. This JSP could then be requested and any code it 
> contained would be executed by the server.
> CVE-2017-12616:When using a VirtualDirContext with Apache Tomcat 7.0.0 to 
> 7.0.80, it was possible to use a specially crafted request, bypass security 
> constraints, or get the source code of JSPs for resources served by the 
> VirtualDirContext, thereby cased code disclosure.
> 
> Scope
> CVE-2017-12615:Apache Tomcat 7.0.0 - 7.0.79
> CVE-2017-12616:Apache Tomcat 7.0.0 - 7.0.80
> 
> Solution
> The official release of the Apache Tomcat 7.0.81 version has fixed the two 
> vulnerabilities and recommends upgrading to the latest version.
> 
> Reference
> https://tomcat.apache.org/security-7.html
> http://tomcat.apache.org/security-7.html#Fixed_in_Apache_Tomcat_7.0.81
> 
> 
> Diffs
> -
> 
>   pom.xml 3958014c 
> 
> 
> Diff: https://reviews.apache.org/r/62495/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengjianhua
> 
>

Re: Review Request 62495: RANGER-1797:Tomcat Security Vulnerability Alert. The version of the tomcat for ranger should upgrade to 7.0.81.

2017-10-09 Thread pengjianhua 


> On Oct. 10, 2017, 5:19 a.m., bhavik patel wrote:
> > @pengjianhua : Any updates on this?

I am testing SSL/Kerberos for Ranger KMS.


- pengjianhua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62495/#review187494
---


On Sept. 22, 2017, 8:35 a.m., pengjianhua wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62495/
> ---
> 
> (Updated Sept. 22, 2017, 8:35 a.m.)
> 
> 
> Review request for ranger, Alok Lal, Ankita Sinha, Don Bosco Durai, Colm O 
> hEigeartaigh, Gautam Borad, Madhan Neethiraj, Ramesh Mani, Selvamohan 
> Neethiraj, Velmurugan Periasamy, and Qiang Zhang.
> 
> 
> Bugs: RANGER-1797
> https://issues.apache.org/jira/browse/RANGER-1797
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> [Security Vulnerability Alert] Tomcat Information leakage and remote code 
> execution vulnerabilities.
> 
> CVE ID:
> CVE-2017-12615\CVE-2017-12616
> 
> Description
> CVE-2017-12615:When running Apache Tomcat 7.0.0 to 7.0.79 on Windows with 
> HTTP PUTs enabled, it was possible to upload a JSP file to the server via a 
> specially crafted request. This JSP could then be requested and any code it 
> contained would be executed by the server.
> CVE-2017-12616:When using a VirtualDirContext with Apache Tomcat 7.0.0 to 
> 7.0.80, it was possible to use a specially crafted request, bypass security 
> constraints, or get the source code of JSPs for resources served by the 
> VirtualDirContext, thereby cased code disclosure.
> 
> Scope
> CVE-2017-12615:Apache Tomcat 7.0.0 - 7.0.79
> CVE-2017-12616:Apache Tomcat 7.0.0 - 7.0.80
> 
> Solution
> The official release of the Apache Tomcat 7.0.81 version has fixed the two 
> vulnerabilities and recommends upgrading to the latest version.
> 
> Reference
> https://tomcat.apache.org/security-7.html
> http://tomcat.apache.org/security-7.html#Fixed_in_Apache_Tomcat_7.0.81
> 
> 
> Diffs
> -
> 
>   pom.xml 3958014c 
> 
> 
> Diff: https://reviews.apache.org/r/62495/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengjianhua
> 
>