Re: Review Request 72350: RANGER-2791: Ignore .iml files from RAT check for schema-registry plugin

[Velmurugan Periasamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72350/#review220282
---


Ship it!




Ship It!

- Velmurugan Periasamy


On April 10, 2020, 7:06 p.m., Abhay Kulkarni wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72350/
> ---
> 
> (Updated April 10, 2020, 7:06 p.m.)
> 
> 
> Review request for ranger and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2791
> https://issues.apache.org/jira/browse/RANGER-2791
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> schema-registry plugin build with "verify" option fails if in presence of 
> .iml files in the source tree. These files must be excluded from rat-check.
> 
> 
> Diffs
> -
> 
>   plugin-schema-registry/pom.xml 6bd2d9766 
> 
> 
> Diff: https://reviews.apache.org/r/72350/diff/1/
> 
> 
> Testing
> ---
> 
> Compiled clean with verify option successfully
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>

Review Request 72350: RANGER-2791: Ignore .iml files from RAT check for schema-registry plugin

[Abhay Kulkarni]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72350/
---

Review request for ranger and Velmurugan Periasamy.


Bugs: RANGER-2791
https://issues.apache.org/jira/browse/RANGER-2791


Repository: ranger


Description
---

schema-registry plugin build with "verify" option fails if in presence of .iml 
files in the source tree. These files must be excluded from rat-check.


Diffs
-

  plugin-schema-registry/pom.xml 6bd2d9766 


Diff: https://reviews.apache.org/r/72350/diff/1/


Testing
---

Compiled clean with verify option successfully


Thanks,

Abhay Kulkarni

[jira] [Created] (RANGER-2791) Ignore .iml files from RAT check for schema-registry plugin

[Abhay Kulkarni (Jira)]

Abhay Kulkarni created RANGER-2791:
--

 Summary: Ignore .iml files from RAT check for schema-registry 
plugin
 Key: RANGER-2791
 URL: https://issues.apache.org/jira/browse/RANGER-2791
 Project: Ranger
  Issue Type: Improvement
  Components: Ranger
Affects Versions: master
Reporter: Abhay Kulkarni
Assignee: Abhay Kulkarni


schema-registry plugin build with "verify" option fails if in presence of .iml 
files in the source tree. These files must be excluded from rat-check.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72272: Upgrade and improve Presto plugin

[Bolke de Bruin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72272/
---

(Updated April 10, 2020, 6 p.m.)


Review request for ranger, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, and 
Ramesh Mani.


Changes
---

make sure not to return null for filter query owners


Bugs: https://issues.apache.org/jira/browse/RANGER-2754

https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/RANGER-2754


Repository: ranger


Description
---

Upgrade and improve Presto plugin
- Presto SQL 331 has changed its security API and has Row level / column 
masking functionality
- Upgraded Hadoop dependency to 3.1.3 (from 3.1.1) due to improved security 
handling
- New features like session properties and system properties


Diffs (updated)
-

  agents-common/src/main/resources/service-defs/ranger-servicedef-presto.json 
56a8f5ac0 
  distro/src/main/assembly/plugin-presto.xml d2075bfe7 
  plugin-presto/pom.xml b63f7dede 
  
plugin-presto/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControl.java
 3ab63f590 
  
plugin-presto/src/test/java/org/apache/ranger/authorization/presto/authorizer/RangerAdminClientImpl.java
 PRE-CREATION 
  
plugin-presto/src/test/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControlTest.java
 PRE-CREATION 
  plugin-presto/src/test/resources/log4j.properties PRE-CREATION 
  plugin-presto/src/test/resources/presto-policies.json PRE-CREATION 
  plugin-presto/src/test/resources/ranger-presto-security.xml PRE-CREATION 
  pom.xml b62d9b663 
  ranger-presto-plugin-shim/pom.xml d8ff88d0f 
  
ranger-presto-plugin-shim/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerConfig.java
 67b0d2434 
  
ranger-presto-plugin-shim/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControl.java
 e89f646e1 


Diff: https://reviews.apache.org/r/72272/diff/6/

Changes: https://reviews.apache.org/r/72272/diff/5-6/


Testing
---

- New Unit tests added
- Tested locally in production


Thanks,

Bolke de Bruin

[jira] [Updated] (RANGER-2754) Update presto dependency and implement row/column level security

[Bolke de Bruin (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated RANGER-2754:
---
Attachment: 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch

> Update presto dependency and implement row/column level security
> 
>
> Key: RANGER-2754
> URL: https://issues.apache.org/jira/browse/RANGER-2754
> Project: Ranger
>  Issue Type: Improvement
>  Components: plugins
>Affects Versions: master
>Reporter: Bolke de Bruin
>Assignee: Bolke de Bruin
>Priority: Major
> Attachments: 
> 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch, 
> 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch, 
> 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch, 
> 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch, 
> 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch, 
> RANGER-2754-v2.patch, RANGER-2754.patch
>
>
> 1. PrestoSql has changed its Security API hence the Ranger plugin has stopped 
> working for versions > ~321. 
> 2. Presto master now has row/column level security support



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (RANGER-2754) Update presto dependency and implement row/column level security

[Bolke de Bruin (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated RANGER-2754:
---
Attachment: 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch

> Update presto dependency and implement row/column level security
> 
>
> Key: RANGER-2754
> URL: https://issues.apache.org/jira/browse/RANGER-2754
> Project: Ranger
>  Issue Type: Improvement
>  Components: plugins
>Affects Versions: master
>Reporter: Bolke de Bruin
>Assignee: Bolke de Bruin
>Priority: Major
> Attachments: 
> 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch, 
> 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch, 
> 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch, 
> 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch, 
> RANGER-2754-v2.patch, RANGER-2754.patch
>
>
> 1. PrestoSql has changed its Security API hence the Ranger plugin has stopped 
> working for versions > ~321. 
> 2. Presto master now has row/column level security support



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72272: Upgrade and improve Presto plugin

[Bolke de Bruin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72272/
---

(Updated April 10, 2020, 5:16 p.m.)


Review request for ranger, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, and 
Ramesh Mani.


Changes
---

Includes tests for filter(Catalogs/Schema/Tables)


Bugs: https://issues.apache.org/jira/browse/RANGER-2754

https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/RANGER-2754


Repository: ranger


Description
---

Upgrade and improve Presto plugin
- Presto SQL 331 has changed its security API and has Row level / column 
masking functionality
- Upgraded Hadoop dependency to 3.1.3 (from 3.1.1) due to improved security 
handling
- New features like session properties and system properties


Diffs (updated)
-

  agents-common/src/main/resources/service-defs/ranger-servicedef-presto.json 
56a8f5ac0 
  distro/src/main/assembly/plugin-presto.xml d2075bfe7 
  plugin-presto/pom.xml b63f7dede 
  
plugin-presto/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControl.java
 3ab63f590 
  
plugin-presto/src/test/java/org/apache/ranger/authorization/presto/authorizer/RangerAdminClientImpl.java
 PRE-CREATION 
  
plugin-presto/src/test/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControlTest.java
 PRE-CREATION 
  plugin-presto/src/test/resources/log4j.properties PRE-CREATION 
  plugin-presto/src/test/resources/presto-policies.json PRE-CREATION 
  plugin-presto/src/test/resources/ranger-presto-security.xml PRE-CREATION 
  pom.xml b62d9b663 
  ranger-presto-plugin-shim/pom.xml d8ff88d0f 
  
ranger-presto-plugin-shim/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerConfig.java
 67b0d2434 
  
ranger-presto-plugin-shim/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControl.java
 e89f646e1 


Diff: https://reviews.apache.org/r/72272/diff/5/

Changes: https://reviews.apache.org/r/72272/diff/4-5/


Testing
---

- New Unit tests added
- Tested locally in production


Thanks,

Bolke de Bruin

[jira] [Commented] (RANGER-2789) GET API service/xusers/users turns very slow when there are more than 1000 users

[RickyMa (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080611#comment-17080611
 ] 

RickyMa commented on RANGER-2789:
-

[~pradeep] Thanks for your reply. :)

> GET API service/xusers/users turns very slow when there are more than 1000 
> users
> 
>
> Key: RANGER-2789
> URL: https://issues.apache.org/jira/browse/RANGER-2789
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.0.0
> Environment: hdp 2.2
>Reporter: RickyMa
>Priority: Minor
>  Labels: optimization, performance, ranger, rangeradmin, slow
> Attachments: RANGER-2789.patch, image-2020-04-10-19-37-31-570.png
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We have more than 1000 users in production environment.
> When calling API 'service/xusers/users?pageSize=1000=0' using HTTP 
> GET method, Ranger Admin takes 10+ minutes to response.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (RANGER-2786) Ranger usersync group modifyTimestamp parsing should be in 24 hours format

[Lars Francke (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080496#comment-17080496
 ] 

Lars Francke commented on RANGER-2786:
--

I believe you attached the wrong file.

 

It'd also be great if you could provide a short description of the issue.

>  Ranger usersync group modifyTimestamp parsing should be in 24 hours format
> ---
>
> Key: RANGER-2786
> URL: https://issues.apache.org/jira/browse/RANGER-2786
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Reporter: Pradeep Agrawal
>Assignee: Pradeep Agrawal
>Priority: Major
> Fix For: 2.1.0, 2.1
>
> Attachments: 
> .0001-RANGER-2786-Ranger-usersync-group-modifyTimestamp-pa.patch.swp
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (RANGER-2450) Fix Failing HBase test cases

[Pradeep Agrawal (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080486#comment-17080486
 ] 

Pradeep Agrawal commented on RANGER-2450:
-

[~rajeshbabu] Not sure if that can be upgraded in this release.

> Fix Failing HBase test cases
> 
>
> Key: RANGER-2450
> URL: https://issues.apache.org/jira/browse/RANGER-2450
> Project: Ranger
>  Issue Type: Sub-task
>  Components: Ranger
>Affects Versions: 2.0.0
>Reporter: Pradeep Agrawal
>Assignee: Pradeep Agrawal
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72325: RANGER-2772 : Adding the functionality of merging the policy

[Pradeep Agrawal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72325/#review220280
---




security-admin/src/main/java/org/apache/ranger/rest/ServiceRESTUtil.java
Line 880 (original), 999 (patched)


if this is not required, the please remove it.


- Pradeep Agrawal


On April 9, 2020, 1:25 p.m., Dineshkumar Yadav wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72325/
> ---
> 
> (Updated April 9, 2020, 1:25 p.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Gautam Borad, Kishor Gollapalliwar, 
> Abhay Kulkarni, Mehul Parikh, Pradeep Agrawal, Vishal Suvagia, and Velmurugan 
> Periasamy.
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Handling the scenario when non-existing policy send with mergeIfExist=true
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 
> 38b49820b 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceRESTUtil.java 
> d85028c95 
> 
> 
> Diff: https://reviews.apache.org/r/72325/diff/3/
> 
> 
> Testing
> ---
> 
> Test case:
> 
> Creating non-existing policy with below param
> a) mergeIfExists=true
> b) serviceName
> c) policyName
> 
> Expected: New policy should be created.
> 
> Creating policy with below param and role which is already exist
> a) mergeIfExists=true
> b) serviceName
> c) policyName
> 
> Expected: role should get merged.
> 
> Creating policy with below param and policy condition in kafka service which 
> is already exist
> a) mergeIfExists=true
> b) serviceName
> c) policyName
> 
> Expected: condition should get merged.
> 
> 
> Thanks,
> 
> Dineshkumar Yadav
> 
>

[jira] [Updated] (RANGER-2790) Import start and import end are not in sequence

[Mahesh Hanumant Bandal (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahesh Hanumant Bandal updated RANGER-2790:
---
Attachment: RANGER-2790-V1.patch

> Import start and import end are not in sequence
> ---
>
> Key: RANGER-2790
> URL: https://issues.apache.org/jira/browse/RANGER-2790
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Reporter: Harshal Chavan
>Priority: Major
> Attachments: RANGER-2790-V1.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (RANGER-2789) GET API service/xusers/users turns very slow when there are more than 1000 users

[Pradeep Agrawal (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080468#comment-17080468
 ] 

Pradeep Agrawal commented on RANGER-2789:
-

[~RickyMa] Good finding and thanks for the fix. Currently I don't have 
environment to test this patch. I shall update you on  this soon.

> GET API service/xusers/users turns very slow when there are more than 1000 
> users
> 
>
> Key: RANGER-2789
> URL: https://issues.apache.org/jira/browse/RANGER-2789
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.0.0
> Environment: hdp 2.2
>Reporter: RickyMa
>Priority: Minor
>  Labels: optimization, performance, ranger, rangeradmin, slow
> Attachments: RANGER-2789.patch, image-2020-04-10-19-37-31-570.png
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We have more than 1000 users in production environment.
> When calling API 'service/xusers/users?pageSize=1000=0' using HTTP 
> GET method, Ranger Admin takes 10+ minutes to response.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72307: RANGER-2775 : Pagination not working for role in Ranger admin UI page

[Pradeep Agrawal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72307/#review220279
---


Ship it!




Ship It!

- Pradeep Agrawal


On 四月 7, 2020, 2:09 p.m., Mahesh Bandal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72307/
> ---
> 
> (Updated 四月 7, 2020, 2:09 p.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Gautam Borad, Abhay Kulkarni, Madhan 
> Neethiraj, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, 
> and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2775
> https://issues.apache.org/jira/browse/RANGER-2775
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> This issue occurs when we click on role tab in Ranger admin UI page. In 
> response json "startIndex","pageSize","totalCount","resultSize", "sortType", 
> "sortBy" values were incorrect. To solve this issue I have created a new 
> method getRole which applies searchfilter on role object and returns 
> RangerRoleList response object with appropriate  values.
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/java/org/apache/ranger/biz/RoleDBStore.java 
> 9d2f0ba25 
>   security-admin/src/main/java/org/apache/ranger/rest/RoleREST.java aa031ae65 
> 
> 
> Diff: https://reviews.apache.org/r/72307/diff/2/
> 
> 
> Testing
> ---
> 
> 1.  Created/Deleted multiple roles with/without assigning groups/users to it.
> 2.  All roles are divided equally on every page. Max 25 roles are present on 
> each page.
> 
> 
> Thanks,
> 
> Mahesh Bandal
> 
>

[jira] [Updated] (RANGER-2790) Import start and import end are not in sequence

[Nitin Galave (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nitin Galave updated RANGER-2790:
-
Attachment: (was: 0001-RANGER-2790.patch)

> Import start and import end are not in sequence
> ---
>
> Key: RANGER-2790
> URL: https://issues.apache.org/jira/browse/RANGER-2790
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Reporter: Harshal Chavan
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (RANGER-2790) Import start and import end are not in sequence

[Nitin Galave (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nitin Galave updated RANGER-2790:
-
Attachment: 0001-RANGER-2790.patch

> Import start and import end are not in sequence
> ---
>
> Key: RANGER-2790
> URL: https://issues.apache.org/jira/browse/RANGER-2790
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Reporter: Harshal Chavan
>Priority: Major
> Attachments: 0001-RANGER-2790.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [ranger] rickyma commented on issue #61: RANGER-2789: GET API service/xusers/users turns very slow

[GitBox]

rickyma commented on issue #61: RANGER-2789: GET API service/xusers/users turns 
very slow 
URL: https://github.com/apache/ranger/pull/61#issuecomment-611998443
 
 
   Copied from the issue link: 
   We have more than 1000 users in production environment.
   
   When calling API 'service/xusers/users?pageSize=1000=0' using 
HTTP GET method, Ranger Admin takes 10+ minutes to response.
   
   I've already uploaded the patch. By applying this patch, the response time 
has decreased from 10+ minutes to around 30 seconds in our production 
environment.
   
   The problem of this issue is that the original code will query the database 
for every one of the 1000+ users:
   
https://github.com/apache/ranger/blob/master/security-admin/src/main/java/org/apache/ranger/service/XUserServiceBase.java#L84
   
   The main ideas of the uploaded patch are:
   1.Find all users in the database at one time.
   2.Use HashMap as cache to store all the database info in memory.
   3.Just use the cache during any follow-up db operation.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Comment Edited] (RANGER-2789) GET API service/xusers/users turns very slow when there are more than 1000 users

[RickyMa (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080429#comment-17080429
 ] 

RickyMa edited comment on RANGER-2789 at 4/10/20, 11:49 AM:


I've already uploaded the patch. By applying this patch, the response time has 
decreased from 10+ minutes to around 30 seconds in our production environment.

The problem of this issue is that the original code will query the database for 
every one of the 1000+ users:

[https://github.com/apache/ranger/blob/master/security-admin/src/main/java/org/apache/ranger/service/XUserServiceBase.java#L84]

!image-2020-04-10-19-37-31-570.png!

The main ideas of the uploaded patch are:
 # Find all users in the database at one time.
 # Use HashMap as cache to store all the database info in memory.
 # Just use the cache during any follow-up db operation.

 


was (Author: rickyma):
I've already uploaded the patch. By applying this patch, the response time has 
decreased from 10+ minutes to 30 seconds in our production environment.

The problem of this issue is that the original code will query the database for 
every one of the 1000+ users:

[https://github.com/apache/ranger/blob/master/security-admin/src/main/java/org/apache/ranger/service/XUserServiceBase.java#L84]

!image-2020-04-10-19-37-31-570.png!

The main ideas of the uploaded patch are:
 # Find all users in the database at one time.
 # Use HashMap as cache to store all the database info in memory.
 # Just use the cache during any follow-up db operation.

 

> GET API service/xusers/users turns very slow when there are more than 1000 
> users
> 
>
> Key: RANGER-2789
> URL: https://issues.apache.org/jira/browse/RANGER-2789
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.0.0
> Environment: hdp 2.2
>Reporter: RickyMa
>Priority: Minor
>  Labels: optimization, performance, ranger, rangeradmin, slow
> Attachments: RANGER-2789.patch, image-2020-04-10-19-37-31-570.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We have more than 1000 users in production environment.
> When calling API 'service/xusers/users?pageSize=1000=0' using HTTP 
> GET method, Ranger Admin takes 10+ minutes to response.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (RANGER-2789) GET API service/xusers/users turns very slow when there are more than 1000 users

[RickyMa (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080429#comment-17080429
 ] 

RickyMa edited comment on RANGER-2789 at 4/10/20, 11:48 AM:


I've already uploaded the patch. By applying this patch, the response time has 
decreased from 10+ minutes to 30 seconds in our production environment.

The problem of this issue is that the original code will query the database for 
every one of the 1000+ users:

[https://github.com/apache/ranger/blob/master/security-admin/src/main/java/org/apache/ranger/service/XUserServiceBase.java#L84]

!image-2020-04-10-19-37-31-570.png!

The main ideas of the uploaded patch are:
 # Find all users in the database at one time.
 # Use HashMap as cache to store all the database info in memory.
 # Just use the cache during any follow-up db operation.

 


was (Author: rickyma):
I've already uploaded the patch. By applying this patch, the response time has 
decreased from 10+ minutes to 30 seconds in our production environment.

The problem of this issue is that the original code will query the database for 
every one of the 1000+ users:

[https://github.com/apache/ranger/blob/master/security-admin/src/main/java/org/apache/ranger/service/XUserServiceBase.java#L84]

!image-2020-04-10-19-37-31-570.png!

The main ideas of the uploaded patch are:
 # Find all users in the database at one time.
 # Use HashMap as cache to store all the database info in memory.
 # Any follow-up db operation can just use the cache.

 

> GET API service/xusers/users turns very slow when there are more than 1000 
> users
> 
>
> Key: RANGER-2789
> URL: https://issues.apache.org/jira/browse/RANGER-2789
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.0.0
> Environment: hdp 2.2
>Reporter: RickyMa
>Priority: Minor
>  Labels: optimization, performance, ranger, rangeradmin, slow
> Attachments: RANGER-2789.patch, image-2020-04-10-19-37-31-570.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We have more than 1000 users in production environment.
> When calling API 'service/xusers/users?pageSize=1000=0' using HTTP 
> GET method, Ranger Admin takes 10+ minutes to response.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (RANGER-2789) GET API service/xusers/users turns very slow when there are more than 1000 users

[RickyMa (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080436#comment-17080436
 ] 

RickyMa commented on RANGER-2789:
-

Review request URL：[https://reviews.apache.org/r/72346/]

> GET API service/xusers/users turns very slow when there are more than 1000 
> users
> 
>
> Key: RANGER-2789
> URL: https://issues.apache.org/jira/browse/RANGER-2789
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.0.0
> Environment: hdp 2.2
>Reporter: RickyMa
>Priority: Minor
>  Labels: optimization, performance, ranger, rangeradmin, slow
> Attachments: RANGER-2789.patch, image-2020-04-10-19-37-31-570.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We have more than 1000 users in production environment.
> When calling API 'service/xusers/users?pageSize=1000=0' using HTTP 
> GET method, Ranger Admin takes 10+ minutes to response.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (RANGER-2789) GET API service/xusers/users turns very slow when there are more than 1000 users

[RickyMa (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080429#comment-17080429
 ] 

RickyMa edited comment on RANGER-2789 at 4/10/20, 11:47 AM:


I've already uploaded the patch. By applying this patch, the response time has 
decreased from 10+ minutes to 30 seconds in our production environment.

The problem of this issue is that the original code will query the database for 
every one of the 1000+ users:

[https://github.com/apache/ranger/blob/master/security-admin/src/main/java/org/apache/ranger/service/XUserServiceBase.java#L84]

!image-2020-04-10-19-37-31-570.png!

The main ideas of the uploaded patch are:
 # Find all users in the database at one time.
 # Use HashMap as cache to store all the database info in memory.
 # Any follow-up db operation can just use the cache.

 


was (Author: rickyma):
I've already uploaded the patch. By applying this patch, the response time has 
decreased from 10+ minutes to 30 seconds in our production environment.

The problem of this issue is that the original code will query the database for 
every one of the 1000+ users:

[https://github.com/apache/ranger/blob/master/security-admin/src/main/java/org/apache/ranger/service/XUserServiceBase.java#L84]

!image-2020-04-10-19-37-31-570.png!

The main ideas of the uploaded patch is:
 # Find all users in the database at one time.
 # Use HashMap as cache to store all the database info in memory.
 # Any follow-up db operation can just use the cache.

 

> GET API service/xusers/users turns very slow when there are more than 1000 
> users
> 
>
> Key: RANGER-2789
> URL: https://issues.apache.org/jira/browse/RANGER-2789
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.0.0
> Environment: hdp 2.2
>Reporter: RickyMa
>Priority: Minor
>  Labels: optimization, performance, ranger, rangeradmin, slow
> Attachments: RANGER-2789.patch, image-2020-04-10-19-37-31-570.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We have more than 1000 users in production environment.
> When calling API 'service/xusers/users?pageSize=1000=0' using HTTP 
> GET method, Ranger Admin takes 10+ minutes to response.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review Request 72346: RANGER-2789: GET API service/xusers/users turns very slow when there are more than 1000 users

[Haoxiang Ma]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72346/
---

Review request for ranger.


Bugs: RANGER-2789
https://issues.apache.org/jira/browse/RANGER-2789


Repository: ranger


Description
---

GET API service/xusers/users turns very slow when there are more than 1000 users


Diffs
-

  security-admin/src/main/java/org/apache/ranger/db/XXGroupUserDao.java 
1e41e708a 
  security-admin/src/main/java/org/apache/ranger/entity/XXUser.java 0464e7b6e 
  security-admin/src/main/java/org/apache/ranger/service/XUserService.java 
6ff8823da 
  security-admin/src/main/java/org/apache/ranger/service/XUserServiceBase.java 
9cdc14ebf 
  security-admin/src/main/java/org/apache/ranger/view/VXUser.java 96f6468f7 
  security-admin/src/main/resources/META-INF/jpa_named_queries.xml 22e71e549 


Diff: https://reviews.apache.org/r/72346/diff/1/


Testing
---

1.Compilation OK
2.Already used in production environment


Thanks,

Haoxiang Ma

[jira] [Commented] (RANGER-2789) GET API service/xusers/users turns very slow when there are more than 1000 users

[RickyMa (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080429#comment-17080429
 ] 

RickyMa commented on RANGER-2789:
-

I've already uploaded the patch. By applying this patch, the response time has 
decreased from 10+ minutes to 30 seconds in our production environment.

The problem of this issue is that the original code will query the database for 
every one of the 1000+ users:

[https://github.com/apache/ranger/blob/master/security-admin/src/main/java/org/apache/ranger/service/XUserServiceBase.java#L84]

!image-2020-04-10-19-37-31-570.png!

The main ideas of the uploaded patch is:
 # Find all users in the database at one time.
 # Use HashMap as cache to store all the database info in memory.
 # Any follow-up db operation can just use the cache.

 

> GET API service/xusers/users turns very slow when there are more than 1000 
> users
> 
>
> Key: RANGER-2789
> URL: https://issues.apache.org/jira/browse/RANGER-2789
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.0.0
> Environment: hdp 2.2
>Reporter: RickyMa
>Priority: Minor
>  Labels: optimization, performance, ranger, rangeradmin, slow
> Attachments: RANGER-2789.patch, image-2020-04-10-19-37-31-570.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We have more than 1000 users in production environment.
> When calling API 'service/xusers/users?pageSize=1000=0' using HTTP 
> GET method, Ranger Admin takes 10+ minutes to response.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (RANGER-2789) GET API service/xusers/users turns very slow when there are more than 1000 users

[RickyMa (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

RickyMa updated RANGER-2789:

Attachment: image-2020-04-10-19-37-31-570.png

> GET API service/xusers/users turns very slow when there are more than 1000 
> users
> 
>
> Key: RANGER-2789
> URL: https://issues.apache.org/jira/browse/RANGER-2789
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.0.0
> Environment: hdp 2.2
>Reporter: RickyMa
>Priority: Minor
>  Labels: optimization, performance, ranger, rangeradmin, slow
> Attachments: RANGER-2789.patch, image-2020-04-10-19-37-31-570.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We have more than 1000 users in production environment.
> When calling API 'service/xusers/users?pageSize=1000=0' using HTTP 
> GET method, Ranger Admin takes 10+ minutes to response.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (RANGER-2789) GET API service/xusers/users turns very slow when there are more than 1000 users

[RickyMa (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

RickyMa updated RANGER-2789:

Attachment: (was: RANGER-2789.patch)

> GET API service/xusers/users turns very slow when there are more than 1000 
> users
> 
>
> Key: RANGER-2789
> URL: https://issues.apache.org/jira/browse/RANGER-2789
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.0.0
> Environment: hdp 2.2
>Reporter: RickyMa
>Priority: Minor
>  Labels: optimization, performance, ranger, rangeradmin, slow
> Attachments: RANGER-2789.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We have more than 1000 users in production environment.
> When calling API 'service/xusers/users?pageSize=1000=0' using HTTP 
> GET method, Ranger Admin takes 10+ minutes to response.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (RANGER-2789) GET API service/xusers/users turns very slow when there are more than 1000 users

[RickyMa (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

RickyMa updated RANGER-2789:

Attachment: RANGER-2789.patch

> GET API service/xusers/users turns very slow when there are more than 1000 
> users
> 
>
> Key: RANGER-2789
> URL: https://issues.apache.org/jira/browse/RANGER-2789
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.0.0
> Environment: hdp 2.2
>Reporter: RickyMa
>Priority: Minor
>  Labels: optimization, performance, ranger, rangeradmin, slow
> Attachments: RANGER-2789.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We have more than 1000 users in production environment.
> When calling API 'service/xusers/users?pageSize=1000=0' using HTTP 
> GET method, Ranger Admin takes 10+ minutes to response.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (RANGER-2789) GET API service/xusers/users turns very slow when there are more than 1000 users

[RickyMa (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

RickyMa updated RANGER-2789:

Attachment: RANGER-2789.patch

> GET API service/xusers/users turns very slow when there are more than 1000 
> users
> 
>
> Key: RANGER-2789
> URL: https://issues.apache.org/jira/browse/RANGER-2789
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.0.0
> Environment: hdp 2.2
>Reporter: RickyMa
>Priority: Minor
>  Labels: optimization, performance, ranger, rangeradmin, slow
> Attachments: RANGER-2789.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We have more than 1000 users in production environment.
> When calling API 'service/xusers/users?pageSize=1000=0' using HTTP 
> GET method, Ranger Admin takes 10+ minutes to response.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [ranger] rickyma opened a new pull request #61: RANGER-2789: GET API service/xusers/users turns very slow

[GitBox]

rickyma opened a new pull request #61: RANGER-2789: GET API 
service/xusers/users turns very slow 
URL: https://github.com/apache/ranger/pull/61
 
 
   RANGER-2789: GET API service/xusers/users turns very slow when there are 
more than 1000 users.
   https://issues.apache.org/jira/browse/RANGER-2789


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (RANGER-2790) Import start and import end are not in sequence

[Harshal Chavan (Jira)]

Harshal Chavan created RANGER-2790:
--

 Summary: Import start and import end are not in sequence
 Key: RANGER-2790
 URL: https://issues.apache.org/jira/browse/RANGER-2790
 Project: Ranger
  Issue Type: Bug
  Components: Ranger
Reporter: Harshal Chavan






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (RANGER-2789) GET API service/xusers/users turns very slow when there are more than 1000 users

[MaHaoxiang (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080372#comment-17080372
 ] 

MaHaoxiang commented on RANGER-2789:


We found the problem, later I will upload the patch attachment.

> GET API service/xusers/users turns very slow when there are more than 1000 
> users
> 
>
> Key: RANGER-2789
> URL: https://issues.apache.org/jira/browse/RANGER-2789
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.0.0
> Environment: hdp 2.2
>Reporter: MaHaoxiang
>Priority: Minor
>  Labels: optimization, performance, ranger, rangeradmin, slow
>
> We have more than 1000 users in production environment.
> When calling API 'service/xusers/users?pageSize=1000=0' using HTTP 
> GET method, Ranger Admin takes 10+ minutes to response.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72272: Upgrade and improve Presto plugin

[Bolke de Bruin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72272/
---

(Updated April 10, 2020, 9:09 a.m.)


Review request for ranger, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, and 
Ramesh Mani.


Changes
---

remove unused import


Bugs: https://issues.apache.org/jira/browse/RANGER-2754

https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/RANGER-2754


Repository: ranger


Description
---

Upgrade and improve Presto plugin
- Presto SQL 331 has changed its security API and has Row level / column 
masking functionality
- Upgraded Hadoop dependency to 3.1.3 (from 3.1.1) due to improved security 
handling
- New features like session properties and system properties


Diffs (updated)
-

  agents-common/src/main/resources/service-defs/ranger-servicedef-presto.json 
56a8f5ac0 
  distro/src/main/assembly/plugin-presto.xml d2075bfe7 
  plugin-presto/pom.xml b63f7dede 
  
plugin-presto/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControl.java
 3ab63f590 
  
plugin-presto/src/test/java/org/apache/ranger/authorization/presto/authorizer/RangerAdminClientImpl.java
 PRE-CREATION 
  
plugin-presto/src/test/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControlTest.java
 PRE-CREATION 
  plugin-presto/src/test/resources/log4j.properties PRE-CREATION 
  plugin-presto/src/test/resources/presto-policies.json PRE-CREATION 
  plugin-presto/src/test/resources/ranger-presto-security.xml PRE-CREATION 
  pom.xml 22926fd7d 
  ranger-presto-plugin-shim/pom.xml d8ff88d0f 
  
ranger-presto-plugin-shim/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerConfig.java
 67b0d2434 
  
ranger-presto-plugin-shim/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControl.java
 e89f646e1 


Diff: https://reviews.apache.org/r/72272/diff/4/

Changes: https://reviews.apache.org/r/72272/diff/3-4/


Testing
---

- New Unit tests added
- Tested locally in production


Thanks,

Bolke de Bruin

[jira] [Updated] (RANGER-2754) Update presto dependency and implement row/column level security

[Bolke de Bruin (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated RANGER-2754:
---
Attachment: 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch

> Update presto dependency and implement row/column level security
> 
>
> Key: RANGER-2754
> URL: https://issues.apache.org/jira/browse/RANGER-2754
> Project: Ranger
>  Issue Type: Improvement
>  Components: plugins
>Affects Versions: master
>Reporter: Bolke de Bruin
>Assignee: Bolke de Bruin
>Priority: Major
> Attachments: 
> 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch, 
> 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch, 
> 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch, 
> RANGER-2754-v2.patch, RANGER-2754.patch
>
>
> 1. PrestoSql has changed its Security API hence the Ranger plugin has stopped 
> working for versions > ~321. 
> 2. Presto master now has row/column level security support



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72331: Upgrade log4j dependency

[Harshal Chavan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72331/#review220278
---



Ranger Tagsync start is failing with an error

Exception in thread "main" java.lang.NoClassDefFoundError: 
org/apache/log4j/Logger
at 
org.apache.ranger.tagsync.process.TagSynchronizer.(TagSynchronizer.java:40)

- Harshal Chavan


On April 7, 2020, 12:17 p.m., Bolke de Bruin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72331/
> ---
> 
> (Updated April 7, 2020, 12:17 p.m.)
> 
> 
> Review request for ranger and Madhan Neethiraj.
> 
> 
> Bugs: RANGER-2782
> https://issues.apache.org/jira/browse/RANGER-2782
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> The current log4j version in ranger is end of life and contains critical 
> security Vulnerabilities
> CVE-2019-17571
> 
> 
> Diffs
> -
> 
>   agents-audit/pom.xml 8ac1edf4f 
>   agents-common/pom.xml c78dc5fc2 
>   agents-cred/pom.xml cd1b8f3e3 
>   embeddedwebserver/pom.xml 8574c5721 
>   kms/pom.xml 3bf20fdd4 
>   plugin-schema-registry/pom.xml 6bd2d9766 
>   pom.xml 22926fd7d 
>   ranger-examples/sampleapp/pom.xml 494fea3a5 
>   security-admin/pom.xml fc4a20020 
>   tagsync/pom.xml b8340c063 
>   ugsync/pom.xml b1d695af1 
>   unixauthclient/pom.xml fa5d40966 
>   unixauthservice/pom.xml 7cd6aecc2 
> 
> 
> Diff: https://reviews.apache.org/r/72331/diff/1/
> 
> 
> Testing
> ---
> 
> - started ranger
> 
> 
> Thanks,
> 
> Bolke de Bruin
> 
>

[jira] [Updated] (RANGER-2754) Update presto dependency and implement row/column level security

[Bolke de Bruin (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin updated RANGER-2754:
---
Attachment: 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch

> Update presto dependency and implement row/column level security
> 
>
> Key: RANGER-2754
> URL: https://issues.apache.org/jira/browse/RANGER-2754
> Project: Ranger
>  Issue Type: Improvement
>  Components: plugins
>Affects Versions: master
>Reporter: Bolke de Bruin
>Assignee: Bolke de Bruin
>Priority: Major
> Attachments: 
> 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch, 
> 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch, 
> RANGER-2754-v2.patch, RANGER-2754.patch
>
>
> 1. PrestoSql has changed its Security API hence the Ranger plugin has stopped 
> working for versions > ~321. 
> 2. Presto master now has row/column level security support



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (RANGER-2754) Update presto dependency and implement row/column level security

[Bolke de Bruin (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080363#comment-17080363
 ] 

Bolke de Bruin commented on RANGER-2754:


[~liujiayi771] I have included your changes in the latest update to this patch. 
I hope you don't mind.

> Update presto dependency and implement row/column level security
> 
>
> Key: RANGER-2754
> URL: https://issues.apache.org/jira/browse/RANGER-2754
> Project: Ranger
>  Issue Type: Improvement
>  Components: plugins
>Affects Versions: master
>Reporter: Bolke de Bruin
>Assignee: Bolke de Bruin
>Priority: Major
> Attachments: 
> 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch, 
> 0001-RANGER-2754-Upgrade-presto-dependency-and-improve-lo.patch, 
> RANGER-2754-v2.patch, RANGER-2754.patch
>
>
> 1. PrestoSql has changed its Security API hence the Ranger plugin has stopped 
> working for versions > ~321. 
> 2. Presto master now has row/column level security support



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72272: Upgrade and improve Presto plugin

[Bolke de Bruin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72272/
---

(Updated April 10, 2020, 8:55 a.m.)


Review request for ranger, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, and 
Ramesh Mani.


Changes
---

Addressed issues


Bugs: https://issues.apache.org/jira/browse/RANGER-2754

https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/RANGER-2754


Repository: ranger


Description
---

Upgrade and improve Presto plugin
- Presto SQL 331 has changed its security API and has Row level / column 
masking functionality
- Upgraded Hadoop dependency to 3.1.3 (from 3.1.1) due to improved security 
handling
- New features like session properties and system properties


Diffs (updated)
-

  agents-common/src/main/resources/service-defs/ranger-servicedef-presto.json 
56a8f5ac0 
  distro/src/main/assembly/plugin-presto.xml d2075bfe7 
  plugin-presto/pom.xml b63f7dede 
  
plugin-presto/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControl.java
 3ab63f590 
  
plugin-presto/src/test/java/org/apache/ranger/authorization/presto/authorizer/RangerAdminClientImpl.java
 PRE-CREATION 
  
plugin-presto/src/test/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControlTest.java
 PRE-CREATION 
  plugin-presto/src/test/resources/log4j.properties PRE-CREATION 
  plugin-presto/src/test/resources/presto-policies.json PRE-CREATION 
  plugin-presto/src/test/resources/ranger-presto-security.xml PRE-CREATION 
  pom.xml 22926fd7d 
  ranger-presto-plugin-shim/pom.xml d8ff88d0f 
  
ranger-presto-plugin-shim/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerConfig.java
 67b0d2434 
  
ranger-presto-plugin-shim/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControl.java
 e89f646e1 


Diff: https://reviews.apache.org/r/72272/diff/3/

Changes: https://reviews.apache.org/r/72272/diff/2-3/


Testing
---

- New Unit tests added
- Tested locally in production


Thanks,

Bolke de Bruin

Re: Review Request 72272: Upgrade and improve Presto plugin

[Bolke de Bruin]

> On April 5, 2020, 6:09 p.m., Madhan Neethiraj wrote:
> > plugin-presto/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControl.java
> > Lines 74 (patched)
> > 
> >
> > Since 'useUgi' is set only in the constructor,  consider marking this 
> > as a 'final'. Same applies for 'rangerPlugin' as well. Please review.

that won't work for useUgi. Declaring it final thecompiler complains that it 
cannot set it.


> On April 5, 2020, 6:09 p.m., Madhan Neethiraj wrote:
> > plugin-presto/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControl.java
> > Lines 157 (patched)
> > 
> >
> > result.isRowFilterEnabled() already checks if the filter-expr is empty 
> > or not; so 'StringUtils.isNotEmpty(result.getFilterExpr());' is not needed 
> > here. Please  review.

came from the Hive Authorizer. Removed


> On April 5, 2020, 6:09 p.m., Madhan Neethiraj wrote:
> > ranger-presto-plugin-shim/pom.xml
> > Lines 93 (patched)
> > 
> >
> > The changes in this module don't seem to refer hadoop-hdfs library 
> > contents directly. Is it necessary to explicitly add this dependency?

I'm not getting it into the assembly otherwise for auditing purposes? please 
advise.


> On April 5, 2020, 6:09 p.m., Madhan Neethiraj wrote:
> > ranger-presto-plugin-shim/pom.xml
> > Lines 134 (patched)
> > 
> >
> > The changes in this module don't seem to refer protobuf-java library 
> > contents directly. Is it necessary to explicitly add this dependency?

Idem as above. How to get it in the assembly? It seems not to be included 
otherwise (struggling with that overall btw)


- Bolke


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72272/#review220223
---


On March 30, 2020, 5:20 p.m., Bolke de Bruin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72272/
> ---
> 
> (Updated March 30, 2020, 5:20 p.m.)
> 
> 
> Review request for ranger, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, 
> and Ramesh Mani.
> 
> 
> Bugs: https://issues.apache.org/jira/browse/RANGER-2754
> 
> https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/RANGER-2754
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Upgrade and improve Presto plugin
> - Presto SQL 331 has changed its security API and has Row level / column 
> masking functionality
> - Upgraded Hadoop dependency to 3.1.3 (from 3.1.1) due to improved security 
> handling
> - New features like session properties and system properties
> 
> 
> Diffs
> -
> 
>   agents-common/src/main/resources/service-defs/ranger-servicedef-presto.json 
> 56a8f5ac0 
>   distro/src/main/assembly/plugin-presto.xml d2075bfe7 
>   plugin-presto/pom.xml b63f7dede 
>   
> plugin-presto/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControl.java
>  3ab63f590 
>   
> plugin-presto/src/test/java/org/apache/ranger/authorization/presto/authorizer/RangerAdminClientImpl.java
>  PRE-CREATION 
>   
> plugin-presto/src/test/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControlTest.java
>  PRE-CREATION 
>   plugin-presto/src/test/resources/log4j.properties PRE-CREATION 
>   plugin-presto/src/test/resources/presto-policies.json PRE-CREATION 
>   plugin-presto/src/test/resources/ranger-presto-security.xml PRE-CREATION 
>   pom.xml 22926fd7d 
>   ranger-presto-plugin-shim/pom.xml d8ff88d0f 
>   
> ranger-presto-plugin-shim/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerConfig.java
>  67b0d2434 
>   
> ranger-presto-plugin-shim/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControl.java
>  e89f646e1 
> 
> 
> Diff: https://reviews.apache.org/r/72272/diff/2/
> 
> 
> Testing
> ---
> 
> - New Unit tests added
> - Tested locally in production
> 
> 
> Thanks,
> 
> Bolke de Bruin
> 
>

[jira] [Created] (RANGER-2789) GET API service/xusers/users turns very slow when there are more than 1000 users

[MaHaoxiang (Jira)]

MaHaoxiang created RANGER-2789:
--

 Summary: GET API service/xusers/users turns very slow when there 
are more than 1000 users
 Key: RANGER-2789
 URL: https://issues.apache.org/jira/browse/RANGER-2789
 Project: Ranger
  Issue Type: Bug
  Components: Ranger
Affects Versions: 2.0.0
 Environment: hdp 2.2
Reporter: MaHaoxiang


We have more than 1000 users in production environment.

When calling API 'service/xusers/users?pageSize=1000=0' using HTTP 
GET method, Ranger Admin takes 10+ minutes to response.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (RANGER-2782) Upgrade log4j dependency

[Bhavik Patel (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080350#comment-17080350
 ] 

Bhavik Patel commented on RANGER-2782:
--

[~vel] and [~bolke]:  Migrating logging to 
[slf4j|https://issues.apache.org/jira/browse/RANGER-2545] rather-than updating 
log4j version  

> Upgrade log4j dependency
> 
>
> Key: RANGER-2782
> URL: https://issues.apache.org/jira/browse/RANGER-2782
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.0.0
>Reporter: Bolke de Bruin
>Assignee: Bolke de Bruin
>Priority: Blocker
> Fix For: 2.1.0
>
> Attachments: 
> 0001-RANGER-2782-Upgrade-log4j-to-a-supported-version.patch
>
>
> The current log4j version in ranger is end of life and contains critical 
> security Vulnerabilities
> CVE-2019-17571



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (RANGER-2782) Upgrade log4j dependency

[Bolke de Bruin (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080307#comment-17080307
 ] 

Bolke de Bruin commented on RANGER-2782:


Yes working on that. It will be much larger though.

> Upgrade log4j dependency
> 
>
> Key: RANGER-2782
> URL: https://issues.apache.org/jira/browse/RANGER-2782
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.0.0
>Reporter: Bolke de Bruin
>Assignee: Bolke de Bruin
>Priority: Blocker
> Fix For: 2.1.0
>
> Attachments: 
> 0001-RANGER-2782-Upgrade-log4j-to-a-supported-version.patch
>
>
> The current log4j version in ranger is end of life and contains critical 
> security Vulnerabilities
> CVE-2019-17571



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72061: RANGER-2716: Add catalogs/schemas/tables filter in presto plugin

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72061/#review220277
---




ranger-presto-plugin-shim/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControl.java
Line 106 (original), 106 (patched)


deactivatePluginClassLoader() should be called before the shim method 
returns - as shown below:

  try {
activatePluginClassLoader();

return systemAccessControlImpl.filterCatalogs(identity, catalogs);
  } finally {
deactivatePluginClassLoader();
  }

Please review and update other methods as well (#167,#229).


- Madhan Neethiraj


On Jan. 30, 2020, 12:45 p.m., Jiayi Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72061/
> ---
> 
> (Updated Jan. 30, 2020, 12:45 p.m.)
> 
> 
> Review request for ranger, Don Bosco Durai, Colm O hEigeartaigh, Abhay 
> Kulkarni, Madhan Neethiraj, Mehul Parikh, Nitin Galave, pengjianhua, Pradeep 
> Agrawal, Ramesh Mani, Selvamohan Neethiraj, Velmurugan Periasamy, and Zsombor 
> Gegesy.
> 
> 
> Bugs: RANGER-2716
> https://issues.apache.org/jira/browse/RANGER-2716
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Presto plugin returns the input set of catalogs/schemas/tables directly at 
> present. However, this causes a problem, that is, when the user uses show 
> catalogs/schemas/tables, all catalogs/schemas/tables are displayed, even 
> though the user does not have the permissions to display all 
> catalogs/schemas/tables.
> 
> We need to fix the filterCatalogs/Schemas/Tables functions to filter 
> catalogs/schemas/tables, so that the user can only see the 
> catalogs/schemas/tables in which the user has SELECT permission.
> 
> 
> Diffs
> -
> 
>   
> plugin-presto/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControl.java
>  3ab63f590 
>   
> ranger-presto-plugin-shim/src/main/java/org/apache/ranger/authorization/presto/authorizer/RangerSystemAccessControl.java
>  e89f646e1 
> 
> 
> Diff: https://reviews.apache.org/r/72061/diff/1/
> 
> 
> Testing
> ---
> 
> show catalogs/schemas/tables only display the catalogs/schemas/tables in 
> which the user has SELECT permission.
> 
> 
> Thanks,
> 
> Jiayi Liu
> 
>

Review Request 72344: RANGER-2786: Ranger usersync group modifyTimestamp parsing should be in 24 hours format

[Pradeep Agrawal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72344/
---

Review request for ranger, Ankita Sinha, Dhaval Shah, Dineshkumar Yadav, Gautam 
Borad, Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, 
Mehul Parikh, Nitin Galave, Nixon Rodrigues, Ramesh Mani, Sailaja Polavarapu, 
and Velmurugan Periasamy.


Bugs: RANGER-2786
https://issues.apache.org/jira/browse/RANGER-2786


Repository: ranger


Description
---

**Problem Statement:** Currently Ranger usersync group modifyTimestamp parsing 
is in 12 hours format. (not sure its as per the requirement or by mistake). for 
the parsing date format is choosen as "MMddhhmmss" where hh can convert 
hour value to 12 hours format. for example: if a group is synced at 00:05am UTC 
then this will highest user sync time stamp but after the conversion value 
becomes 12:05 which is actually future timestamp and for the next 12 hours 
delta sync of groups may not sync any groups. 

**Proposed solution:** Conversion should be done in 24 hours format by using 
hours in HH format rather hh.

**Note:** 
1) To reproduce this issue frequently i have to switch my default ldap delta 
sync interval to 5  minute as compare to 1 hour.
2) I am not sure about the current behaviour of different ldap servers and AD 
without this patch. some might be working so i request reviewers to add their 
feedback on this.
3) It will be better if reviewers can test this patch in their environment and 
share  their experiance here or on the jira.


Diffs
-

  
ugsync/src/main/java/org/apache/ranger/ldapusersync/process/LdapDeltaUserGroupBuilder.java
 3ff247005 


Diff: https://reviews.apache.org/r/72344/diff/1/


Testing
---

without this patch groups are not syncing if usersync is started one day ago 
and group is created next day after 00:00 UTC
Tested this patch by creating ldap group around 00:05am UTC and wait for the 
sync.

**Result:** Groups are syncing successfully with this patch.


Thanks,

Pradeep Agrawal