Re: Review Request 73719: RANGER-3435: Add unique index on guid, service and zone_id column of x_policy table

2021-12-02 Thread Kishor Gollapalliwar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73719/#review223792
---


Ship it!




Ship It!

- Kishor Gollapalliwar


On Nov. 24, 2021, 3:44 p.m., Pradeep Agrawal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73719/
> ---
> 
> (Updated Nov. 24, 2021, 3:44 p.m.)
> 
> 
> Review request for ranger, Dineshkumar Yadav, Abhay Kulkarni, Madhan 
> Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan 
> Periasamy.
> 
> 
> Bugs: RANGER-3435
> https://issues.apache.org/jira/browse/RANGER-3435
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> **Problem Statement:** After first commit of RANGER-3435 
> https://reviews.apache.org/r/73594/  x_policy table have unique constraint on 
> guid and service column. if there are more than one zone and policies 
> exported from one zone is imported in the other zone then policy guid will 
> remain same. since guid are same for both the policies, we need to restrict 1 
> entry only for the same guid under a specific service and zone.
> 
> **Proposed Solution:**
> it will be better to include zone_id also with guid and service column for 
> the unique key creation so that the same restriction can be enforced from db 
> end.
> 
> 
> Diffs
> -
> 
>   security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 
> e444e78eb 
>   
> security-admin/db/mysql/patches/057-add-unique-constraint-on-x_policy-table-guid-service-column.sql
>  357b7efe3 
>   security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql 
> 9e5da70fc 
>   
> security-admin/db/oracle/patches/057-add-unique-constraint-on-x_policy-table-guid-service-column.sql
>  580841c6b 
>   security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 
> 9fd45037b 
>   
> security-admin/db/postgres/patches/057-add-unique-constraint-on-x_policy-table-guid-service-column.sql
>  81718aae4 
>   
> security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql
>  bdccecc19 
>   
> security-admin/db/sqlanywhere/patches/057-add-unique-constraint-on-x_policy-table-guid-service-column.sql
>  16ad476e4 
>   security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 
> 8515ac0b0 
>   
> security-admin/db/sqlserver/patches/057-add-unique-constraint-on-x_policy-table-guid-service-column.sql
>  3037988e2 
> 
> 
> Diff: https://reviews.apache.org/r/73719/diff/3/
> 
> 
> Testing
> ---
> 
> Tested the patch for MySQL, Oracle, Postgres and MSSQL.
> unique constraint is being created in x_policy table for a fresh installation 
> and upgrade case as well.
> 
> 
> Thanks,
> 
> Pradeep Agrawal
> 
>

[jira] [Created] (RANGER-3531) Store Spooled Audits in Avro Format

2021-12-02 Thread David Mollitor (Jira) 
David Mollitor created RANGER-3531:
--

 Summary: Store Spooled Audits in Avro Format
 Key: RANGER-3531
 URL: https://issues.apache.org/jira/browse/RANGER-3531
 Project: Ranger
  Issue Type: Improvement
  Components: audit
Reporter: David Mollitor


Ranger plugins store Audit records in JSON format when spooling them to disk.

 

For compactness and performance sake, please allow Ranger plugs to (optionally) 
use Avro format for spooled audits.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (RANGER-3532) Delete Archived Spooled Audit Logs Based on TTL

2021-12-02 Thread David Mollitor (Jira) 
David Mollitor created RANGER-3532:
--

 Summary: Delete Archived Spooled Audit Logs Based on TTL
 Key: RANGER-3532
 URL: https://issues.apache.org/jira/browse/RANGER-3532
 Project: Ranger
  Issue Type: Improvement
  Components: audit
Reporter: David Mollitor


As I understand it,...

When an audit destination (HDFS/SOLR) is offline, Ranger plugin can spool audit 
messages to the local disk.  Once the destination comes back online, the Ranger 
plugin will resume transmitting audit messages.  Once all audit messages are 
transmitted, the log file containing the message is sent to the audit 'archive' 
directory.  From there, if there are more than (configurable) 100 archived 
audit log files, then some number of files are deleted to bring that number 
down to 100.

 

This can be problematic if the number of audits is very large (and therefore 
spooled audit log files are very large) and they can sit in the archive 
directory for very long periods of time.  As I understand it, the only way for 
them to be deleted is if another outage event occurs and more files are 
created, always keeping the total number of files at 100.

 

Please add an additional criteria for deleting files: TTL

 

Delete archived audit files which are older than (configurable) a week.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (RANGER-3519) Provide an option to optimize space needed by Trie objects

2021-12-02 Thread Abhay Kulkarni (Jira) 


[ 
https://issues.apache.org/jira/browse/RANGER-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17452528#comment-17452528
 ] 

Abhay Kulkarni commented on RANGER-3519:


Additional commit:

master:

[https://github.com/apache/ranger/commit/5852efde1cba728ad580231ad02145ea72861186]

 

> Provide an option to optimize space needed by Trie objects
> --
>
> Key: RANGER-3519
> URL: https://issues.apache.org/jira/browse/RANGER-3519
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Reporter: Abhay Kulkarni
>Assignee: Abhay Kulkarni
>Priority: Major
> Fix For: 3.0.0
>
>
> When the number of policies (and/or tagged resources) is large, the data 
> structures used by Ranger as indexes for policies (and/or tagged resources) 
> may need a very large heap memory because they are optimized for fast lookup. 
> It is desirable to be able to configure Ranger to have these structures 
> optimized for space in order to keep the heap requirements within acceptable 
> limit at the cost of somewhat slower lookup.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: Review Request 73720: RANGER-3439: REST api to get or delete ranger policy based on guid, service name and zone name

2021-12-02 Thread Abhay Kulkarni 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73720/#review223794
---




security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java
Line 416 (original), 416 (patched)


Please update the PublicAPI documentation accordingly. (lines 416 and 521)


- Abhay Kulkarni


On Nov. 22, 2021, 10:22 a.m., Pradeep Agrawal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73720/
> ---
> 
> (Updated Nov. 22, 2021, 10:22 a.m.)
> 
> 
> Review request for ranger, Dineshkumar Yadav, Abhay Kulkarni, Madhan 
> Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan 
> Periasamy.
> 
> 
> Bugs: RANGER-3439
> https://issues.apache.org/jira/browse/RANGER-3439
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> **Problem statement:** This RR is modification of the work done in 
> RANGER-3439 (https://reviews.apache.org/r/73601/) which is already committed, 
> changes are needed for the changes proposed in 
> https://reviews.apache.org/r/73719/
> 
> **Proposed solution:** API getPolicyByGUIDAndServiceName and 
> deletePolicyByGUIDAndServiceName can be modified to address the requirement 
> which shall accept the guid service name and zone name as request parameters 
> input and provide the get policy or delete policy option.
> API:
> a) getPolicyByGUIDAndServiceNameAndZoneName(guid, service, zone): reads the 
> input values and returns the policy object.
> b) deletePolicyByGUIDAndServiceNameAndZoneName(guid, service, zone) : reads 
> the input values and deletes the respective policy object.
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 
> cb57b9913 
>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyDao.java 
> 3558337a3 
>   security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java 
> 6ab3d52a0 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 
> 3ba29653b 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml a19f7f1d8 
> 
> 
> Diff: https://reviews.apache.org/r/73720/diff/2/
> 
> 
> Testing
> ---
> 
> Tested getPolicyByGUIDAndServiceNameAndZoneName() API and was able to recieve 
> the matching policy object.
> Tested deletePolicyByGUIDAndServiceNameAndZoneName() API and was able to 
> delete the respective policy object.
> 
> **Sample curl requests:**
> 
> curl -u admin:Ranger1234 -H "Accept: application/json" -H "Content-Type: 
> application/json" -X GET 
> 'http://localhost:6080/service/plugins/policies/guid/0be7457b-35c7-4ca9-bd08-938d98a3e724?serviceName=cm_hive'
> 
> curl -u admin:Ranger1234 -H "Accept: application/json" -H "Content-Type: 
> application/json" -X GET 
> 'http://localhost:6080/service/plugins/policies/guid/ad88dd6f-1d85-4a67-8e84-813809c83da0?serviceName=cm_hive&zoneName=zone1'
> 
> 
> curl -u admin:Ranger1234 -H "Accept: application/json" -H "Content-Type: 
> application/json" -X DELETE 
> 'http://localhost:6080/service/plugins/policies/guid/0be7457b-35c7-4ca9-bd08-938d98a3e724?serviceName=cm_hive'
> 
> curl -u admin:Ranger1234 -H "Accept: application/json" -H "Content-Type: 
> application/json" -X DELETE 
> 'http://localhost:6080/service/plugins/policies/guid/ad88dd6f-1d85-4a67-8e84-813809c83da0?serviceName=cm_hive&zoneName=zone1'
> 
> 
> Thanks,
> 
> Pradeep Agrawal
> 
>

Re: Review Request 73720: RANGER-3439: REST api to get or delete ranger policy based on guid, service name and zone name

2021-12-02 Thread Abhay Kulkarni 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73720/#review223795
---


Ship it!




Ship It!

- Abhay Kulkarni


On Nov. 22, 2021, 10:22 a.m., Pradeep Agrawal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73720/
> ---
> 
> (Updated Nov. 22, 2021, 10:22 a.m.)
> 
> 
> Review request for ranger, Dineshkumar Yadav, Abhay Kulkarni, Madhan 
> Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan 
> Periasamy.
> 
> 
> Bugs: RANGER-3439
> https://issues.apache.org/jira/browse/RANGER-3439
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> **Problem statement:** This RR is modification of the work done in 
> RANGER-3439 (https://reviews.apache.org/r/73601/) which is already committed, 
> changes are needed for the changes proposed in 
> https://reviews.apache.org/r/73719/
> 
> **Proposed solution:** API getPolicyByGUIDAndServiceName and 
> deletePolicyByGUIDAndServiceName can be modified to address the requirement 
> which shall accept the guid service name and zone name as request parameters 
> input and provide the get policy or delete policy option.
> API:
> a) getPolicyByGUIDAndServiceNameAndZoneName(guid, service, zone): reads the 
> input values and returns the policy object.
> b) deletePolicyByGUIDAndServiceNameAndZoneName(guid, service, zone) : reads 
> the input values and deletes the respective policy object.
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 
> cb57b9913 
>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyDao.java 
> 3558337a3 
>   security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java 
> 6ab3d52a0 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 
> 3ba29653b 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml a19f7f1d8 
> 
> 
> Diff: https://reviews.apache.org/r/73720/diff/2/
> 
> 
> Testing
> ---
> 
> Tested getPolicyByGUIDAndServiceNameAndZoneName() API and was able to recieve 
> the matching policy object.
> Tested deletePolicyByGUIDAndServiceNameAndZoneName() API and was able to 
> delete the respective policy object.
> 
> **Sample curl requests:**
> 
> curl -u admin:Ranger1234 -H "Accept: application/json" -H "Content-Type: 
> application/json" -X GET 
> 'http://localhost:6080/service/plugins/policies/guid/0be7457b-35c7-4ca9-bd08-938d98a3e724?serviceName=cm_hive'
> 
> curl -u admin:Ranger1234 -H "Accept: application/json" -H "Content-Type: 
> application/json" -X GET 
> 'http://localhost:6080/service/plugins/policies/guid/ad88dd6f-1d85-4a67-8e84-813809c83da0?serviceName=cm_hive&zoneName=zone1'
> 
> 
> curl -u admin:Ranger1234 -H "Accept: application/json" -H "Content-Type: 
> application/json" -X DELETE 
> 'http://localhost:6080/service/plugins/policies/guid/0be7457b-35c7-4ca9-bd08-938d98a3e724?serviceName=cm_hive'
> 
> curl -u admin:Ranger1234 -H "Accept: application/json" -H "Content-Type: 
> application/json" -X DELETE 
> 'http://localhost:6080/service/plugins/policies/guid/ad88dd6f-1d85-4a67-8e84-813809c83da0?serviceName=cm_hive&zoneName=zone1'
> 
> 
> Thanks,
> 
> Pradeep Agrawal
> 
>

Re: Review Request 73720: RANGER-3439: REST api to get or delete ranger policy based on guid, service name and zone name

2021-12-02 Thread Pradeep Agrawal 


> On Dec. 2, 2021, 9:11 p.m., Abhay Kulkarni wrote:
> > security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java
> > Line 416 (original), 416 (patched)
> > 
> >
> > Please update the PublicAPI documentation accordingly. (lines 416 and 
> > 521)

This is being tracked separately here : 
https://issues.apache.org/jira/browse/RANGER-3501


- Pradeep


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73720/#review223794
---


On Nov. 22, 2021, 10:22 a.m., Pradeep Agrawal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73720/
> ---
> 
> (Updated Nov. 22, 2021, 10:22 a.m.)
> 
> 
> Review request for ranger, Dineshkumar Yadav, Abhay Kulkarni, Madhan 
> Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan 
> Periasamy.
> 
> 
> Bugs: RANGER-3439
> https://issues.apache.org/jira/browse/RANGER-3439
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> **Problem statement:** This RR is modification of the work done in 
> RANGER-3439 (https://reviews.apache.org/r/73601/) which is already committed, 
> changes are needed for the changes proposed in 
> https://reviews.apache.org/r/73719/
> 
> **Proposed solution:** API getPolicyByGUIDAndServiceName and 
> deletePolicyByGUIDAndServiceName can be modified to address the requirement 
> which shall accept the guid service name and zone name as request parameters 
> input and provide the get policy or delete policy option.
> API:
> a) getPolicyByGUIDAndServiceNameAndZoneName(guid, service, zone): reads the 
> input values and returns the policy object.
> b) deletePolicyByGUIDAndServiceNameAndZoneName(guid, service, zone) : reads 
> the input values and deletes the respective policy object.
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 
> cb57b9913 
>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyDao.java 
> 3558337a3 
>   security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java 
> 6ab3d52a0 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 
> 3ba29653b 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml a19f7f1d8 
> 
> 
> Diff: https://reviews.apache.org/r/73720/diff/2/
> 
> 
> Testing
> ---
> 
> Tested getPolicyByGUIDAndServiceNameAndZoneName() API and was able to recieve 
> the matching policy object.
> Tested deletePolicyByGUIDAndServiceNameAndZoneName() API and was able to 
> delete the respective policy object.
> 
> **Sample curl requests:**
> 
> curl -u admin:Ranger1234 -H "Accept: application/json" -H "Content-Type: 
> application/json" -X GET 
> 'http://localhost:6080/service/plugins/policies/guid/0be7457b-35c7-4ca9-bd08-938d98a3e724?serviceName=cm_hive'
> 
> curl -u admin:Ranger1234 -H "Accept: application/json" -H "Content-Type: 
> application/json" -X GET 
> 'http://localhost:6080/service/plugins/policies/guid/ad88dd6f-1d85-4a67-8e84-813809c83da0?serviceName=cm_hive&zoneName=zone1'
> 
> 
> curl -u admin:Ranger1234 -H "Accept: application/json" -H "Content-Type: 
> application/json" -X DELETE 
> 'http://localhost:6080/service/plugins/policies/guid/0be7457b-35c7-4ca9-bd08-938d98a3e724?serviceName=cm_hive'
> 
> curl -u admin:Ranger1234 -H "Accept: application/json" -H "Content-Type: 
> application/json" -X DELETE 
> 'http://localhost:6080/service/plugins/policies/guid/ad88dd6f-1d85-4a67-8e84-813809c83da0?serviceName=cm_hive&zoneName=zone1'
> 
> 
> Thanks,
> 
> Pradeep Agrawal
> 
>

[jira] [Commented] (RANGER-3439) Add rest api to get or delete ranger policy based on guid

2021-12-02 Thread Pradeep Agrawal (Jira) 


[ 
https://issues.apache.org/jira/browse/RANGER-3439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17452729#comment-17452729
 ] 

Pradeep Agrawal commented on RANGER-3439:
-

committed the updated patch : 
https://github.com/apache/ranger/commit/000e6351ee4628979a20e2b72ac6f226e6dd1c0e

> Add rest api to get or delete ranger policy based on guid
> -
>
> Key: RANGER-3439
> URL: https://issues.apache.org/jira/browse/RANGER-3439
> Project: Ranger
>  Issue Type: Sub-task
>  Components: Ranger
>Affects Versions: 3.0.0
>Reporter: Pradeep Agrawal
>Assignee: Pradeep Agrawal
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 
> 0001-RANGER-3439-REST-api-to-get-or-delete-ranger-policy-.patch, 
> 0002-RANGER-3439-Add-rest-api-to-get-or-delete-ranger-pol.patch
>
>
> Ranger should allow to get or delete ranger policy based on policy guid.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)