[jira] [Updated] (RANGER-1832) Export REST API should return exact matching results if polResource param is provided

[Pradeep Agrawal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pradeep Agrawal updated RANGER-1832:

Attachment: RANGER-1832-master.patch

> Export REST API should return exact matching results if polResource param is 
> provided
> -
>
> Key: RANGER-1832
> URL: https://issues.apache.org/jira/browse/RANGER-1832
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 0.7.0, 1.0.0, 0.7.1
>Reporter: Pradeep Agrawal
>Assignee: Pradeep Agrawal
> Fix For: 1.0.0, 0.7.2
>
> Attachments: RANGER-1832-master.patch
>
>
> Currently, Export REST API returns partial matching results also even if 
> polResource param is provided. 
> Use Case : 
> 1) Create a ranger hdfs policy with resource path /tmp/abcdefg
> 2) Call REST API to export policy for resource path: /tmp/abcd
> http://localhost:6080/service/plugins/policies/exportJson?serviceName=source_hadoop&polResource=/tmp/abcd&resource:path=/tmp/abcd&serviceType=hdfs&resourceMatchScope=self_or_ancestor";
> ER: REST call should not return policy for resource /tmp/abcdefg
> AR: REST call is returning policy for resource /tmp/abcdefg



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (RANGER-1836) LOG class is imported error at getTransactionLog() and processFieldToCreateTrxLog() method for RangerServiceService class

[Qiang Zhang (JIRA)]

Qiang Zhang created RANGER-1836:
---

 Summary: LOG class is imported error at getTransactionLog() and 
processFieldToCreateTrxLog() method for RangerServiceService class
 Key: RANGER-1836
 URL: https://issues.apache.org/jira/browse/RANGER-1836
 Project: Ranger
  Issue Type: Bug
  Components: Ranger
Affects Versions: master
Reporter: Qiang Zhang
Assignee: Qiang Zhang


LOG class is imported error for RangerServiceService class

RangerServiceService.java import:
import java.util.logging.Logger;

I think java.util.logging.Logger class should be repalced with:
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 62969: RANGER-1832: Export REST API should return exact matching results if polResource param is provided

[Pradeep Agrawal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62969/
---

Review request for ranger, Ankita Sinha, Don Bosco Durai, bhavik patel, Gautam 
Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Selvamohan 
Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.


Bugs: RANGER-1832
https://issues.apache.org/jira/browse/RANGER-1832


Repository: ranger


Description
---

**Problem Statement:** Currently, Export REST API returns partial matching 
results also even if polResource param is provided.
Use Case : 
1) Create a ranger hdfs policy with resource path /tmp/abcdefg
2) Call REST API to export policy for resource path: /tmp/abcd
http://localhost:6080/service/plugins/policies/exportJson?serviceName=source_hadoop&polResource=/tmp/abcd&resource:path=/tmp/abcd&serviceType=hdfs&resourceMatchScope=self_or_ancestor";

Expected Result: REST call should not return policy for resource /tmp/abcdefg
Actual Result: REST call is returning policy for resource /tmp/abcdefg

**Proposed Solution:**
Added a method in ServiceUtil.getMatchingPoliciesForResource() which shall 
filter out partial matching resource policies according to given resource. 

** Note :** Proposed solution is having support of only HDFS and Hive 
service/resource type; support for other services can be added later.


Diffs
-

  security-admin/src/main/java/org/apache/ranger/common/ServiceUtil.java 
6864c5a 
  security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 9330edd 


Diff: https://reviews.apache.org/r/62969/diff/1/


Testing
---

**Steps Performed (with patch):**
1. After mvn Build; untar the Ranger module and updated install.properties for 
MySQL DB flavor. 
2. Called setup.sh to execute Ranger setup script. 
3. Started Ranger admin.

**Steps for HDFS service:**
1. Created a hdfs service 'source_hadoop'.
2. Created a ranger hdfs policy(hdfs_policy1) with resource path /tmp/abcdefg
3. Created a ranger hdfs policy(hdfs_policy2) with resource path: /tmp/abcd
4. Created a ranger hdfs policy(hdfs_policy3) with resource path: 
/tmp/abcd/file.txt
5. Called below given REST API to export policies for resource path: /tmp/abcd
http://localhost:6080/service/plugins/policies/exportJson?serviceName=source_hadoop&polResource=/tmp/abcd&resource:path=/tmp/abcd&serviceType=hdfs&resourceMatchScope=self_or_ancestor";

**Expected Behavior:**
Above REST should return two policies(hdfs_policy2 and hdfs_policy3) which are 
having resource /tmp/abcd and /tmp/abcd/file.txt respectively.

**Actual Behavior:**
Returned JSON response was having only two policies(hdfs_policy2 and 
hdfs_policy3).

---

**Steps for HIVE service:**
1. Created a hive service 'source_hive'.
2. Created a ranger hive policy(hive_policy1) with 
resource:database=default123,table=*,column=*
3. Created a ranger hive policy(hive_policy2) with 
resource:database=default,table=*,column=*
4. Created a ranger hive datamask policy(hive_policy3) with 
resource:database=default,table=table1,column=column1
5. Created a ranger hive rowlevel filter policy(hive_policy4) with 
resource:database=default,table=table2
6. Called below given REST API to export policies for resource:database=default
http://localhost:6080/service/plugins/policies/exportJson?serviceName=source_hive&polResource=default&resource:database=default&serviceType=hive&resourceMatchScope=self_or_ancestor";

**Expected Behavior:**
Above REST should return three policies(hive_policy2, hive_policy3 and 
hive_policy4).

**Actual Behavior:**
Returned JSON response was having only three policies(hive_policy2, 
hive_policy3 and hive_policy4).


Thanks,

Pradeep Agrawal

[jira] [Updated] (RANGER-1836) LOG class is imported error for RangerServiceService class

[Qiang Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qiang Zhang updated RANGER-1836:

Summary: LOG class is imported error for RangerServiceService class  (was: 
LOG class is imported error at getTransactionLog() and 
processFieldToCreateTrxLog() method for RangerServiceService class)

> LOG class is imported error for RangerServiceService class
> --
>
> Key: RANGER-1836
> URL: https://issues.apache.org/jira/browse/RANGER-1836
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: master
>Reporter: Qiang Zhang
>Assignee: Qiang Zhang
>
> LOG class is imported error for RangerServiceService class
> RangerServiceService.java import:
> import java.util.logging.Logger;
> I think java.util.logging.Logger class should be repalced with:
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 62971: LOG class is imported error for RangerServiceService class

[Qiang Zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62971/
---

Review request for ranger, Alok Lal, Ankita Sinha, Don Bosco Durai, Colm O 
hEigeartaigh, Gautam Borad, Madhan Neethiraj, pengjianhua, Ramesh Mani, 
Selvamohan Neethiraj, Velmurugan Periasamy, and Qiang Zhang.


Bugs: RANGER-1836
https://issues.apache.org/jira/browse/RANGER-1836


Repository: ranger


Description
---

LOG class is imported error for RangerServiceService class
RangerServiceService.java import:
import java.util.logging.Logger;
I think java.util.logging.Logger class should be repalced with:
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;


Diffs
-

  
security-admin/src/main/java/org/apache/ranger/service/RangerServiceService.java
 3dd761a2 


Diff: https://reviews.apache.org/r/62971/diff/1/


Testing
---


Thanks,

Qiang Zhang

[jira] [Updated] (RANGER-1836) LOG class is imported error for RangerServiceService class

[Qiang Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qiang Zhang updated RANGER-1836:

Attachment: 0001-RANGER-1836-LOG-class-is-imported-error-for-RangerSe.patch

> LOG class is imported error for RangerServiceService class
> --
>
> Key: RANGER-1836
> URL: https://issues.apache.org/jira/browse/RANGER-1836
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: master
>Reporter: Qiang Zhang
>Assignee: Qiang Zhang
> Attachments: 
> 0001-RANGER-1836-LOG-class-is-imported-error-for-RangerSe.patch
>
>
> LOG class is imported error for RangerServiceService class
> RangerServiceService.java import:
> import java.util.logging.Logger;
> I think java.util.logging.Logger class should be repalced with:
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62969: RANGER-1832: Export REST API should return exact matching results if polResource param is provided

[Velmurugan Periasamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62969/#review187949
---


Ship it!




Ship It!

- Velmurugan Periasamy


On Oct. 13, 2017, 9:49 a.m., Pradeep Agrawal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62969/
> ---
> 
> (Updated Oct. 13, 2017, 9:49 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Don Bosco Durai, bhavik patel, 
> Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, 
> Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-1832
> https://issues.apache.org/jira/browse/RANGER-1832
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> **Problem Statement:** Currently, Export REST API returns partial matching 
> results also even if polResource param is provided.
> Use Case : 
> 1) Create a ranger hdfs policy with resource path /tmp/abcdefg
> 2) Call REST API to export policy for resource path: /tmp/abcd
> http://localhost:6080/service/plugins/policies/exportJson?serviceName=source_hadoop&polResource=/tmp/abcd&resource:path=/tmp/abcd&serviceType=hdfs&resourceMatchScope=self_or_ancestor";
> 
> Expected Result: REST call should not return policy for resource /tmp/abcdefg
> Actual Result: REST call is returning policy for resource /tmp/abcdefg
> 
> **Proposed Solution:**
> Added a method in ServiceUtil.getMatchingPoliciesForResource() which shall 
> filter out partial matching resource policies according to given resource. 
> 
> ** Note :** Proposed solution is having support of only HDFS and Hive 
> service/resource type; support for other services can be added later.
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/java/org/apache/ranger/common/ServiceUtil.java 
> 6864c5a 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 
> 9330edd 
> 
> 
> Diff: https://reviews.apache.org/r/62969/diff/1/
> 
> 
> Testing
> ---
> 
> **Steps Performed (with patch):**
> 1. After mvn Build; untar the Ranger module and updated install.properties 
> for MySQL DB flavor. 
> 2. Called setup.sh to execute Ranger setup script. 
> 3. Started Ranger admin.
> 
> **Steps for HDFS service:**
> 1. Created a hdfs service 'source_hadoop'.
> 2. Created a ranger hdfs policy(hdfs_policy1) with resource path /tmp/abcdefg
> 3. Created a ranger hdfs policy(hdfs_policy2) with resource path: /tmp/abcd
> 4. Created a ranger hdfs policy(hdfs_policy3) with resource path: 
> /tmp/abcd/file.txt
> 5. Called below given REST API to export policies for resource path: /tmp/abcd
> http://localhost:6080/service/plugins/policies/exportJson?serviceName=source_hadoop&polResource=/tmp/abcd&resource:path=/tmp/abcd&serviceType=hdfs&resourceMatchScope=self_or_ancestor";
> 
> **Expected Behavior:**
> Above REST should return two policies(hdfs_policy2 and hdfs_policy3) which 
> are having resource /tmp/abcd and /tmp/abcd/file.txt respectively.
> 
> **Actual Behavior:**
> Returned JSON response was having only two policies(hdfs_policy2 and 
> hdfs_policy3).
> 
> ---
> 
> **Steps for HIVE service:**
> 1. Created a hive service 'source_hive'.
> 2. Created a ranger hive policy(hive_policy1) with 
> resource:database=default123,table=*,column=*
> 3. Created a ranger hive policy(hive_policy2) with 
> resource:database=default,table=*,column=*
> 4. Created a ranger hive datamask policy(hive_policy3) with 
> resource:database=default,table=table1,column=column1
> 5. Created a ranger hive rowlevel filter policy(hive_policy4) with 
> resource:database=default,table=table2
> 6. Called below given REST API to export policies for 
> resource:database=default
> http://localhost:6080/service/plugins/policies/exportJson?serviceName=source_hive&polResource=default&resource:database=default&serviceType=hive&resourceMatchScope=self_or_ancestor";
> 
> **Expected Behavior:**
> Above REST should return three policies(hive_policy2, hive_policy3 and 
> hive_policy4).
> 
> **Actual Behavior:**
> Returned JSON response was having only three policies(hive_policy2, 
> hive_policy3 and hive_policy4).
> 
> 
> Thanks,
> 
> Pradeep Agrawal
> 
>

Re: Review Request 62969: RANGER-1832: Export REST API should return exact matching results if polResource param is provided

[Gautam Borad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62969/#review187950
---


Ship it!




Ship It!

- Gautam Borad


On Oct. 13, 2017, 9:49 a.m., Pradeep Agrawal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62969/
> ---
> 
> (Updated Oct. 13, 2017, 9:49 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Don Bosco Durai, bhavik patel, 
> Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, 
> Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-1832
> https://issues.apache.org/jira/browse/RANGER-1832
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> **Problem Statement:** Currently, Export REST API returns partial matching 
> results also even if polResource param is provided.
> Use Case : 
> 1) Create a ranger hdfs policy with resource path /tmp/abcdefg
> 2) Call REST API to export policy for resource path: /tmp/abcd
> http://localhost:6080/service/plugins/policies/exportJson?serviceName=source_hadoop&polResource=/tmp/abcd&resource:path=/tmp/abcd&serviceType=hdfs&resourceMatchScope=self_or_ancestor";
> 
> Expected Result: REST call should not return policy for resource /tmp/abcdefg
> Actual Result: REST call is returning policy for resource /tmp/abcdefg
> 
> **Proposed Solution:**
> Added a method in ServiceUtil.getMatchingPoliciesForResource() which shall 
> filter out partial matching resource policies according to given resource. 
> 
> ** Note :** Proposed solution is having support of only HDFS and Hive 
> service/resource type; support for other services can be added later.
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/java/org/apache/ranger/common/ServiceUtil.java 
> 6864c5a 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 
> 9330edd 
> 
> 
> Diff: https://reviews.apache.org/r/62969/diff/1/
> 
> 
> Testing
> ---
> 
> **Steps Performed (with patch):**
> 1. After mvn Build; untar the Ranger module and updated install.properties 
> for MySQL DB flavor. 
> 2. Called setup.sh to execute Ranger setup script. 
> 3. Started Ranger admin.
> 
> **Steps for HDFS service:**
> 1. Created a hdfs service 'source_hadoop'.
> 2. Created a ranger hdfs policy(hdfs_policy1) with resource path /tmp/abcdefg
> 3. Created a ranger hdfs policy(hdfs_policy2) with resource path: /tmp/abcd
> 4. Created a ranger hdfs policy(hdfs_policy3) with resource path: 
> /tmp/abcd/file.txt
> 5. Called below given REST API to export policies for resource path: /tmp/abcd
> http://localhost:6080/service/plugins/policies/exportJson?serviceName=source_hadoop&polResource=/tmp/abcd&resource:path=/tmp/abcd&serviceType=hdfs&resourceMatchScope=self_or_ancestor";
> 
> **Expected Behavior:**
> Above REST should return two policies(hdfs_policy2 and hdfs_policy3) which 
> are having resource /tmp/abcd and /tmp/abcd/file.txt respectively.
> 
> **Actual Behavior:**
> Returned JSON response was having only two policies(hdfs_policy2 and 
> hdfs_policy3).
> 
> ---
> 
> **Steps for HIVE service:**
> 1. Created a hive service 'source_hive'.
> 2. Created a ranger hive policy(hive_policy1) with 
> resource:database=default123,table=*,column=*
> 3. Created a ranger hive policy(hive_policy2) with 
> resource:database=default,table=*,column=*
> 4. Created a ranger hive datamask policy(hive_policy3) with 
> resource:database=default,table=table1,column=column1
> 5. Created a ranger hive rowlevel filter policy(hive_policy4) with 
> resource:database=default,table=table2
> 6. Called below given REST API to export policies for 
> resource:database=default
> http://localhost:6080/service/plugins/policies/exportJson?serviceName=source_hive&polResource=default&resource:database=default&serviceType=hive&resourceMatchScope=self_or_ancestor";
> 
> **Expected Behavior:**
> Above REST should return three policies(hive_policy2, hive_policy3 and 
> hive_policy4).
> 
> **Actual Behavior:**
> Returned JSON response was having only three policies(hive_policy2, 
> hive_policy3 and hive_policy4).
> 
> 
> Thanks,
> 
> Pradeep Agrawal
> 
>

[jira] [Created] (RANGER-1837) HDFS Audit Compression

[Kevin Risden (JIRA)]

Kevin Risden created RANGER-1837:


 Summary: HDFS Audit Compression
 Key: RANGER-1837
 URL: https://issues.apache.org/jira/browse/RANGER-1837
 Project: Ranger
  Issue Type: Improvement
  Components: audit
Reporter: Kevin Risden


My team has done some research and found that Ranger HDFS audits are:
* Stored as JSON objects (one per line)
* Not compressed

This is currently very verbose and would benefit from compression since this 
data is not frequently accessed. 

>From Bosco on the mailing list:
{quote}You are right, currently one of the options is saving the audits in HDFS 
itself as JSON files in one folder per day. I have loaded these JSON files from 
the folder into Hive as compressed ORC format. The compressed files in ORC were 
less than 10% of the original size. So, it was significant decrease in size. 
Also, it is easier to run analytics on the Hive tables.
 
So, there are couple of ways of doing it.
 
Write an Oozie job which runs every night and loads the previous day worth 
audit logs into ORC or other format
Write a AuditDestination which can write into the format you want to.
 
Regardless which approach you take, this would be a good feature for 
Ranger.{quote}

http://mail-archives.apache.org/mod_mbox/ranger-user/201710.mbox/%3CCAJU9nmiYzzUUX1uDEysLAcMti4iLmX7RE%3DmN2%3DdoLaaQf87njQ%40mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1837) HDFS Audit Compression

[Kevin Risden (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203539#comment-16203539
 ] 

Kevin Risden commented on RANGER-1837:
--

Also from Bosco on the mailing list:
{quote}
f we write as ORC or other file format directly, then we have to see how to 
batch the audits. In the Audit V3 implementation, we did some optimization to 
avoid store (local write) and forward, instead build the batch in the memory 
itself and do bulk write (each Destination has different policies). But in the 
previous release, we did re-introduce an option to store and forward to HDFS 
due to HDFS file closure issue.
 
I personally don’t know what would be a good batch size. But we can build on 
top that code to write in the format we want to. And make the output write 
configurable to support different types.
{quote}

>From Ramesh on the mailing list:
{quote}
+1 for your suggestion on having a Audit FileFormat as a feature in the Ranger 
Audit Framework.  

In that case HDFSAuditDestination should have the provision to use a FileFormat 
before writing, where as SolrDestination might not require this.  

Each configured AuditDestination can have a Format conversion before writing, 
we don’t need to have this format all the way from Audit generation point.
{quote}

> HDFS Audit Compression
> --
>
> Key: RANGER-1837
> URL: https://issues.apache.org/jira/browse/RANGER-1837
> Project: Ranger
>  Issue Type: Improvement
>  Components: audit
>Reporter: Kevin Risden
>
> My team has done some research and found that Ranger HDFS audits are:
> * Stored as JSON objects (one per line)
> * Not compressed
> This is currently very verbose and would benefit from compression since this 
> data is not frequently accessed. 
> From Bosco on the mailing list:
> {quote}You are right, currently one of the options is saving the audits in 
> HDFS itself as JSON files in one folder per day. I have loaded these JSON 
> files from the folder into Hive as compressed ORC format. The compressed 
> files in ORC were less than 10% of the original size. So, it was significant 
> decrease in size. Also, it is easier to run analytics on the Hive tables.
>  
> So, there are couple of ways of doing it.
>  
> Write an Oozie job which runs every night and loads the previous day worth 
> audit logs into ORC or other format
> Write a AuditDestination which can write into the format you want to.
>  
> Regardless which approach you take, this would be a good feature for 
> Ranger.{quote}
> http://mail-archives.apache.org/mod_mbox/ranger-user/201710.mbox/%3CCAJU9nmiYzzUUX1uDEysLAcMti4iLmX7RE%3DmN2%3DdoLaaQf87njQ%40mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (RANGER-1837) HDFS Audit Compression

[Kevin Risden (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203539#comment-16203539
 ] 

Kevin Risden edited comment on RANGER-1837 at 10/13/17 1:09 PM:


Also from Bosco on the mailing list:
{quote}
If we write as ORC or other file format directly, then we have to see how to 
batch the audits. In the Audit V3 implementation, we did some optimization to 
avoid store (local write) and forward, instead build the batch in the memory 
itself and do bulk write (each Destination has different policies). But in the 
previous release, we did re-introduce an option to store and forward to HDFS 
due to HDFS file closure issue.
 
I personally don’t know what would be a good batch size. But we can build on 
top that code to write in the format we want to. And make the output write 
configurable to support different types.
{quote}

>From Ramesh on the mailing list:
{quote}
+1 for your suggestion on having a Audit FileFormat as a feature in the Ranger 
Audit Framework.  

In that case HDFSAuditDestination should have the provision to use a FileFormat 
before writing, where as SolrDestination might not require this.  

Each configured AuditDestination can have a Format conversion before writing, 
we don’t need to have this format all the way from Audit generation point.
{quote}


was (Author: risdenk):
Also from Bosco on the mailing list:
{quote}
f we write as ORC or other file format directly, then we have to see how to 
batch the audits. In the Audit V3 implementation, we did some optimization to 
avoid store (local write) and forward, instead build the batch in the memory 
itself and do bulk write (each Destination has different policies). But in the 
previous release, we did re-introduce an option to store and forward to HDFS 
due to HDFS file closure issue.
 
I personally don’t know what would be a good batch size. But we can build on 
top that code to write in the format we want to. And make the output write 
configurable to support different types.
{quote}

>From Ramesh on the mailing list:
{quote}
+1 for your suggestion on having a Audit FileFormat as a feature in the Ranger 
Audit Framework.  

In that case HDFSAuditDestination should have the provision to use a FileFormat 
before writing, where as SolrDestination might not require this.  

Each configured AuditDestination can have a Format conversion before writing, 
we don’t need to have this format all the way from Audit generation point.
{quote}

> HDFS Audit Compression
> --
>
> Key: RANGER-1837
> URL: https://issues.apache.org/jira/browse/RANGER-1837
> Project: Ranger
>  Issue Type: Improvement
>  Components: audit
>Reporter: Kevin Risden
>
> My team has done some research and found that Ranger HDFS audits are:
> * Stored as JSON objects (one per line)
> * Not compressed
> This is currently very verbose and would benefit from compression since this 
> data is not frequently accessed. 
> From Bosco on the mailing list:
> {quote}You are right, currently one of the options is saving the audits in 
> HDFS itself as JSON files in one folder per day. I have loaded these JSON 
> files from the folder into Hive as compressed ORC format. The compressed 
> files in ORC were less than 10% of the original size. So, it was significant 
> decrease in size. Also, it is easier to run analytics on the Hive tables.
>  
> So, there are couple of ways of doing it.
>  
> Write an Oozie job which runs every night and loads the previous day worth 
> audit logs into ORC or other format
> Write a AuditDestination which can write into the format you want to.
>  
> Regardless which approach you take, this would be a good feature for 
> Ranger.{quote}
> http://mail-archives.apache.org/mod_mbox/ranger-user/201710.mbox/%3CCAJU9nmiYzzUUX1uDEysLAcMti4iLmX7RE%3DmN2%3DdoLaaQf87njQ%40mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (RANGER-1837) HDFS Audit Compression

[Kevin Risden (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203539#comment-16203539
 ] 

Kevin Risden edited comment on RANGER-1837 at 10/13/17 1:16 PM:


>From Sean on the mailing list:
{quote}
I’ve been looking at the same. Even in small clusters the size of Ranger Audits 
is considerable. The files compress well. But compressed JSON will be difficult 
to query.
 
Would engineering Ranger to write directly to ORC be reasonable?
{quote}

Also from Bosco on the mailing list:
{quote}
If we write as ORC or other file format directly, then we have to see how to 
batch the audits. In the Audit V3 implementation, we did some optimization to 
avoid store (local write) and forward, instead build the batch in the memory 
itself and do bulk write (each Destination has different policies). But in the 
previous release, we did re-introduce an option to store and forward to HDFS 
due to HDFS file closure issue.
 
I personally don’t know what would be a good batch size. But we can build on 
top that code to write in the format we want to. And make the output write 
configurable to support different types.
{quote}

>From Ramesh on the mailing list:
{quote}
+1 for your suggestion on having a Audit FileFormat as a feature in the Ranger 
Audit Framework.  

In that case HDFSAuditDestination should have the provision to use a FileFormat 
before writing, where as SolrDestination might not require this.  

Each configured AuditDestination can have a Format conversion before writing, 
we don’t need to have this format all the way from Audit generation point.
{quote}


was (Author: risdenk):
Also from Bosco on the mailing list:
{quote}
If we write as ORC or other file format directly, then we have to see how to 
batch the audits. In the Audit V3 implementation, we did some optimization to 
avoid store (local write) and forward, instead build the batch in the memory 
itself and do bulk write (each Destination has different policies). But in the 
previous release, we did re-introduce an option to store and forward to HDFS 
due to HDFS file closure issue.
 
I personally don’t know what would be a good batch size. But we can build on 
top that code to write in the format we want to. And make the output write 
configurable to support different types.
{quote}

>From Ramesh on the mailing list:
{quote}
+1 for your suggestion on having a Audit FileFormat as a feature in the Ranger 
Audit Framework.  

In that case HDFSAuditDestination should have the provision to use a FileFormat 
before writing, where as SolrDestination might not require this.  

Each configured AuditDestination can have a Format conversion before writing, 
we don’t need to have this format all the way from Audit generation point.
{quote}

> HDFS Audit Compression
> --
>
> Key: RANGER-1837
> URL: https://issues.apache.org/jira/browse/RANGER-1837
> Project: Ranger
>  Issue Type: Improvement
>  Components: audit
>Reporter: Kevin Risden
>
> My team has done some research and found that Ranger HDFS audits are:
> * Stored as JSON objects (one per line)
> * Not compressed
> This is currently very verbose and would benefit from compression since this 
> data is not frequently accessed. 
> From Bosco on the mailing list:
> {quote}You are right, currently one of the options is saving the audits in 
> HDFS itself as JSON files in one folder per day. I have loaded these JSON 
> files from the folder into Hive as compressed ORC format. The compressed 
> files in ORC were less than 10% of the original size. So, it was significant 
> decrease in size. Also, it is easier to run analytics on the Hive tables.
>  
> So, there are couple of ways of doing it.
>  
> Write an Oozie job which runs every night and loads the previous day worth 
> audit logs into ORC or other format
> Write a AuditDestination which can write into the format you want to.
>  
> Regardless which approach you take, this would be a good feature for 
> Ranger.{quote}
> http://mail-archives.apache.org/mod_mbox/ranger-user/201710.mbox/%3CCAJU9nmiYzzUUX1uDEysLAcMti4iLmX7RE%3DmN2%3DdoLaaQf87njQ%40mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1832) Export REST API should return exact matching results if polResource param is provided

[Pradeep Agrawal (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203656#comment-16203656
 ] 

Pradeep Agrawal commented on RANGER-1832:
-

Patch committed in master branch: 
https://github.com/apache/ranger/commit/478f5ffc737b4c238db67e844e88fd73c61dc5da
Patch committed in ranger-0.7 branch : 
https://github.com/apache/ranger/commit/c2e1ec9c79e47491c465cb4967aa5b08fbcda360

> Export REST API should return exact matching results if polResource param is 
> provided
> -
>
> Key: RANGER-1832
> URL: https://issues.apache.org/jira/browse/RANGER-1832
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 0.7.0, 1.0.0, 0.7.1
>Reporter: Pradeep Agrawal
>Assignee: Pradeep Agrawal
> Fix For: 1.0.0, 0.7.2
>
> Attachments: RANGER-1832-master.patch
>
>
> Currently, Export REST API returns partial matching results also even if 
> polResource param is provided. 
> Use Case : 
> 1) Create a ranger hdfs policy with resource path /tmp/abcdefg
> 2) Call REST API to export policy for resource path: /tmp/abcd
> http://localhost:6080/service/plugins/policies/exportJson?serviceName=source_hadoop&polResource=/tmp/abcd&resource:path=/tmp/abcd&serviceType=hdfs&resourceMatchScope=self_or_ancestor";
> ER: REST call should not return policy for resource /tmp/abcdefg
> AR: REST call is returning policy for resource /tmp/abcdefg



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1707) Traverse check in RangerHdfsAuthorizer works incorrectly

[Zsombor Gegesy (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203795#comment-16203795
 ] 

Zsombor Gegesy commented on RANGER-1707:


Running the tests with 3.0.0-beta1 show the same, changed behaviour, which is 
fixed by this patch. However, there will be other problems preventing a 
flawless upgrade to 3.0.
I've noticed that this needs hbase 2.0, which has breaking changes for Ranger, 
and the KMS part is too in a bad shape. 

> Traverse check in RangerHdfsAuthorizer works incorrectly
> 
>
> Key: RANGER-1707
> URL: https://issues.apache.org/jira/browse/RANGER-1707
> Project: Ranger
>  Issue Type: Bug
>  Components: plugins
>Affects Versions: 1.0.0
>Reporter: Zsombor Gegesy
>Assignee: Zsombor Gegesy
>  Labels: hdfs-2.8
> Fix For: 1.0.0
>
> Attachments: 
> 0001-RANGER-1707-Fix-hdfs-traverse-check-which-problem-wa.patch
>
>
> Traversal check in RangerHdfsAuthorizer works incorrectly, when it is asked 
> for access to /a/b/c.txt, it only checks that if there are a policy which 
> grants EXEC to /a/b, but if it there aren't any, then it doesn't check, if 
> there is a policy which grants READ, WRITE or EXEC to /a/b/c.txt explicitly, 
> which would mean, that the path is accessible to the user.
>  This hasn't noticed by the current unit tests, because HDFS before 2.8.0 
> doesn't called the traversal check before reading or writing a file, however 
> it will cause problem with 2.8.0, where FSDirectory.resolvePath will perform 
> a mandatory traversal check.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62967: RANGER-1835:The installer of the security admin should not repeatedly add a user to the same group.

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62967/#review187986
---


Ship it!




Ship It!

- Alejandro Fernandez


On Oct. 13, 2017, 3:22 a.m., pengjianhua wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62967/
> ---
> 
> (Updated Oct. 13, 2017, 3:22 a.m.)
> 
> 
> Review request for ranger, Alok Lal, Ankita Sinha, Don Bosco Durai, Colm O 
> hEigeartaigh, Gautam Borad, Madhan Neethiraj, Ramesh Mani, Selvamohan 
> Neethiraj, Velmurugan Periasamy, and Qiang Zhang.
> 
> 
> Bugs: RANGER-1835
> https://issues.apache.org/jira/browse/RANGER-1835
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> The installer of the security admin will repeatedly add a user to the same 
> group if the user exists and the user belongs to the group. The installer 
> should check whether the user belongs to the group before adding user to 
> group.
> 
> 
> Diffs
> -
> 
>   security-admin/scripts/setup.sh 1630684c 
> 
> 
> Diff: https://reviews.apache.org/r/62967/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengjianhua
> 
>

Re: Review Request 62971: LOG class is imported error for RangerServiceService class

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62971/#review187987
---



What testing was done?

- Alejandro Fernandez


On Oct. 13, 2017, 10 a.m., Qiang Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62971/
> ---
> 
> (Updated Oct. 13, 2017, 10 a.m.)
> 
> 
> Review request for ranger, Alok Lal, Ankita Sinha, Don Bosco Durai, Colm O 
> hEigeartaigh, Gautam Borad, Madhan Neethiraj, pengjianhua, Ramesh Mani, 
> Selvamohan Neethiraj, Velmurugan Periasamy, and Qiang Zhang.
> 
> 
> Bugs: RANGER-1836
> https://issues.apache.org/jira/browse/RANGER-1836
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> LOG class is imported error for RangerServiceService class
> RangerServiceService.java import:
> import java.util.logging.Logger;
> I think java.util.logging.Logger class should be repalced with:
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> 
> 
> Diffs
> -
> 
>   
> security-admin/src/main/java/org/apache/ranger/service/RangerServiceService.java
>  3dd761a2 
> 
> 
> Diff: https://reviews.apache.org/r/62971/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Qiang Zhang
> 
>

Re: Review Request 62971: LOG class is imported error for RangerServiceService class

[Ramesh Mani]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62971/#review188005
---


Ship it!




Ship It!

- Ramesh Mani


On Oct. 13, 2017, 10 a.m., Qiang Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62971/
> ---
> 
> (Updated Oct. 13, 2017, 10 a.m.)
> 
> 
> Review request for ranger, Alok Lal, Ankita Sinha, Don Bosco Durai, Colm O 
> hEigeartaigh, Gautam Borad, Madhan Neethiraj, pengjianhua, Ramesh Mani, 
> Selvamohan Neethiraj, Velmurugan Periasamy, and Qiang Zhang.
> 
> 
> Bugs: RANGER-1836
> https://issues.apache.org/jira/browse/RANGER-1836
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> LOG class is imported error for RangerServiceService class
> RangerServiceService.java import:
> import java.util.logging.Logger;
> I think java.util.logging.Logger class should be repalced with:
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> 
> 
> Diffs
> -
> 
>   
> security-admin/src/main/java/org/apache/ranger/service/RangerServiceService.java
>  3dd761a2 
> 
> 
> Diff: https://reviews.apache.org/r/62971/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Qiang Zhang
> 
>

[jira] [Updated] (RANGER-1834) row filter policies are not returned by policy search

[Madhan Neethiraj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated RANGER-1834:
-
Attachment: RANGER-1834.patch

Patch from [~abhayk] is attached.

> row filter policies are not returned by policy search
> -
>
> Key: RANGER-1834
> URL: https://issues.apache.org/jira/browse/RANGER-1834
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: master, 0.7.1
>Reporter: Abhay Kulkarni
>Assignee: Abhay Kulkarni
> Fix For: master
>
> Attachments: RANGER-1834.patch
>
>
> ServiceREST APIs for searching policies filtered by resource and 
> resourceMatchScope do not return row-filter policies.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1834) row filter policies are not returned by policy search

[Madhan Neethiraj (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204165#comment-16204165
 ] 

Madhan Neethiraj commented on RANGER-1834:
--

+1 for the patch. Thanks [~abhayk].

> row filter policies are not returned by policy search
> -
>
> Key: RANGER-1834
> URL: https://issues.apache.org/jira/browse/RANGER-1834
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: master, 0.7.1
>Reporter: Abhay Kulkarni
>Assignee: Abhay Kulkarni
> Fix For: 0.7.2
>
> Attachments: RANGER-1834.patch
>
>
> ServiceREST APIs for searching policies filtered by resource and 
> resourceMatchScope do not return row-filter policies.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62971: LOG class is imported error for RangerServiceService class

[pengjianhua]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62971/#review188054
---


Ship it!




Ship It!

- pengjianhua


On 十月 13, 2017, 10 a.m., Qiang Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62971/
> ---
> 
> (Updated 十月 13, 2017, 10 a.m.)
> 
> 
> Review request for ranger, Alok Lal, Ankita Sinha, Don Bosco Durai, Colm O 
> hEigeartaigh, Gautam Borad, Madhan Neethiraj, pengjianhua, Ramesh Mani, 
> Selvamohan Neethiraj, Velmurugan Periasamy, and Qiang Zhang.
> 
> 
> Bugs: RANGER-1836
> https://issues.apache.org/jira/browse/RANGER-1836
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> LOG class is imported error for RangerServiceService class
> RangerServiceService.java import:
> import java.util.logging.Logger;
> I think java.util.logging.Logger class should be repalced with:
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> 
> 
> Diffs
> -
> 
>   
> security-admin/src/main/java/org/apache/ranger/service/RangerServiceService.java
>  3dd761a2 
> 
> 
> Diff: https://reviews.apache.org/r/62971/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Qiang Zhang
> 
>

[jira] [Resolved] (RANGER-1836) LOG class is imported error for RangerServiceService class

[peng.jianhua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

peng.jianhua resolved RANGER-1836.
--
   Resolution: Fixed
Fix Version/s: master
   1.0.0

> LOG class is imported error for RangerServiceService class
> --
>
> Key: RANGER-1836
> URL: https://issues.apache.org/jira/browse/RANGER-1836
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: master
>Reporter: Qiang Zhang
>Assignee: Qiang Zhang
> Fix For: 1.0.0, master
>
> Attachments: 
> 0001-RANGER-1836-LOG-class-is-imported-error-for-RangerSe.patch
>
>
> LOG class is imported error for RangerServiceService class
> RangerServiceService.java import:
> import java.util.logging.Logger;
> I think java.util.logging.Logger class should be repalced with:
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62918: RangerPolicy should use equals() to check equal for object of resources/policyItems/denyPolicyItems/allowExceptions/denyExceptions/dataMaskPolicyItems/rowFilterPolicyItems

[Qiang Zhang]

> On Oct. 12, 2017, 7:43 p.m., Abhay Kulkarni wrote:
> > agents-common/src/main/java/org/apache/ranger/plugin/model/RangerPolicy.java
> > Line 238 (original), 238 (patched)
> > 
> >
> > I don't think this JIRA is necessary, and if committed, will cause 
> > performance degradation.
> > 
> > RangerPolicy (and other model objects in Ranger) serve as pure data 
> > objects; they are repository of related data and provide getter/setter 
> > methods to access/manipulate these data. The '==' is deliberately used 
> > here, because only pointer comparison is desired (and not object 
> > comparison), just to ensure that data being held and data being supplied is 
> > not exactly the same object. There is no need here for deep, object level 
> > equality check.
> > 
> > This comment applies to all such changes in this patch.
> > 
> > Thanks!

Ok. I close the issue. Thanks.


- Qiang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62918/#review187854
---


On Oct. 12, 2017, 2:25 a.m., Qiang Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62918/
> ---
> 
> (Updated Oct. 12, 2017, 2:25 a.m.)
> 
> 
> Review request for ranger, Alok Lal, Ankita Sinha, Don Bosco Durai, Colm O 
> hEigeartaigh, Gautam Borad, Madhan Neethiraj, Ramesh Mani, Selvamohan 
> Neethiraj, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-1829
> https://issues.apache.org/jira/browse/RANGER-1829
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> When RangerPolicy check the equal for object of 
> resources/policyItems/denyPolicyItems/allowExceptions/denyExceptions/dataMaskPolicyItems/rowFilterPolicyItems,
>  it uses the "==" operator.
> But it should not use "==", use "equals()" is correctly.
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/model/RangerPolicy.java 
> 60daed9 
> 
> 
> Diff: https://reviews.apache.org/r/62918/diff/1/
> 
> 
> Testing
> ---
> 
> tested it!
> 
> 
> Thanks,
> 
> Qiang Zhang
> 
>

Re: Review Request 62967: RANGER-1835:The installer of the security admin should not repeatedly add a user to the same group.

[Qiang Zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62967/#review188056
---


Ship it!




Ship It!

- Qiang Zhang


On Oct. 13, 2017, 3:22 a.m., pengjianhua wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62967/
> ---
> 
> (Updated Oct. 13, 2017, 3:22 a.m.)
> 
> 
> Review request for ranger, Alok Lal, Ankita Sinha, Don Bosco Durai, Colm O 
> hEigeartaigh, Gautam Borad, Madhan Neethiraj, Ramesh Mani, Selvamohan 
> Neethiraj, Velmurugan Periasamy, and Qiang Zhang.
> 
> 
> Bugs: RANGER-1835
> https://issues.apache.org/jira/browse/RANGER-1835
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> The installer of the security admin will repeatedly add a user to the same 
> group if the user exists and the user belongs to the group. The installer 
> should check whether the user belongs to the group before adding user to 
> group.
> 
> 
> Diffs
> -
> 
>   security-admin/scripts/setup.sh 1630684c 
> 
> 
> Diff: https://reviews.apache.org/r/62967/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengjianhua
> 
>