[jira] [Assigned] (RANGER-2112) Ranger KMS broken with JDK 8 update 171

2018-07-05 Thread Pradeep Agrawal (JIRA) 


 [ 
https://issues.apache.org/jira/browse/RANGER-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pradeep Agrawal reassigned RANGER-2112:
---

Assignee: Pradeep Agrawal

> Ranger KMS broken with JDK 8 update 171
> ---
>
> Key: RANGER-2112
> URL: https://issues.apache.org/jira/browse/RANGER-2112
> Project: Ranger
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 0.7.0
>Reporter: Hernan Fernandez
>Assignee: Pradeep Agrawal
>Priority: Major
>
> After update to JDK 8 update 171 Ranger KMS UI
> 1) Ranger KMS UI > Encryption: will show the key list as the following.
> keyname (empty)
> Cipher (empty)
> Version 0
> Attributes (empty)
> Create (empty)
>  
> !image-2018-05-22-10-19-13-599.png!
>  
> 2) hadoop key -list -metadata
> Listing keys for KeyProvider: 
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@7d322cad
> testkey1 : null 
>  
>  *ROOT CAUSE*
>  This may be related to
> {code:java}
> New Features 
> security-libs/javax.crypto  
> Enhanced KeyStore Mechanisms
> A new security property named jceks.key.serialFilter has been introduced. If 
> this filter is configured, the JCEKS KeyStore uses it during the 
> deserialization of the encrypted Key object stored inside a SecretKeyEntry. 
> If it is not configured or if the filter result is UNDECIDED (for example, 
> none of the patterns match), then the filter configured by jdk.serialFilter 
> is consulted. If the system property jceks.key.serialFilter is also supplied, 
> it supersedes the security property value defined here. The filter pattern 
> uses the same format as jdk.serialFilter. The default pattern allows 
> java.lang.Enum, java.security.KeyRep, java.security.KeyRep$Type, and 
> javax.crypto.spec.SecretKeySpec but rejects all the others. Customers storing 
> a SecretKey that does not serialize to the above types must modify the filter 
> to make the key extractable.
> {code}
> http://www.oracle.com/technetwork/java/javase/8u171-relnotes-430.html
>  b) second option this is related to 3DES disabled on java.security (to be 
> tested)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (RANGER-2112) Ranger KMS broken with JDK 8 update 171

2018-07-05 Thread Pradeep Agrawal (JIRA) 


[ 
https://issues.apache.org/jira/browse/RANGER-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534474#comment-16534474
 ] 

Pradeep Agrawal edited comment on RANGER-2112 at 7/6/18 6:58 AM:
-

Looks like a bug reported here : [https://github.com/jcryptool/core/issues/120]

Seems its fixed in Hadoop KMS : 
https://issues.apache.org/jira/browse/HADOOP-15473


was (Author: pradeep.agrawal):
Looks like a bug reported here : https://github.com/jcryptool/core/issues/120

> Ranger KMS broken with JDK 8 update 171
> ---
>
> Key: RANGER-2112
> URL: https://issues.apache.org/jira/browse/RANGER-2112
> Project: Ranger
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 0.7.0
>Reporter: Hernan Fernandez
>Priority: Major
>
> After update to JDK 8 update 171 Ranger KMS UI
> 1) Ranger KMS UI > Encryption: will show the key list as the following.
> keyname (empty)
> Cipher (empty)
> Version 0
> Attributes (empty)
> Create (empty)
>  
> !image-2018-05-22-10-19-13-599.png!
>  
> 2) hadoop key -list -metadata
> Listing keys for KeyProvider: 
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@7d322cad
> testkey1 : null 
>  
>  *ROOT CAUSE*
>  This may be related to
> {code:java}
> New Features 
> security-libs/javax.crypto  
> Enhanced KeyStore Mechanisms
> A new security property named jceks.key.serialFilter has been introduced. If 
> this filter is configured, the JCEKS KeyStore uses it during the 
> deserialization of the encrypted Key object stored inside a SecretKeyEntry. 
> If it is not configured or if the filter result is UNDECIDED (for example, 
> none of the patterns match), then the filter configured by jdk.serialFilter 
> is consulted. If the system property jceks.key.serialFilter is also supplied, 
> it supersedes the security property value defined here. The filter pattern 
> uses the same format as jdk.serialFilter. The default pattern allows 
> java.lang.Enum, java.security.KeyRep, java.security.KeyRep$Type, and 
> javax.crypto.spec.SecretKeySpec but rejects all the others. Customers storing 
> a SecretKey that does not serialize to the above types must modify the filter 
> to make the key extractable.
> {code}
> http://www.oracle.com/technetwork/java/javase/8u171-relnotes-430.html
>  b) second option this is related to 3DES disabled on java.security (to be 
> tested)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (RANGER-2112) Ranger KMS broken with JDK 8 update 171

2018-07-05 Thread Pradeep Agrawal (JIRA) 


[ 
https://issues.apache.org/jira/browse/RANGER-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534474#comment-16534474
 ] 

Pradeep Agrawal commented on RANGER-2112:
-

Looks like a bug reported here : https://github.com/jcryptool/core/issues/120

> Ranger KMS broken with JDK 8 update 171
> ---
>
> Key: RANGER-2112
> URL: https://issues.apache.org/jira/browse/RANGER-2112
> Project: Ranger
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 0.7.0
>Reporter: Hernan Fernandez
>Priority: Major
>
> After update to JDK 8 update 171 Ranger KMS UI
> 1) Ranger KMS UI > Encryption: will show the key list as the following.
> keyname (empty)
> Cipher (empty)
> Version 0
> Attributes (empty)
> Create (empty)
>  
> !image-2018-05-22-10-19-13-599.png!
>  
> 2) hadoop key -list -metadata
> Listing keys for KeyProvider: 
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@7d322cad
> testkey1 : null 
>  
>  *ROOT CAUSE*
>  This may be related to
> {code:java}
> New Features 
> security-libs/javax.crypto  
> Enhanced KeyStore Mechanisms
> A new security property named jceks.key.serialFilter has been introduced. If 
> this filter is configured, the JCEKS KeyStore uses it during the 
> deserialization of the encrypted Key object stored inside a SecretKeyEntry. 
> If it is not configured or if the filter result is UNDECIDED (for example, 
> none of the patterns match), then the filter configured by jdk.serialFilter 
> is consulted. If the system property jceks.key.serialFilter is also supplied, 
> it supersedes the security property value defined here. The filter pattern 
> uses the same format as jdk.serialFilter. The default pattern allows 
> java.lang.Enum, java.security.KeyRep, java.security.KeyRep$Type, and 
> javax.crypto.spec.SecretKeySpec but rejects all the others. Customers storing 
> a SecretKey that does not serialize to the above types must modify the filter 
> to make the key extractable.
> {code}
> http://www.oracle.com/technetwork/java/javase/8u171-relnotes-430.html
>  b) second option this is related to 3DES disabled on java.security (to be 
> tested)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin

2018-07-05 Thread Kent Yao (JIRA) 


[ 
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534375#comment-16534375
 ] 

Kent Yao commented on RANGER-2128:
--

bq. I was able to use the latest Ranger Hive Plugin with Spark

That' s great! 

bq. I have to do a bit of hack to use Jersey2

While organizing all related jars in $SPARK_HOME/jars/ranger-hive-plugin-impl/ 
and using shim and classloader, it seems to have no problems of jar 
conflicting. Did you add all jars to spark's classloader?


bq. If you know to get that somehow, then our implementation of the Spark 
plugin will be much simpler and not dependent on the HiveAuthorizer.

My apology for not covering Spark‘s Own Thrift Server. Let me give a try 
please. 


> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Assignee: Kent Yao
>Priority: Major
> Fix For: 2.0.0
>
>
> Implement SparkSQL plugin



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin

2018-07-05 Thread Don Bosco Durai (JIRA) 


[ 
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534345#comment-16534345
 ] 

Don Bosco Durai commented on RANGER-2128:
-

Hi [~Qin Yao], I was able to use the latest Ranger Hive Plugin with Spark. I 
have to do a bit of hack to use Jersey2 and also comment out one line in the 
Hive plugin code.

Next, I am trying to see if I can integrate which SparkSQL Thrift server. I 
will update this JIRA after that. Hopefully by tomorrow.

I also wrote a standalone Spark plugin and updated your code to call it. 
However, when I ran it in SparkSQL Thrift server, it was called, but I was not 
able to get the effective user (when doAs is false). I traced the code and it 
seems the Hive SessionState that is created has the effective user, however, I 
am not able to get that outside of it. If you know to get that somehow, then 
our implementation of the Spark plugin will be much simpler and not dependent 
on the HiveAuthorizer.


> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Assignee: Kent Yao
>Priority: Major
> Fix For: 2.0.0
>
>
> Implement SparkSQL plugin



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67835: RANGER-2150: Unit test coverage for XUserMgr and UserMgr class

2018-07-05 Thread Zsombor Gegesy 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67835/#review205764
---




security-admin/src/main/java/org/apache/ranger/biz/UserMgr.java
Line 621 (original), 621 (patched)


No need to make it public, the test class is in the same package, so 
package protected would be fine.



security-admin/src/main/java/org/apache/ranger/biz/UserMgr.java
Line 813 (original), 813 (patched)


No need to make it public, the test class is in the same package, so 
package protected would be fine.



security-admin/src/main/java/org/apache/ranger/biz/UserMgr.java
Line 1186 (original), 1186 (patched)


No need to make it public, the test class is in the same package, so 
package protected would be fine.



security-admin/src/main/java/org/apache/ranger/biz/XUserMgr.java
Line 328 (original), 328 (patched)


No need to make it public, the test class is in the same package, so 
package protected would be fine.



security-admin/src/main/java/org/apache/ranger/biz/XUserMgr.java
Lines 637 (patched)


The vxGroupUserList = new VXGroupUserList() assignment is unnecessary, as 
it is reassigned in the next line



security-admin/src/main/java/org/apache/ranger/biz/XUserMgr.java
Line 875 (original), 870 (patched)


No need to make it public, the test class is in the same package, so 
package protected would be fine.



security-admin/src/main/java/org/apache/ranger/biz/XUserMgr.java
Line 1270 (original)


returnList is assigned in the next lines - no need to declare to be null.



security-admin/src/main/java/org/apache/ranger/biz/XUserMgr.java
Line 2258 (original), 2251 (patched)


No need to make it public, the test class is in the same package, so 
package protected would be fine.



security-admin/src/test/java/org/apache/ranger/biz/TestUserMgr.java
Lines 191 (patched)


You can annotate this method with @After and JUnit will call it 
automatically after every test method, no need to call it by hand.


- Zsombor Gegesy


On July 5, 2018, 8:48 p.m., Pradeep Agrawal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67835/
> ---
> 
> (Updated July 5, 2018, 8:48 p.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Don Bosco Durai, Gautam Borad, Abhay 
> Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Selvamohan Neethiraj, 
> Sailaja Polavarapu, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2150
> https://issues.apache.org/jira/browse/RANGER-2150
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> **Problem Statement:** Unit test coverage for XUserMgr and UserMgr class need 
> to be increased.
> 
> Currently Unit Test coverage of XUserMgr is 33.5% and UserMgr class is 54.9%
> Total No. of test cases in TestXUserMgr class are: 37
> Total No. of test cases in TestUserMgr class are: 23
> 
> **Proposed Solution:**
> Proposed patch contains lot of additional unit test cases and modification in 
> existing unit test cases.
> Statistics after the patch are:
> Test coverage of XUserMgr is 90.2% and UserMgr class is 85.3%
> Total No. of test cases in TestXUserMgr class are: 100
> Total No. of test cases in TestUserMgr class are: 41
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/java/org/apache/ranger/biz/UserMgr.java 6d94e4f 
>   security-admin/src/main/java/org/apache/ranger/biz/XUserMgr.java 7f5eab7 
>   security-admin/src/test/java/org/apache/ranger/biz/TestUserMgr.java 10453a5 
>   security-admin/src/test/java/org/apache/ranger/biz/TestXUserMgr.java 
> 39c60a9 
> 
> 
> Diff: https://reviews.apache.org/r/67835/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on my local environment by using command :
> 
> mvn clean compile package assembly:assembly install
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] ranger . SUCCESS [  9.399 
> s]
> [INFO] Jdbc SQL Connector . SUCCESS [  2.039 
> s]
> [INFO] Credential Support . SUCCESS [  0.792 
> s]
> [INFO] Audit Component  SUCCESS [  2.001 
> s]
> [INFO] Common library for Plugins . SUCCESS [  3.299 
> s]
> [INFO] Installer Support Component  SUCCESS [  0.398 
> s]
> [INFO] Cr

[jira] [Commented] (RANGER-1951) build problems with the saveVersion.py script

2018-07-05 Thread Zsombor Gegesy (JIRA) 


[ 
https://issues.apache.org/jira/browse/RANGER-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534167#comment-16534167
 ] 

Zsombor Gegesy commented on RANGER-1951:


New patch added!

> build problems with the saveVersion.py script
> -
>
> Key: RANGER-1951
> URL: https://issues.apache.org/jira/browse/RANGER-1951
> Project: Ranger
>  Issue Type: Bug
>  Components: build-infra
>Affects Versions: 0.7.1
>Reporter: Zsombor Gegesy
>Assignee: Zsombor Gegesy
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: RANGER-1951-1.patch, RANGER-1951-2.patch
>
>
> Currently the saveVersion.py has the following problems:
> * it doesn't work with python3 due to 'inconsistent whitespace usage' and 
> because in python3 the byte array is different from a string
> * The checksum is generated from all the java source files from 
> ranger-util/target, which contains at most one java file - a previously 
> generated ranger-util/target/gen/org/apache/ranger/common/package-info.java 
> * 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (RANGER-1951) build problems with the saveVersion.py script

2018-07-05 Thread Zsombor Gegesy (JIRA) 


 [ 
https://issues.apache.org/jira/browse/RANGER-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zsombor Gegesy updated RANGER-1951:
---
Attachment: RANGER-1951-2.patch

> build problems with the saveVersion.py script
> -
>
> Key: RANGER-1951
> URL: https://issues.apache.org/jira/browse/RANGER-1951
> Project: Ranger
>  Issue Type: Bug
>  Components: build-infra
>Affects Versions: 0.7.1
>Reporter: Zsombor Gegesy
>Assignee: Zsombor Gegesy
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: RANGER-1951-1.patch, RANGER-1951-2.patch
>
>
> Currently the saveVersion.py has the following problems:
> * it doesn't work with python3 due to 'inconsistent whitespace usage' and 
> because in python3 the byte array is different from a string
> * The checksum is generated from all the java source files from 
> ranger-util/target, which contains at most one java file - a previously 
> generated ranger-util/target/gen/org/apache/ranger/common/package-info.java 
> * 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 67836: RANGER-1951 - fix to work outside of a git repository

2018-07-05 Thread Zsombor Gegesy 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67836/
---

Review request for ranger.


Bugs: RANGER-1951
https://issues.apache.org/jira/browse/RANGER-1951


Repository: ranger


Description
---

Fix the python script, which calculates the source code hash, to work even 
outside of a git repository.


Diffs
-

  ranger-util/src/scripts/saveVersion.py acf82bfb6 


Diff: https://reviews.apache.org/r/67836/diff/1/


Testing
---

Tested with python2 and python3, inside a git repository, and outside of it.


Thanks,

Zsombor Gegesy

Review Request 67835: RANGER-2150: Unit test coverage for XUserMgr and UserMgr class

2018-07-05 Thread Pradeep Agrawal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67835/
---

Review request for ranger, Ankita Sinha, Don Bosco Durai, Gautam Borad, Abhay 
Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Selvamohan Neethiraj, 
Sailaja Polavarapu, and Velmurugan Periasamy.


Bugs: RANGER-2150
https://issues.apache.org/jira/browse/RANGER-2150


Repository: ranger


Description
---

**Problem Statement:** Unit test coverage for XUserMgr and UserMgr class need 
to be increased.

Currently Unit Test coverage of XUserMgr is 33.5% and UserMgr class is 54.9%
Total No. of test cases in TestXUserMgr class are: 37
Total No. of test cases in TestUserMgr class are: 23

**Proposed Solution:**
Proposed patch contains lot of additional unit test cases and modification in 
existing unit test cases.
Statistics after the patch are:
Test coverage of XUserMgr is 90.2% and UserMgr class is 85.3%
Total No. of test cases in TestXUserMgr class are: 100
Total No. of test cases in TestUserMgr class are: 41


Diffs
-

  security-admin/src/main/java/org/apache/ranger/biz/UserMgr.java 6d94e4f 
  security-admin/src/main/java/org/apache/ranger/biz/XUserMgr.java 7f5eab7 
  security-admin/src/test/java/org/apache/ranger/biz/TestUserMgr.java 10453a5 
  security-admin/src/test/java/org/apache/ranger/biz/TestXUserMgr.java 39c60a9 


Diff: https://reviews.apache.org/r/67835/diff/1/


Testing
---

Tested on my local environment by using command :

mvn clean compile package assembly:assembly install

[INFO] Reactor Summary:
[INFO]
[INFO] ranger . SUCCESS [  9.399 s]
[INFO] Jdbc SQL Connector . SUCCESS [  2.039 s]
[INFO] Credential Support . SUCCESS [  0.792 s]
[INFO] Audit Component  SUCCESS [  2.001 s]
[INFO] Common library for Plugins . SUCCESS [  3.299 s]
[INFO] Installer Support Component  SUCCESS [  0.398 s]
[INFO] Credential Builder . SUCCESS [  1.066 s]
[INFO] Embedded Web Server Invoker  SUCCESS [  1.243 s]
[INFO] Key Management Service . SUCCESS [  2.776 s]
[INFO] ranger-plugin-classloader .. SUCCESS [  0.463 s]
[INFO] HBase Security Plugin Shim . SUCCESS [  2.522 s]
[INFO] HBase Security Plugin .. SUCCESS [  1.758 s]
[INFO] Hdfs Security Plugin ... SUCCESS [  1.822 s]
[INFO] Hive Security Plugin ... SUCCESS [  7.668 s]
[INFO] Knox Security Plugin Shim .. SUCCESS [  1.452 s]
[INFO] Knox Security Plugin ... SUCCESS [  4.083 s]
[INFO] Storm Security Plugin .. SUCCESS [  1.046 s]
[INFO] YARN Security Plugin ... SUCCESS [  0.924 s]
[INFO] Ranger Util  SUCCESS [  7.301 s]
[INFO] Unix Authentication Client . SUCCESS [  1.258 s]
[INFO] Security Admin Web Application . SUCCESS [01:31 min]
[INFO] KAFKA Security Plugin .. SUCCESS [  4.827 s]
[INFO] SOLR Security Plugin ... SUCCESS [  5.235 s]
[INFO] NiFi Security Plugin ... SUCCESS [  0.950 s]
[INFO] Unix User Group Synchronizer ... SUCCESS [03:34 min]
[INFO] Ldap Config Check Tool . SUCCESS [  2.540 s]
[INFO] Unix Authentication Service  SUCCESS [  4.010 s]
[INFO] KMS Security Plugin  SUCCESS [  5.922 s]
[INFO] Tag Synchronizer ... SUCCESS [ 12.373 s]
[INFO] Hdfs Security Plugin Shim .. SUCCESS [ 12.243 s]
[INFO] Hive Security Plugin Shim .. SUCCESS [01:00 min]
[INFO] YARN Security Plugin Shim .. SUCCESS [  6.474 s]
[INFO] Storm Security Plugin shim . SUCCESS [  3.564 s]
[INFO] KAFKA Security Plugin Shim . SUCCESS [  1.029 s]
[INFO] SOLR Security Plugin Shim .. SUCCESS [  6.671 s]
[INFO] Atlas Security Plugin Shim . SUCCESS [  2.348 s]
[INFO] KMS Security Plugin Shim ... SUCCESS [  2.024 s]
[INFO] ranger-examples  SUCCESS [  0.422 s]
[INFO] Ranger Examples - Conditions and ContextEnrichers .. SUCCESS [  1.269 s]
[INFO] Ranger Examples - SampleApp  SUCCESS [  0.380 s]
[INFO] Ranger Examples - Ranger Plugin for SampleApp .. SUCCESS [  0.942 s]
[INFO] Ranger Tools ..

[jira] [Commented] (RANGER-2149) REST search queries make Ranger incredibly slow

2018-07-05 Thread Pradeep Agrawal (JIRA) 


[ 
https://issues.apache.org/jira/browse/RANGER-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534110#comment-16534110
 ] 

Pradeep Agrawal commented on RANGER-2149:
-

[~aposledov] MySQL 5.1 is not recommended for Ranger, if possible use 5.6

I will try this exercise at my end and shall post the stats. It should not take 
more than a second or two.

 

> REST search queries make Ranger incredibly slow 
> 
>
> Key: RANGER-2149
> URL: https://issues.apache.org/jira/browse/RANGER-2149
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 0.7.0
>Reporter: Alexander Posledov
>Assignee: Pradeep Agrawal
>Priority: Major
>
> If amount of Ranger policies is relatively big (3000-4000), then REST search 
> queries (e.g. search policy by name) make Ranger very slow. Example:
> curl -u admin:admin -X GET 
> http://host:6080/service/plugins/policies?policyName=somePolicyName
> 2-3 such queries (if we have ~3k policies) can cause too long delays even for 
> simple requests, while these searches aren't completed. In some cases, when 
> we ran several search queries simultaneously, this Ranger behavior caused 9 
> min delay for simple adding a new policy.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (RANGER-2149) REST search queries make Ranger incredibly slow

2018-07-05 Thread Pradeep Agrawal (JIRA) 


[ 
https://issues.apache.org/jira/browse/RANGER-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534107#comment-16534107
 ] 

Pradeep Agrawal commented on RANGER-2149:
-

[~bosco] : We don't have an index on the name column of x_policy table. Still, 
It should not take this much time so will try this at my end and shall post the 
stats.

> REST search queries make Ranger incredibly slow 
> 
>
> Key: RANGER-2149
> URL: https://issues.apache.org/jira/browse/RANGER-2149
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 0.7.0
>Reporter: Alexander Posledov
>Assignee: Pradeep Agrawal
>Priority: Major
>
> If amount of Ranger policies is relatively big (3000-4000), then REST search 
> queries (e.g. search policy by name) make Ranger very slow. Example:
> curl -u admin:admin -X GET 
> http://host:6080/service/plugins/policies?policyName=somePolicyName
> 2-3 such queries (if we have ~3k policies) can cause too long delays even for 
> simple requests, while these searches aren't completed. In some cases, when 
> we ran several search queries simultaneously, this Ranger behavior caused 9 
> min delay for simple adding a new policy.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (RANGER-2150) Unit test coverage for XUserMgr and UserMgr class

2018-07-05 Thread Pradeep Agrawal (JIRA) 


[ 
https://issues.apache.org/jira/browse/RANGER-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534100#comment-16534100
 ] 

Pradeep Agrawal commented on RANGER-2150:
-

The proposed patch contains a lot of additional unit test cases and 
modification in existing unit test cases.

Statistics after the patch are: Test coverage of XUserMgr is 90.2% and UserMgr 
class is 85.3%.

Total No. of test cases in TestUserMgr class is: 41

Total No. of test cases in TestXUserMgr class are: 100

> Unit test coverage for XUserMgr and UserMgr class
> -
>
> Key: RANGER-2150
> URL: https://issues.apache.org/jira/browse/RANGER-2150
> Project: Ranger
>  Issue Type: Test
>  Components: Ranger
>Affects Versions: 1.0.0
>Reporter: Pradeep Agrawal
>Assignee: Pradeep Agrawal
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: 
> 0001-RANGER-2150-Unit-test-coverage-for-XUserMgr-and-User.patch
>
>
> Unit Test coverage of XUserMgr is 33.5% and UserMgr class is 54.9%
> Total No. of test cases in TestXUserMgr class: 37
> Total No. of test cases in TestUserMgr class: 23
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (RANGER-2150) Unit test coverage for XUserMgr and UserMgr class

2018-07-05 Thread Pradeep Agrawal (JIRA) 


 [ 
https://issues.apache.org/jira/browse/RANGER-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pradeep Agrawal updated RANGER-2150:

Attachment: 0001-RANGER-2150-Unit-test-coverage-for-XUserMgr-and-User.patch

> Unit test coverage for XUserMgr and UserMgr class
> -
>
> Key: RANGER-2150
> URL: https://issues.apache.org/jira/browse/RANGER-2150
> Project: Ranger
>  Issue Type: Test
>  Components: Ranger
>Affects Versions: 1.0.0
>Reporter: Pradeep Agrawal
>Assignee: Pradeep Agrawal
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: 
> 0001-RANGER-2150-Unit-test-coverage-for-XUserMgr-and-User.patch
>
>
> Unit Test coverage of XUserMgr is 33.5% and UserMgr class is 54.9%
> Total No. of test cases in TestXUserMgr class: 37
> Total No. of test cases in TestUserMgr class: 23
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (RANGER-2150) Unit test coverage for XUserMgr and UserMgr class

2018-07-05 Thread Velmurugan Periasamy (JIRA) 


 [ 
https://issues.apache.org/jira/browse/RANGER-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Velmurugan Periasamy updated RANGER-2150:
-
Fix Version/s: 2.0.0

> Unit test coverage for XUserMgr and UserMgr class
> -
>
> Key: RANGER-2150
> URL: https://issues.apache.org/jira/browse/RANGER-2150
> Project: Ranger
>  Issue Type: Test
>  Components: Ranger
>Affects Versions: 1.0.0
>Reporter: Pradeep Agrawal
>Assignee: Pradeep Agrawal
>Priority: Major
> Fix For: 2.0.0
>
>
> Unit Test coverage of XUserMgr is 33.5% and UserMgr class is 54.9%
> Total No. of test cases in TestXUserMgr class: 37
> Total No. of test cases in TestUserMgr class: 23
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[VOTE] Apache Ranger Release 1.1.0-rc2

2018-07-05 Thread Velmurugan Periasamy 
Hello Rangers:

Thank you for your contribution to Apache Ranger community. Apache Ranger 1.1.0 
release candidate #2 is now available for a vote within dev community. Note: 
RC1 (https://dist.apache.org/repos/dist/dev/ranger/1.1.0-rc1/) is canceled due 
to https://issues.apache.org/jira/browse/RANGER-1951. 

Links to RC2 release artifacts are given below. Kindly request all Rangers 
(Dev's & PMC members) to review and vote on this release.

 
Git tag for the release:
https://github.com/apache/ranger/tree/ranger-1.1.0-rc2 (last commit id:  
1732536f1de4ac3f1699f660e089385487004261)


Sources for the release:
https://dist.apache.org/repos/dist/dev/ranger/1.1.0-rc2/apache-ranger-1.1.0.tar.gz

 
Source release verification:
 
PGP Signature:
https://dist.apache.org/repos/dist/dev/ranger/1.1.0-rc2/apache-ranger-1.1.0.tar.gz.asc
  
MD5/SHA Hashes:
https://dist.apache.org/repos/dist/dev/ranger/1.1.0-rc2/apache-ranger-1.1.0.tar.gz.mds
https://dist.apache.org/repos/dist/dev/ranger/1.1.0-rc2/apache-ranger-1.1.0.tar.gz.sha1
https://dist.apache.org/repos/dist/dev/ranger/1.1.0-rc2/apache-ranger-1.1.0.tar.gz.sha256
https://dist.apache.org/repos/dist/dev/ranger/1.1.0-rc2/apache-ranger-1.1.0.tar.gz.sha512


Keys to verify the signature of the release artifact are available at: 
https://dist.apache.org/repos/dist/release/ranger/KEYS
 

Release Notes:
https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+1.1.0+-+Release+Notes
 

Build verification steps can be found at:
   http://ranger.apache.org/quick_start_guide.html

 
The vote will be open for at least 72 hours or until necessary number of votes 
are reached.
[ ] +1  approve
[ ] +0  no opinion
[ ] -1  disapprove (and reason why)
 
Here is my +1

Thank you,
Vel

[jira] [Updated] (RANGER-2150) Unit test coverage for XUserMgr and UserMgr class

2018-07-05 Thread Pradeep Agrawal (JIRA) 


 [ 
https://issues.apache.org/jira/browse/RANGER-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pradeep Agrawal updated RANGER-2150:

Fix Version/s: (was: 1.1.0)

> Unit test coverage for XUserMgr and UserMgr class
> -
>
> Key: RANGER-2150
> URL: https://issues.apache.org/jira/browse/RANGER-2150
> Project: Ranger
>  Issue Type: Test
>  Components: Ranger
>Affects Versions: 1.0.0
>Reporter: Pradeep Agrawal
>Assignee: Pradeep Agrawal
>Priority: Major
>
> Unit Test coverage of XUserMgr is 33.5% and UserMgr class is 54.9%
> Total No. of test cases in TestXUserMgr class: 37
> Total No. of test cases in TestUserMgr class: 23
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (RANGER-2150) Unit test coverage for XUserMgr and UserMgr class

2018-07-05 Thread Pradeep Agrawal (JIRA) 
Pradeep Agrawal created RANGER-2150:
---

 Summary: Unit test coverage for XUserMgr and UserMgr class
 Key: RANGER-2150
 URL: https://issues.apache.org/jira/browse/RANGER-2150
 Project: Ranger
  Issue Type: Test
  Components: Ranger
Affects Versions: 1.0.0
Reporter: Pradeep Agrawal
Assignee: Pradeep Agrawal
 Fix For: 1.1.0


Unit Test coverage of XUserMgr is 33.5% and UserMgr class is 54.9%

Total No. of test cases in TestXUserMgr class: 37

Total No. of test cases in TestUserMgr class: 23

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (RANGER-1951) build problems with the saveVersion.py script

2018-07-05 Thread Velmurugan Periasamy (JIRA) 


[ 
https://issues.apache.org/jira/browse/RANGER-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534019#comment-16534019
 ] 

Velmurugan Periasamy edited comment on RANGER-1951 at 7/5/18 6:58 PM:
--

This patch creates issue during preparation of source release for 1.1.0. So I 
am reopening this issue and reverting this commit in ranger-1.1 and ranger-1 
branches. Leaving the commit in master branch and setting fix version as 2.0.0. 

[~zsombor] - Can you please take a look?

+Steps to reproduce:+ (as seen here 
[http://ranger.apache.org/quick_start_guide.html)|http://ranger.apache.org/quick_start_guide.html)+]

1] Git clone and build the project

2] Create the source artifacts for release

3] Now as part of verification, try building from the source tar.gz file. It 
fails with the below error.

{{[INFO] 
}}
 {{[INFO] Building Ranger Util 1.1.0}}
 {{[INFO] 
}}
 {{[INFO]}}
 {{[INFO] — maven-clean-plugin:2.6.1:clean (default-clean) @ ranger-util ---}}
 {{[INFO] Deleting 
/Users/user123/apache-release/1.1.0/verification/ver2/apache-ranger-1.1.0/ranger-util/target}}
 {{[INFO]}}
 {{[INFO] — maven-enforcer-plugin:1.4.1:enforce (enforce-maven-version) @ 
ranger-util ---}}
 {{[INFO]}}
 {{[INFO] — maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ 
ranger-util ---}}
 {{[INFO]}}
 {{[INFO] — maven-antrun-plugin:1.7:run (generate-version-annotation) @ 
ranger-util ---}}
 {{[INFO] Executing tasks}}{{main:}}
 {{ [exec] fatal: Not a git repository (or any of the parent directories): 
.git}}
 {{ [exec] Traceback (most recent call last):}}
 {{ [exec] File 
"/Users/user123/apache-release/1.1.0/verification/ver2/apache-ranger-1.1.0/ranger-util/src/scripts/saveVersion.py",
 line 150, in }}
 {{ [exec] main()}}
 {{ [exec] File 
"/Users/user123/apache-release/1.1.0/verification/ver2/apache-ranger-1.1.0/ranger-util/src/scripts/saveVersion.py",
 line 68, in main}}
 {{ [exec] output = check_output(query)}}
 {{ [exec] File 
"/Users/user123/apache-release/1.1.0/verification/ver2/apache-ranger-1.1.0/ranger-util/src/scripts/saveVersion.py",
 line 40, in check_output}}
 {{ [exec] output = subprocess.check_output(query)}}
 {{ [exec] File 
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
 line 573, in check_output}}
 {{ [exec] raise CalledProcessError(retcode, cmd, output=output)}}
 {{ [exec] subprocess.CalledProcessError: Command '['git', 'rev-parse', 
'HEAD']' returned non-zero exit status 128}}
 {{[INFO]}}
 {{[INFO] 
}}
 {{[INFO] Skipping ranger}}
 {{[INFO] This project has been banned from the build due to previous 
failures.}}
 {{[INFO] 
}}{{}}

 


was (Author: vperiasamy):
This patch creates issue during preparation of source release for 1.1.0. So I 
am reopening this issue and reverting this commit in ranger-1.1 and ranger-1 
branches. Leaving the commit in master branch and setting fix version as 2.0.0. 

[~zsombor] - Can you please take a look?

+Steps to reproduce: (as seen here 
http://ranger.apache.org/quick_start_guide.html)+

1] Git clone and build the project

2] Create the source artifacts for release

3] Now as part of verification, try building from the source tar.gz file. It 
fails with the below error.

{{[INFO] 
}}
{{[INFO] Building Ranger Util 1.1.0}}
{{[INFO] 
}}
{{[INFO]}}
{{[INFO] --- maven-clean-plugin:2.6.1:clean (default-clean) @ ranger-util ---}}
{{[INFO] Deleting 
/Users/user123/apache-release/1.1.0/verification/ver2/apache-ranger-1.1.0/ranger-util/target}}
{{[INFO]}}
{{[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-maven-version) @ 
ranger-util ---}}
{{[INFO]}}
{{[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ 
ranger-util ---}}
{{[INFO]}}
{{[INFO] --- maven-antrun-plugin:1.7:run (generate-version-annotation) @ 
ranger-util ---}}
{{[INFO] Executing tasks}}{{main:}}
{{ [exec] fatal: Not a git repository (or any of the parent directories): .git}}
{{ [exec] Traceback (most recent call last):}}
{{ [exec] File 
"/Users/user123/apache-release/1.1.0/verification/ver2/apache-ranger-1.1.0/ranger-util/src/scripts/saveVersion.py",
 line 150, in }}
{{ [exec] main()}}
{{ [exec] File 
"/Users/user123/apache-release/1.1.0/verification/ver2/apache-ranger-1.1.0/ranger-util/src/scripts/saveVersion.py",
 line 68, in main}}
{{ [exec] output = check_output(query)}}
{{ [exec] File 
"/Users/user123/apache-release/1.1.0/verification/ver2/apache-ranger-1.1.0/ranger-util/src/scripts/saveVersion.py",
 line 40, in check_o

[jira] [Updated] (RANGER-1951) build problems with the saveVersion.py script

2018-07-05 Thread Velmurugan Periasamy (JIRA) 


 [ 
https://issues.apache.org/jira/browse/RANGER-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Velmurugan Periasamy updated RANGER-1951:
-
Fix Version/s: (was: 1.1.0)
   2.0.0

> build problems with the saveVersion.py script
> -
>
> Key: RANGER-1951
> URL: https://issues.apache.org/jira/browse/RANGER-1951
> Project: Ranger
>  Issue Type: Bug
>  Components: build-infra
>Affects Versions: 0.7.1
>Reporter: Zsombor Gegesy
>Assignee: Zsombor Gegesy
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: RANGER-1951-1.patch
>
>
> Currently the saveVersion.py has the following problems:
> * it doesn't work with python3 due to 'inconsistent whitespace usage' and 
> because in python3 the byte array is different from a string
> * The checksum is generated from all the java source files from 
> ranger-util/target, which contains at most one java file - a previously 
> generated ranger-util/target/gen/org/apache/ranger/common/package-info.java 
> * 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (RANGER-1951) build problems with the saveVersion.py script

2018-07-05 Thread Velmurugan Periasamy (JIRA) 


[ 
https://issues.apache.org/jira/browse/RANGER-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534019#comment-16534019
 ] 

Velmurugan Periasamy commented on RANGER-1951:
--

This patch creates issue during preparation of source release for 1.1.0. So I 
am reopening this issue and reverting this commit in ranger-1.1 and ranger-1 
branches. Leaving the commit in master branch and setting fix version as 2.0.0. 

[~zsombor] - Can you please take a look?

+Steps to reproduce: (as seen here 
http://ranger.apache.org/quick_start_guide.html)+

1] Git clone and build the project

2] Create the source artifacts for release

3] Now as part of verification, try building from the source tar.gz file. It 
fails with the below error.

{{[INFO] 
}}
{{[INFO] Building Ranger Util 1.1.0}}
{{[INFO] 
}}
{{[INFO]}}
{{[INFO] --- maven-clean-plugin:2.6.1:clean (default-clean) @ ranger-util ---}}
{{[INFO] Deleting 
/Users/user123/apache-release/1.1.0/verification/ver2/apache-ranger-1.1.0/ranger-util/target}}
{{[INFO]}}
{{[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-maven-version) @ 
ranger-util ---}}
{{[INFO]}}
{{[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ 
ranger-util ---}}
{{[INFO]}}
{{[INFO] --- maven-antrun-plugin:1.7:run (generate-version-annotation) @ 
ranger-util ---}}
{{[INFO] Executing tasks}}{{main:}}
{{ [exec] fatal: Not a git repository (or any of the parent directories): .git}}
{{ [exec] Traceback (most recent call last):}}
{{ [exec] File 
"/Users/user123/apache-release/1.1.0/verification/ver2/apache-ranger-1.1.0/ranger-util/src/scripts/saveVersion.py",
 line 150, in }}
{{ [exec] main()}}
{{ [exec] File 
"/Users/user123/apache-release/1.1.0/verification/ver2/apache-ranger-1.1.0/ranger-util/src/scripts/saveVersion.py",
 line 68, in main}}
{{ [exec] output = check_output(query)}}
{{ [exec] File 
"/Users/user123/apache-release/1.1.0/verification/ver2/apache-ranger-1.1.0/ranger-util/src/scripts/saveVersion.py",
 line 40, in check_output}}
{{ [exec] output = subprocess.check_output(query)}}
{{ [exec] File 
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
 line 573, in check_output}}
{{ [exec] raise CalledProcessError(retcode, cmd, output=output)}}
{{ [exec] subprocess.CalledProcessError: Command '['git', 'rev-parse', 'HEAD']' 
returned non-zero exit status 128}}
{{[INFO]}}
{{[INFO] 
}}
{{[INFO] Skipping ranger}}
{{[INFO] This project has been banned from the build due to previous failures.}}
{{[INFO] 
}}{{}}

 

> build problems with the saveVersion.py script
> -
>
> Key: RANGER-1951
> URL: https://issues.apache.org/jira/browse/RANGER-1951
> Project: Ranger
>  Issue Type: Bug
>  Components: build-infra
>Affects Versions: 0.7.1
>Reporter: Zsombor Gegesy
>Assignee: Zsombor Gegesy
>Priority: Major
> Fix For: 1.1.0
>
> Attachments: RANGER-1951-1.patch
>
>
> Currently the saveVersion.py has the following problems:
> * it doesn't work with python3 due to 'inconsistent whitespace usage' and 
> because in python3 the byte array is different from a string
> * The checksum is generated from all the java source files from 
> ranger-util/target, which contains at most one java file - a previously 
> generated ranger-util/target/gen/org/apache/ranger/common/package-info.java 
> * 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (RANGER-1951) build problems with the saveVersion.py script

2018-07-05 Thread Velmurugan Periasamy (JIRA) 


 [ 
https://issues.apache.org/jira/browse/RANGER-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Velmurugan Periasamy reopened RANGER-1951:
--

> build problems with the saveVersion.py script
> -
>
> Key: RANGER-1951
> URL: https://issues.apache.org/jira/browse/RANGER-1951
> Project: Ranger
>  Issue Type: Bug
>  Components: build-infra
>Affects Versions: 0.7.1
>Reporter: Zsombor Gegesy
>Assignee: Zsombor Gegesy
>Priority: Major
> Fix For: 1.1.0
>
> Attachments: RANGER-1951-1.patch
>
>
> Currently the saveVersion.py has the following problems:
> * it doesn't work with python3 due to 'inconsistent whitespace usage' and 
> because in python3 the byte array is different from a string
> * The checksum is generated from all the java source files from 
> ranger-util/target, which contains at most one java file - a previously 
> generated ranger-util/target/gen/org/apache/ranger/common/package-info.java 
> * 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (RANGER-2017) Ranger KMS encryption good practices

2018-07-05 Thread Velmurugan Periasamy (JIRA) 


 [ 
https://issues.apache.org/jira/browse/RANGER-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Velmurugan Periasamy resolved RANGER-2017.
--
Resolution: Fixed

> Ranger KMS encryption good practices
> 
>
> Key: RANGER-2017
> URL: https://issues.apache.org/jira/browse/RANGER-2017
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Reporter: Selvamohan Neethiraj
>Assignee: bhavik patel
>Priority: Critical
> Fix For: 1.1.0
>
> Attachments: RANGER-2017-master-01.patch, 
> RANGER-2017-master-02.patch, RANGER-2017-master.patch
>
>
> Change the default crypt algorithm to use a stronger cipher algorithm in 
> Ranger KMS (Similar to RANGER-1644 for Ranger Admin). Also make these options 
> configurable.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (RANGER-2148) Update Ranger Hive dependency version to 3.0

2018-07-05 Thread Ramesh Mani (JIRA) 


[ 
https://issues.apache.org/jira/browse/RANGER-2148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533322#comment-16533322
 ] 

Ramesh Mani commented on RANGER-2148:
-

[~coheigea] yes this is targeted for 2.0.0. If you have a patch please raise a 
Jira for Hadoop 3.x and we can review it and piggy bag this JIRA with it. 
Thanks.

> Update Ranger Hive dependency version to 3.0 
> -
>
> Key: RANGER-2148
> URL: https://issues.apache.org/jira/browse/RANGER-2148
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: master
>Reporter: Ramesh Mani
>Assignee: Ramesh Mani
>Priority: Critical
> Fix For: master
>
>
> Update Ranger Hive dependency version to 3.0 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)