Re: Review Request 58272: Ranger-1502: Solr shutdown does not cause the audit log file to be flushed and closed
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58272/ --- (Updated Aug. 4, 2017, 7:54 a.m.) Review request for ranger. Bugs: RANGER-1502 https://issues.apache.org/jira/browse/RANGER-1502 Repository: ranger Description --- The current audit stream close mechanism uses a Java shutdown hook registered with Hadoop's ShutdownHookManager. Solr shutdown, however, somehow does not cause the shutdown hook to be invoked, potentially resulting lost audit logs. We are experiencing lost logs toward HDFS audit destination. The fix is to add invocation of the shutdown hook from the close() call of the Solr's AuthorizationPlugin interface, in addition to the existing invocation mechanism from the JVM invocation of shutdown hooks. Diffs - agents-audit/src/main/java/org/apache/ranger/audit/provider/AuditProviderFactory.java b37011e6 plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java bf22b476 Diff: https://reviews.apache.org/r/58272/diff/1/ Testing --- automated and manual Thanks, Yan Zhou
Re: Review Request 58268: Ranger-1501: Audit Flush to HDFS does not actually cause the audit logs to be flushed to HDFS
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58268/ --- (Updated April 25, 2017, 2:54 p.m.) Review request for ranger. Bugs: RANGER-1501 https://issues.apache.org/jira/browse/RANGER-1501 Repository: ranger Description --- The reason is that HDFS file stream's flush() call does not really flush the data all the way to disk, nor even makes the data visible to HDFS users. See the HDFS semantics of the flush/sync at https://issues.apache.org/jira/browse/HADOOP-6313. Consequently the audit logs on HDFS won't be visible/durable from HDFS client until the log file is closed. This will, among other issues, boost chances of losing audit logs in case of system failure. Diffs - agents-audit/src/main/java/org/apache/ranger/audit/destination/HDFSAuditDestination.java 889b6ffd Diff: https://reviews.apache.org/r/58268/diff/1/ Testing --- Automated and manual Thanks, Yan Zhou
Re: Review Request 53240: Ranger-1181: HDFS Plugin does not allow removal of a non-empty directory if the directory is allowed to be removed by HDFS, but the file inside the directory is allowed to be
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53240/ --- (Updated April 13, 2017, 3:18 p.m.) Review request for ranger. Bugs: RANGER-1181 https://issues.apache.org/jira/browse/RANGER-1181 Repository: ranger Description --- Reproduction Steps: 1. Ranger is installed and HDFS plug-in is enabled. 2. As qaadmin user, create a folder on HDFS with permission 500: hadoop fs -mkdir /tmp/rangertest1 hadoop fs -chmod 500 /tmp/rangertest1 while the /tmp itself has the 777 as the HDFS bits: hadoop fs -ls / drwxrwxrwx - user1 group1 0 2016-10-03 14:54 /tmp 3. Create a Ranger policy p1_1 by granting qaadmin with RWX permission to the folder of /tmp/rangertest1, recursive set to true 4. Wait for around >30 seconds after Policy synced up. 5. Put a file to /tmp/rangertest1 folder: echo "This is a file2" > /tmp/temp hadoop fs -put /tmp/temp /tmp/rangertest1 hadoop fs -ls /tmp/rangertest1 Found 1 items rw-rr- 3 qaadmin hdfs 16 2016-09-21 19:13 /tmp/rangertest1/temp 6. Try to delete the non-empty folder with "-skipTrash" option, but it failed (delete the empty folder could success): hadoop fs -rm -r -skipTrash /tmp/rangertest1 rm: Permission denied: user=qaadmin, access=ALL, inode="/tmp/rangertest1":qaadmin:hdfs:dr-x-- This happens when the HDFS fallback is enabled. The issue is with the synergy between the Ranger HDFS plugin and the HDFS. The removal of the directory of /tmp/rangertest1 is not covered by a Ranger policy but is allowed by of the HDFS permission for its parent, /tmp. The removal of the file inside /tmp/rangertest1 is, however, not allowed by HDFS for the 500 permission of its parent of /tmp/rangertest1, but allowed by Ranger. When Ranger finds that the directory of rangertest1 can't be removed by Ranger, it falls back to HDFS policy which does not allow the removal of the file inside rangertest1 even though allowed by Ranger. We should use HDFS fallback when the checks on ancestor/parent/current/subdir fails before proceed to the next check but with only the failed access check by the HDFS fallback not all of the access checks for ancestor, parent, current and subdir. Diffs - hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java 6f452da Diff: https://reviews.apache.org/r/53240/diff/3/ Testing --- Unit test works ok. Thanks, Yan Zhou
Re: Review Request 58268: Ranger-1501: Audit Flush to HDFS does not actually cause the audit logs to be flushed to HDFS
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58268/ --- (Updated April 7, 2017, 9:50 p.m.) Review request for ranger. Repository: ranger Description --- The reason is that HDFS file stream's flush() call does not really flush the data all the way to disk, nor even makes the data visible to HDFS users. See the HDFS semantics of the flush/sync at https://issues.apache.org/jira/browse/HADOOP-6313. Consequently the audit logs on HDFS won't be visible/durable from HDFS client until the log file is closed. This will, among other issues, boost chances of losing audit logs in case of system failure. Diffs - agents-audit/src/main/java/org/apache/ranger/audit/destination/HDFSAuditDestination.java 889b6ffd Diff: https://reviews.apache.org/r/58268/diff/1/ Testing --- Automated and manual Thanks, Yan Zhou
Review Request 58272: Ranger-1502: Solr shutdown does not cause the audit log file to be flushed and closed
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58272/ --- Review request for ranger. Repository: ranger Description --- The current audit stream close mechanism uses a Java shutdown hook registered with Hadoop's ShutdownHookManager. Solr shutdown, however, somehow does not cause the shutdown hook to be invoked, potentially resulting lost audit logs. We are experiencing lost logs toward HDFS audit destination. The fix is to add invocation of the shutdown hook from the close() call of the Solr's AuthorizationPlugin interface, in addition to the existing invocation mechanism from the JVM invocation of shutdown hooks. Diffs - agents-audit/src/main/java/org/apache/ranger/audit/provider/AuditProviderFactory.java b37011e6 plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java bf22b476 Diff: https://reviews.apache.org/r/58272/diff/1/ Testing --- automated and manual Thanks, Yan Zhou
Review Request 58268: Ranger-1501: Audit Flush to HDFS does not actually cause the audit logs to be flushed to HDFS
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58268/ --- Review request for ranger. Repository: ranger Description --- The reason is that HDFS file stream's flush() call does not really flush the data all the way to disk, nor even makes the data visible to HDFS users. See the HDFS semantics of the flush/sync at https://issues.apache.org/jira/browse/HADOOP-6313. Consequently the audit logs on HDFS won't be visible/durable from HDFS client until the log file is closed. This will, among other issues, boost chances of losing audit logs in case of system failure. Diffs - agents-audit/src/main/java/org/apache/ranger/audit/destination/HDFSAuditDestination.java 889b6ffd Diff: https://reviews.apache.org/r/58268/diff/1/ Testing --- Automated and manual Thanks, Yan Zhou
Review Request 58115: Ranger-1489: Solr plugin fails to get client address
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58115/ --- Review request for ranger. Repository: ranger Description --- An immediate consequence is that IP-range conditions all fail for Solr authorizations. Diffs - plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java 832302c Diff: https://reviews.apache.org/r/58115/diff/1/ Testing --- auto and manual Thanks, Yan Zhou
Re: Review Request 57519: Ranger-1446: Ranger Solr Plugin does not work when the collection list in the request is empty
> On March 13, 2017, 10:50 a.m., Colm O hEigeartaigh wrote: > > Looks good, but please submit the patch using "git commit" and then "git > > format-patch -n HEAD~" Done, thanks. - Yan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57519/#review168749 --- On March 13, 2017, 3:26 p.m., Yan Zhou wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/57519/ > --- > > (Updated March 13, 2017, 3:26 p.m.) > > > Review request for ranger and Colm O hEigeartaigh. > > > Repository: ranger > > > Description > --- > > The fix of Ranger-1095 set the initial value of "denied" to "true" from the > previous "false". One impact of this change is that, when > context.getCollectionRequests() is empty which could be the case in many > invocations Solr makes to Ranger on authorization per client request, the > permission is plainly denied without going to Ranger policy engine. So the > fix changed the default behavior related to "denied". > A proper fix of Ranger-1095 IMO should be just to set the "denied" to "true" > in the catch block without changing the initial value of the variable. > > > Diffs > - > > > plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java > 6160438 > > > Diff: https://reviews.apache.org/r/57519/diff/2/ > > > Testing > --- > > manual unit > > > Thanks, > > Yan Zhou > >
Re: Review Request 57519: Ranger-1446: Ranger Solr Plugin does not work when the collection list in the request is empty
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57519/ --- (Updated March 13, 2017, 3:26 p.m.) Review request for ranger and Colm O hEigeartaigh. Repository: ranger Description --- The fix of Ranger-1095 set the initial value of "denied" to "true" from the previous "false". One impact of this change is that, when context.getCollectionRequests() is empty which could be the case in many invocations Solr makes to Ranger on authorization per client request, the permission is plainly denied without going to Ranger policy engine. So the fix changed the default behavior related to "denied". A proper fix of Ranger-1095 IMO should be just to set the "denied" to "true" in the catch block without changing the initial value of the variable. Diffs (updated) - plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java 6160438 Diff: https://reviews.apache.org/r/57519/diff/2/ Changes: https://reviews.apache.org/r/57519/diff/1-2/ Testing --- manual unit Thanks, Yan Zhou
Re: Can't change the Assignee to oneself?
https://issues.apache.org/jira/browse/RANGER-1252, yzhou2001 A basic question is whether there is a Ranger jira policy that only allows power users to set assignee? Or a grant of right has to be given by a power user ? Thanks, On Wed, Mar 8, 2017 at 8:51 AM, Colm O hEigeartaigh <cohei...@apache.org> wrote: > What JIRA number is it, and what is your JIRA account username? > > Colm. > > On Wed, Mar 8, 2017 at 4:41 PM, Yan Zhou <yzhouopensou...@gmail.com> > wrote: > > > Hi, I filed a Ranger jira and posted a patch. But it seems to me that I > > can't change the assignee field to myself. Is this the way that Ranger > > jira is supposed to work? Or do I need extra steps in order to be able to > > do so? > > > > Thanks! > > > > > > -- > Colm O hEigeartaigh > > Talend Community Coder > http://coders.talend.com >
Re: Review Request 56829: Ranger-1395: LDAP group sync fails with InvalidNameException
> On Feb. 19, 2017, 10:34 p.m., Selvamohan Neethiraj wrote: > > Can you please provide little more details on how the manual testing was > > done. This would be helpful for reviewer .... > > Yan Zhou wrote: > With the fix, the user sync is run ok without the exception after the > removal of the "short user name" from the "or" logic for the group search, > leaving only the full DN as the user name for the group search. Before the > fix, the same search caused the InvalidNameException thrown from the LDAP > server. > > As stated in the Jira, apparently the problem is only with some LDAP > servers. Using the Apache LDAP server in the Ranger automated user sync test, > TestLdapUserGroup, the failure can't be reproduced. > > Sailaja Polavarapu wrote: > Hi Yan Zhou, > Can you please provide some details on the sample schema or ldapsearch > output of a user and a group on your ldap server? And also, can you provide > some details on the ldap server? I have an openldap server with posixUser and > posixGroup accounts and couldn't repro this issue. And also as you mentioned > the unit test cases use Apache Ldap server and those pass as well. > Also, one quick feedback on the changes - > "useShortUserNameInGroupSearch" is set to true only when the groupObjectClass > is set to posixGroup. This may not be right assumption as there may be a > possiblility that the groupObjectClass is set to "top" and the group member > attribute can still be configured with user's short name right? > > Thanks, > Sailaja. It is an IBM Tivoli Directory Server 6.4. What other LDAP server details do you want to see? As for the check on "posixGroup", I did some investigation on the internet, and it appeared that only this would require a group serach based on short user names. And also the test cases introduced in Ranger-893 only covers the "posixGroup". Probably we need to have a more accurate condition to allow and disallow for group search based on short user names. Any suggestions are welcome. - Yan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56829/#review166044 --- On Feb. 19, 2017, 10:30 p.m., Yan Zhou wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56829/ > --- > > (Updated Feb. 19, 2017, 10:30 p.m.) > > > Review request for ranger. > > > Repository: ranger > > > Description > --- > > Some LDAP servers throw exception on group search on posix user names that > are not full DNs. > > > Diffs > - > > > ugsync/src/main/java/org/apache/ranger/ldapusersync/process/LdapUserGroupBuilder.java > 8cf6816 > > ugsync/src/main/java/org/apache/ranger/unixusersync/process/PolicyMgrUserGroupBuilder.java > 070a39b > > Diff: https://reviews.apache.org/r/56829/diff/ > > > Testing > --- > > Manual > > > Thanks, > > Yan Zhou > >
Re: Review Request 56829: Ranger-1395: LDAP group sync fails with InvalidNameException
> On Feb. 19, 2017, 10:34 p.m., Selvamohan Neethiraj wrote: > > Can you please provide little more details on how the manual testing was > > done. This would be helpful for reviewer With the fix, the user sync is run ok without the exception after the removal of the "short user name" from the "or" logic for the group search, leaving only the full DN as the user name for the group search. Before the fix, the same search caused the InvalidNameException thrown from the LDAP server. As stated in the Jira, apparently the problem is only with some LDAP servers. Using the Apache LDAP server in the Ranger automated user sync test, TestLdapUserGroup, the failure can't be reproduced. - Yan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56829/#review166044 --- On Feb. 19, 2017, 10:30 p.m., Yan Zhou wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56829/ > --- > > (Updated Feb. 19, 2017, 10:30 p.m.) > > > Review request for ranger. > > > Repository: ranger > > > Description > --- > > Some LDAP servers throw exception on group search on posix user names that > are not full DNs. > > > Diffs > - > > > ugsync/src/main/java/org/apache/ranger/ldapusersync/process/LdapUserGroupBuilder.java > 8cf6816 > > ugsync/src/main/java/org/apache/ranger/unixusersync/process/PolicyMgrUserGroupBuilder.java > 070a39b > > Diff: https://reviews.apache.org/r/56829/diff/ > > > Testing > --- > > Manual > > > Thanks, > > Yan Zhou > >
Review Request 56829: Ranger-1395: LDAP group sync fails with InvalidNameException
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56829/ --- Review request for ranger. Repository: ranger Description --- Some LDAP servers throw exception on group search on posix user names that are not full DNs. Diffs - ugsync/src/main/java/org/apache/ranger/ldapusersync/process/LdapUserGroupBuilder.java 8cf6816 ugsync/src/main/java/org/apache/ranger/unixusersync/process/PolicyMgrUserGroupBuilder.java 070a39b Diff: https://reviews.apache.org/r/56829/diff/ Testing --- Manual Thanks, Yan Zhou
Review Request 56094: Ranger-1339: DENY and ALLOW EXCLUSION do not work with YARN
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56094/ --- Review request for ranger. Repository: ranger Description --- When a user is denied, or excluded from "allowed", the use of "admin-queue", but is allowed the "submit-app", he is actually unable to submit Yarn jobs at all. The reason is found to be that the "implied grants" are indiscriminately incorporated into allow/deny/allow-exception/deny-exception lists. Actually we need to differentiate two types of implications. The first implication is "equivalent implication". The second is "unequivalent implication". For the "ALL" permission, it is equivalent, meaning that "ALL" implies the all implied permissions together, and vice versa. So DENY "ALL" will rid of any and all other permissions from a user. For YARN's implication from "queue-admin" to "submit-app", it's not equivalent. While "queue-admin" implies "submit-app", it is not the other way around; namely that deny "admin-queue" to a user should not deny his "submit-app" permission. Thus the "implied grants" should not be incorporated from the allow-exception/deny lists if they do not carry the "all" semantics. Diffs - agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerDefaultPolicyEvaluator.java ffb9523 agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java b0d103e agents-common/src/test/resources/policyengine/test_policyengine_yarn.json PRE-CREATION Diff: https://reviews.apache.org/r/56094/diff/ Testing --- Regression, manual, and newly added automated tests. Thanks, Yan Zhou
