[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438636#comment-15438636 ] Don Bosco Durai commented on HAWQ-256: -- Where is the latest API definition? From the JSON in the above comments, I think we have to make sure we have the following: 1. Groups for the user (either we send in the API or we can do the group lookup from the Ranger PDP (Policy Decision Point) Server itself. 2. Send IP of the client 3. Entire user query (if possible). I have seen users asking for entire query along with the audit record. We can truncate to a max predefined size. I have copied [~sneethiraj], [~madhan.neethiraj] and [~kulkabhay] to give their comments also. Thanks > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438560#comment-15438560 ] Don Bosco Durai commented on HAWQ-256: -- I agree. We can disable grant/revoke on the HAWQ side it Ranger is enabled. I feel, those using Ranger will prefer to manage the policies from one place. Also, even if we support GRANT/REVOKE from the HAWQ SQL command, we shouldn't expect that the behavior will be same when Ranger is enabled. E.g. Currently, when you give delegated admin privilege to any user (e.g. user 1) for the resource, then user1 can give any access to other users (e.g. user2) for that resource regardless what permission user1 has. This addresses use case where you don't want Admin to read/write, but be able to manage permissions for others. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428488#comment-15428488 ] Don Bosco Durai commented on HAWQ-256: -- Starting Ranger 0.6 it also supports Kerberos. Before that it was user/password and two-way SSL. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420978#comment-15420978 ] Don Bosco Durai commented on HAWQ-256: -- [~hubertzhang], fallback has been a very contentious topic. It causes ambiguity and confusion when determining which system really allowed. It is better to have only one source of truth. Also unlike HDFS and YARN, in HAWQ it will be two different systems. So if Ranger return "no" or "unknown", but HAWQ allowed it, then the audit records in Ranger will be wrong or incomplete. I would recommend, unless there is a very compelling reasons, we should support only one source of truth. I also feel, users will prefer consistent and uniform way of managing the policies. So they should be okay if we don't give fall back. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420958#comment-15420958 ] Don Bosco Durai commented on HAWQ-256: -- #1. The group "public" is virtual. It is similar to * or all. All users are part of public and you don't need to add users to public nor can you remove users from public. #2. In Ranger 0.6, deny can be used to explicitly deny users or groups. I would say we should target to support Ranger 0.6 and above. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419784#comment-15419784 ] Don Bosco Durai commented on HAWQ-256: -- + [~madhan.neethiraj] In the Ranger case, this might not be feasible. Because we go by "permissive" model, which means if there are no permissions, then it is "deny". So by absence of a permission, it will be difficult to determine "no privilege" v/s "deny". My suggestion would be for admins who want to set the default behaviors, can do it in Ranger itself. E.g. They can pick the resources (Database, table,etc) and give the desired permission to group "public". Which means, all users will at least get the permissions set in this policy. And they can have different defaults for different resources. It will be easy to manage these centrally, than trying to set the defaults in other config files or mechanism. This might be a better option, because now, the policies (including defaults) are in one place and it is easy to audit who set the default policies and how any end user got access to the resource. Would this be okay for the users? > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15409208#comment-15409208 ] Don Bosco Durai commented on HAWQ-256: -- [~lilima], this is looking good. Just couple of comments: 1. 4.3 is not needed. The policies will be cached in Ranger REST Service 2. Section 6: REST API interface - We should get reviewed with few folks in the Ranger team. There could be already something which we could use and extend 3. Section 6 - We should review the Hive model. SQL command might have multiple resources and different actions on them. E.g. Join, CTAS, etc. So single resource might not work. It needs to be be complex object > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384859#comment-15384859 ] Don Bosco Durai commented on HAWQ-256: -- [~hubertzhang], you are correct. When Ranger is used for authorization, then anything internal/local to Hive (e.g. internal users or roles) are not used. The intention is to keep users and groups consistent across the entire eco-system. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384853#comment-15384853 ] Don Bosco Durai commented on HAWQ-256: -- I don't know the internals of HAWQ to comment much, but I feel, this is a broader discussion and we should probably create another JIRA to handle this. If HAWQ is replicating the users only to give access permission, then when using Ranger, it doesn't have to, because Ranger already syncs with AD/LDAP to manage the policies. So HAWQ only needs to authenticate the user and send the username during authorization call. But since I don't know the internals, so can't suggest much here. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381729#comment-15381729 ] Don Bosco Durai commented on HAWQ-256: -- [~hubertzhang], thanks. The JDBC call should suffice for lookup. Ranger only supports one set of users and it is generally what is supported by Hadoop. The source is either AD/LDAP or linux users. In the case of Hive, if Ranger is used, then Hive's internal users or roles are not used, instead Hadoop Common is used to get users and groups. This keeps the users and groups consistent across all components. I would prefer the same behavior for HAWQ. But for any reason HAWQ needs to support it's own users/groups, then they need to be populated in Ranger also. Since Ranger doesn't have namespace for users, it can't do conflict resolution. So we will have to do what you suggest. It would be good if we defer it and see if users really needs it? Because, users using Ranger prefer uniform user and groups. Thanks > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381250#comment-15381250 ] Don Bosco Durai commented on HAWQ-256: -- You got these two correct. We need to add one more to the list. In RangerAdmin UI, when you create a policy, we do auto-suggest by doing a lookup in the databases and schemas in the component. So on the Ranger Admin side, we will need to write the code to query HAWQ. This is generally the existing APIs provided by the components, so in the case HAWQ it would be JDBC or other any other API supported by HAWQ. We just need to track this for completeness purpose and I don't anticipate any work from the HAWQ side. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378322#comment-15378322 ] Don Bosco Durai commented on HAWQ-256: -- 1. The "Add New User" in Ranger is just to add user in the Ranger DB. The users and groups in Ranger are used to help create policies in Ranger. It is not used as source of truth by the component for users or groups. The main reason being, Ranger doesn't do authentication. So you need to rely on AD/LDAP or use local user/password. 2. In the Ranger integration, the policies are stored in the Ranger DB. Ranger provides UI and REST APIs to create the policies. In Hive and HBase, the grant from their CLI calls our plugin running within their process, which in turn calls Ranger REST API. In the case of HAWQ, the C++ client might make the REST API to the proxy Ranger Server to set the policies. 3. The model we suggest is to abstract the authorization layer. The default behavior is the component natively implementation. And those working in a bigger eco-system can alternatively use Ranger or anyone implementing the component's interface. So for native implementation, technically nothing should change. You still will be saving the ACLs the way you are currently storing and using it. When the user choose Ranger as the option, the policies will be stored in Ranger DB in Ranger format and the Ranger implementation will pull the policies and enforce it. So any ACLs stored in the component native storage will not be used. 5. Same as #2. In addition to Ranger UI and REST API, users can also set policies via native component CLI commands. This is primarily for backward compatibility. However, since Ranger support additional conditions, generally it is not possible to set these conditions via native CLI grant commands. Looking forward for the design document. Thanks > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376324#comment-15376324 ] Don Bosco Durai commented on HAWQ-256: -- [~lilima], thanks for listing down the questions. 1. Ranger uses the user from the component. If it is from LDAP/AD or Linux user, then it gets it from there. But if we are creating (only) in HAWQ, then it needs to be imported into Ranger also. Ranger gives different ways of loading users into it's database. API, file import, LDAP/AD, etc. 2. We shouldn't mix and match. In the Hive, HBase, etc, it is either the component or Ranger. Not both. This will be easy for the users to understand and manage 3. If Ranger is down, we should consider as failure. I don't think we should over-engineer this part. We should make the REST API server HA, so minimize the issue of Ranger down case. 4. Refer to my #1 response. I will prefer LDAP/AD, because that is the source of truth for all users across the Hadoop eco system. Also, pretty common in enterprises 5. In Hive and HBase, the grant calls are sent to Ranger, so from the user perspective, there is no change in admin behavior. What is the process in HAWQ? Do we create a design document and review it? We will have to do it for defining the REST APIs and documenting the request flow, etc. Thanks > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058849#comment-15058849 ] Don Bosco Durai commented on HAWQ-256: -- [~lei_chang], if you are going to be working on this. I can help you. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lei Chang > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
