[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867342#comment-15867342 ] Lili Ma commented on HAWQ-256: -- [~kdunn926] I re-looked at your input. 1) Why do they want to use Ranger? What are the scenario and use cases? Ranger provides the missing (and very important) functionality for synchronizing roles and groups from a identity management provider (like LDAP) into HAWQ. Without this capability, roles must be provisioned manually or something like pg-ldap-sync must be used, neither are very enterprise-friendly or "baked" solutions. Actually, I don't think Ranger provides the functionality to sync role/group information into HAWQ. It just sync those information to itself. We may still need to manage the role information in HAWQ to allow them to login. Or, a thorough solution is that HAWQ does not store any user information, but we may not do it now given there are some objects not managed by Ranger. Thoughts? > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf, > HAWQRangerSupportDesign_v0.2.pdf, HAWQRangerSupportDesign_v0.3.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859265#comment-15859265 ] Lili Ma commented on HAWQ-256: -- [~kdunn926] Thanks a lot! The information you provided is very helpful. About item 9, I wonder whether it is a little strange if we record the audit information for catalog table/owner check in Ranger side given that it is not managed by Ranger. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf, > HAWQRangerSupportDesign_v0.2.pdf, HAWQRangerSupportDesign_v0.3.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856444#comment-15856444 ] Kyle R Dunn commented on HAWQ-256: -- [~Lili Ma] - here's some input for you *1) Why do they want to use Ranger? What are the scenario and use cases?* Ranger provides the missing (and very important) functionality for synchronizing roles and groups from a identity management provider (like LDAP) into HAWQ. Without this capability, roles must be provisioned manually or something like pg-ldap-sync must be used, neither are very enterprise-friendly or "baked" solutions. *2) Which version of Ranger do they want to use? Is the version 0.6+ acceptable (shipped in HDP 2.5+) ?* I think any version is a good starting point, in my opinion, it is best we stay aligned with what is available the current GA HDP GA. *3) What are the specific HAWQ objects they want to manage in Ranger, for example, Database/Tablespace/Schema/Table/Sequence/Language/Function/Protocol? Is there anything else?* In my mind, support for schema, table, sequence, function, protocol are more important. Then prioritize database, tablespace - those seem to the more "advanced" usage (compared to the former) for most SQL on Hadoop installations I've seen. *4) What kind of tables do they want to manage? Heap (catalog) table, or data table on HDFS?* Data tables. My opinion, catalog should only be managed by a local superuser. *5) Do they want to restrict superuser privileges? If yes, what kind of privileges do they want to restrict, including catalog table or just the table on HDFS?* I've not seen this requirement, except with PL/x function creation / invocation. *6) Do they want to use Ambari to deploy HAWQ and Ranger?* Whenever possible, yes. *7) Do they have requirements for integration with user management tool such as LDAP?* Absolutely, this is the main motivator from my perspective. *8) Do they have a need to switch back and forth from Ranger? Say, setting Ranger on, and then setting off (using HAWQ native authorization)?* Hard to say here. If it is possible for HAWQ to reach some un-usable state as a result of have Ranger on, then yes, otherwise, it seems unlikely this would be a common activity. *9) Are they ok with the solution that we put system catalog/function/owner check in HAWQ? --- There are a lot of catalog information check(for example, pg_catalog, information_schema, etc) and system embedded function(for example, count, charne, etc) check in a simple SQL command such as “analyze” and “\d”, which will consume a lot of communication cost with Ranger if we process it in Ranger. Also, the embedded catalog/function may not include so much sensitive data. --- HAWQ does owner check under some cases. For example, only the owner who creates the table can drop it. Are the customer OK with that we keep the owner check in HAWQ?* This makes sense to me. Having admin functions only available via a local account but auditable by Ranger is likely a fair tradeoff here. *10) Are they ok with the solution that once Ranger is configured, we will forbid GRANT/REVOKE command in HAWQ?* This seems to be the correct behavior to avoid inconsistencies. *11) Are they ok with the solution that HAWQ handles the privileges check for drop table/create database?* This comes back to the third question - I think it makes sense, others may have a different opinion. *12) Are they ok with the solution that configuring an extra GUC in Ambari side for indicating Ranger on/off?* Not sure here. If Ranger thinks it's managing HAWQ, HAWQ should not be allowed to be "off" in Ambari. For the "disable Ranger" mode in HAWQ, maybe it should be command line only, as it would likely be only for troubleshooting / temporary usage. *13) Are they OK if we don’t provide High Availability with HAWQ Ranger Plugin Service (RPS) in the first (beta) release?* I think this is ok. Right now, it is not easy (or maybe even possible) to have high availability with HAWQ+LDAP, so this is still at parity with current functionality. Hope this helps. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf, > HAWQRangerSupportDesign_v0.2.pdf, HAWQRangerSupportDesign_v0.3.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454137#comment-15454137 ] Lili Ma commented on HAWQ-256: -- [~thebellhead] From technical view, we can restrict HAWQSuperUser privilege in Ranger definitely. But, if we restrict that, HAWQ superuser behavior changes. I think this needs careful discussion, and it's out of the scope of this JIRA. Right? Anyway, if everyone agrees to remove the superuser privileges, we can implement that function. Thanks > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf, > HAWQRangerSupportDesign_v0.2.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452587#comment-15452587 ] Alastair "Bell" Turner commented on HAWQ-256: - Thanks [~lilima] There are three gpadmin users and I think we could have a better discussion if we give them different names. 1. The gpadmin operating system user who own the HAWQ processes and the /hawq/* data on the local file system (OSGPAdmin). This user is not relevant to this issue. 2. The gpadmin Hadoop user (HAWQFileOwner). This is user identity used for HAWQ to access HDFS and owns the files created by HAWQ in HDFS. 3. The gpadmin user in HAWQ (HAWQSuperUser). This user is subject to very few, if any, restrictions on access to data held in HAWQ. For PXF there is also a user which accessed HDFS, Hive, etc on behalf of PXF queries. For consistency let's call this PXFFileOwner. My question about gpadmin access to data in Ranger managed tables is about access by HAWQSuperUser: If access to a table is managed by Ranger then the files containing that table's data in HDFS would be owned by HAWQFileOwner. This is not an issue as long as nobody can log in as HAWQFileOwner. The problem occurs when HAWQSuperUser can read any data in any table. This is currently the case for HAWQ internal tables. If PXFFileOwner has access to data then HAWQSuperUser would also be able to access it through external tables. If access on a database was managed by Ranger through this feature would HAWQSuperUser have access to read the data in that table? If only users authenticated through Ranger had access to data in the table it would not matter that HAWQFileOwner controlled the underlying file, HAWQ would be acting as a PEP and controlling access to the data. This is different from the scenario which I describe in HAWQ-1036 where policy is enforced by HDFS. Either approach would satisfy the requirement for HAWQSuperUser not to have access to the data. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf, > HAWQRangerSupportDesign_v0.2.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15451720#comment-15451720 ] Hubert Zhang commented on HAWQ-256: --- +1 for two stage authorization. Hawq ranger plugin(REST service) manages the access privilege of hawq object, include database, table, function, language and so on. While HDFS ranger plugin manages the access privilege of hdfs file. They are not conflicted with each other. User must first have the privilege to access hawq object(calculated in planner), next user also need to have the privilege to access the hdfs file. Currently, hawq use the admin user to create/append hdfs file, this is convenient for hawq user management. For example, user A own table t1, and if user A grant select and insert privilege of table t1 to user B, user B can directly access table t1, because on HDFS, the files of table t1 are created and accessed both by admin. But user-identity passing down will lead to table t1 is created by user A and user B cannot access file directly, unless add user B to user A's group, or change the file privilege. I do agree "user-identity passing down" is useful especially in hadoop eco, but when implementing it, pay attention to the problem I mentioned above.(Also this is beyond the discussion of issue256) > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf, > HAWQRangerSupportDesign_v0.2.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15451558#comment-15451558 ] Lili Ma commented on HAWQ-256: -- [~thebellhead], quit good questions! 1. In order for tools, syntax checking, etc to work everyone (the HAWQ public role) requires access to the catalog and some of the toolkit. Will Ranger-only access control apply only to user created tables, views and external tables? Yes, since the catalog tables and toolkits are shared and used by various users, Ranger-only access control just applies to user defined objects. But the objects include not only database, table and view, but also include function, language, schema, tablespace and protocol. You can find the detailed objects and privileges in the design doc. 2. If so - will gpadmin and any other HAWQ-defined roles not have access to the data in Ranger managed tables? Just as you mentioned, HAWQ uses gpadmin identity to create files on HDFS, say, when a specified userA creates a table in HAWQ, the HDFS files for the table are created by gpadmin instead of userA. Since Ranger lies in Hadoop eco-system, it usually needs to control both HAWQ and HDFS, I think we need assign gpadmin to the full privileges of hawq data file directory on HDFS in Ranger UI previously. About your concern about the superuser can see all the users' data, I think it's kind of like the "root" role in operation system? If the users have concerns about the DBA/Superuser's unlimited access, I totally agree with you about the solution of "passing down user-identifiy" for solving this problem :) 3. How would this be extended for the hcatalog virtual database in HAWQ? Could the Ranger permissions for the underlying store (for instance Hive) be read and enforced/reported at the HAWQ level? If HAWQ keeps the gpadmin for operating HDFS or external storage, I think we just need grant the privilege to superuser. But if we have implemented the user-identity passing down, say, the data files on HDFS for a table created by userA are owned by userA instead of gpadmin, in this way we need to double connect to Ranger, from HAWQ and HDFS respectively. I haven't include the underlying store privileges check into HAWQ side, that may need multiple code changes. I think keeping the privileges in the component is another choice. Your thoughts? Thanks Lili > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf, > HAWQRangerSupportDesign_v0.2.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15450026#comment-15450026 ] Alastair "Bell" Turner commented on HAWQ-256: - One more specific question: 3. How would this be extended for the hcatalog virtual database in HAWQ? Could the Ranger permissions for the underlying store (for instance Hive) be read and enforced/reported at the HAWQ level? > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf, > HAWQRangerSupportDesign_v0.2.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449919#comment-15449919 ] Alastair "Bell" Turner commented on HAWQ-256: - I've just read the solution design document and I'm not clear on a few points. As background to my specific questions below, I have two main concerns: 1. How does this feature address the requirement for administrators (gpadmin in this case) not to have access to plaintext data? Preventing DBA/sysad/superuser access to data is often cited as a reason for implementing TDE and other Hadoop environment security controls. For users who value this kind of control the unlimited access that gpadmin has to data in HAWQ is a problem. 2. How do the pre-requisites for Ranger integration (this feature) line up with the pre-requisites for impersonation ( HAWQ-1036 )? Ranger integration as described here implies HAWQ becoming a Policy Enforcement Point (PEP) for Ranger and the HAWQ daemon user being granted access to the underlying data stores. Impersonation, in contrast, does not require that the HAWQ (or PXF) daemon users have any access to the underlying data. The identity of the user connecting to HAWQ is passed through to HDFS or the TDE KMS and access is granted or denied on this basis. What is required of HAWQ in this case is not integration with the policy framework but a trusted “pipe" transferring credentials between the end user and the file system or PXF endpoint. Even though these approaches are quite different there are probably some common building blocks, like taking users from an external source rather than the HAWQ catalog. My specific questions are: 1. In order for tools, syntax checking, etc to work everyone (the HAWQ public role) requires access to the catalog and some of the toolkit. Will Ranger-only access control apply only to user created tables, views and external tables? 2. If so - will gpadmin and any other HAWQ-defined roles not have access to the data in Ranger managed tables? > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf, > HAWQRangerSupportDesign_v0.2.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438636#comment-15438636 ] Don Bosco Durai commented on HAWQ-256: -- Where is the latest API definition? From the JSON in the above comments, I think we have to make sure we have the following: 1. Groups for the user (either we send in the API or we can do the group lookup from the Ranger PDP (Policy Decision Point) Server itself. 2. Send IP of the client 3. Entire user query (if possible). I have seen users asking for entire query along with the audit record. We can truncate to a max predefined size. I have copied [~sneethiraj], [~madhan.neethiraj] and [~kulkabhay] to give their comments also. Thanks > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438560#comment-15438560 ] Don Bosco Durai commented on HAWQ-256: -- I agree. We can disable grant/revoke on the HAWQ side it Ranger is enabled. I feel, those using Ranger will prefer to manage the policies from one place. Also, even if we support GRANT/REVOKE from the HAWQ SQL command, we shouldn't expect that the behavior will be same when Ranger is enabled. E.g. Currently, when you give delegated admin privilege to any user (e.g. user 1) for the resource, then user1 can give any access to other users (e.g. user2) for that resource regardless what permission user1 has. This addresses use case where you don't want Admin to read/write, but be able to manage permissions for others. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438530#comment-15438530 ] Lili Ma commented on HAWQ-256: -- Agree on the Grant/Revoke should be disabled is ranger is enabled. In this way, the with grant option will not be considered. Another thing is the owner management. Besides normal ACL, HAWQ has a definition of owner. The owner of the object can do any operation. And for the owner part, "Grant parent role to member role" and "reassign" are two SQL commands for owner control. I think we should move owner control to Ranger, to enable a fully Ranger-centralized access control. Your thoughts? [~vVineet][~bosco] > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438531#comment-15438531 ] Lili Ma commented on HAWQ-256: -- [~bosco] Do you have any feedback on the API definition from Hortonworks side? Thanks > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436134#comment-15436134 ] Hubert Zhang commented on HAWQ-256: --- Agree with [~vVineet] If enable_ranger is ON in Hawq, we should disable grant/revoke/reassign statement on CLI. Here reassign is a Hawq statement which changes the ownership of database objects owned by a database role to other role. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436135#comment-15436135 ] Hubert Zhang commented on HAWQ-256: --- Agree with [~vVineet] If enable_ranger is ON in Hawq, we should disable grant/revoke/reassign statement on CLI. Here reassign is a Hawq statement which changes the ownership of database objects owned by a database role to other role. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434467#comment-15434467 ] Vineet Goel commented on HAWQ-256: -- I found this in the Hive documentation: "The ADMIN permission in Ranger is the equivalent to the WITH GRANT OPTION in SQL standard-based authorization. However, the ADMIN permission gives the grantee the ability to grant all permissions rather than just the permissions possessed by the grantor. With SQL standard-based authorization, the WITH GRANT OPTION applies only to permissions possessed by the grantor." This seems to suggest that "WITH GRANT OPTION" doesn't translate into same behavior at the Ranger level. This is understandable and acceptable I think. Ranger users and Component (Hive or HAWQ) users are likely two separate groups and they don't need to cross in their functions. This likely means, WITH GRANT OPTION on the CLI probably doesn't propagate into any Ranger policy updates and is ignored? Secondly, I'm late to this discussion, but it seems like [~bosco] was suggesting to design in such a way that "native component CLI commands" should not be encouraged, but rather, only Ranger UI/APIs should be used to set those policies (if Ranger authentication is switched ON in the component). If that's the case, I like that idea, to reduce design complexity. Hence, Authentication changes made with GRANT and REVOKE statements on component CLI must be disabled if Ranger authentication is switched ON. If Ranger is not in use, native component behavior remains unchanged. Users are expected not to flip back and forth between using Ranger and not using Ranger. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428488#comment-15428488 ] Don Bosco Durai commented on HAWQ-256: -- Starting Ranger 0.6 it also supports Kerberos. Before that it was user/password and two-way SSL. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425935#comment-15425935 ] Lin Wen commented on HAWQ-256: -- Hi, Don, Since HAWQ will call Ranger REST API to interact with Ranger, so what kind of security method is supported in REST API besides the common way? TLS, or SSL, or Kerberos? Thanks! > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422443#comment-15422443 ] Lili Ma commented on HAWQ-256: -- [~bosco][~vineetgoel][~lei_chang][~hubertzhang][~wenlin] Another thing we need to discuss is whether we support user send "GRANT" SQL besides setting policy in Ranger. If we also support Grant SQL, there is a minor difference between the "with grant option" of Grant SQL and what inside Ranger UI. We need to discuss it clear. Ranger has one button "Delegate Admin" when defining policy, this is different from what HAWQ grant SQL specifies. That button in Ranger means the Ranger internal user has the privileges to operate the given path/object and assign someone else the rights for the objects. That button has no influence on Ranger external user, say, HAWQ internal user. For example, if we add a policy specifying user A has the privileges to select a table T and click on the button and user A is Ranger internal user, then user A has the right to log into Ranger and assign the insert/select privileges for table T to user B. The grant SQL with grant option means that the to-be-granted user has the privilege to grant certain privileges to other users. If the grant privilege specifies just select, then user A can't grant insert privilege to user B. So this is minor different from what Ranger has already provided. If we allow grant/revoke SQL from HAWQ, we need to add "grant" as an action option to the resource. Action option means for each action, it has an attribute which indicates whether this action can be granted by the user. For example, admin grant two privileges: "grant select on t1 to u1" "grant insert on t1 to u1 with grant option" Then u1 grant privilege to u2 "grant select on t1 to u2" result: failed! grant insert on t1 to u2" result: succeed! As a result, u2 can insert on t1, but it cannot select on t1. Correspondingly, in Ranger, we have the following policies(* means with grant privilege): t1 u1 insert*select t1 u2 insert So the conclusion is that we need double the privileges for defining "with grant option" if we want to support Grant/Revoke SQL from HAWQ side. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422420#comment-15422420 ] Lili Ma commented on HAWQ-256: -- [~bosco], as [~hubertzhang] mentioned, HAWQ currently has requests for the privileges combination, either 'ALL', or 'ANY'. Do you think it's feasible to implement it inside Ranger REST API service? Certainly we can do it in HAWQ side, but there will be multiple communications with Ranger REST API, I'm afraid it may increase the time for checking privileges. So it's better to implement this judgement inside Ranger REST service. Your thoughts? > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422417#comment-15422417 ] Lili Ma commented on HAWQ-256: -- [~bosco] Thanks for your suggestion about default behaviors. I think Ranger 0.6 can help us resolve this problem. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422415#comment-15422415 ] Lili Ma commented on HAWQ-256: -- Per offline discussion, we think the integration between LDAP and HAWQ is out of the scope for HAWQ Integration with Apache Ranger. And since HAWQ already supports LDAP sync, we decide to put this to lower priority. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420978#comment-15420978 ] Don Bosco Durai commented on HAWQ-256: -- [~hubertzhang], fallback has been a very contentious topic. It causes ambiguity and confusion when determining which system really allowed. It is better to have only one source of truth. Also unlike HDFS and YARN, in HAWQ it will be two different systems. So if Ranger return "no" or "unknown", but HAWQ allowed it, then the audit records in Ranger will be wrong or incomplete. I would recommend, unless there is a very compelling reasons, we should support only one source of truth. I also feel, users will prefer consistent and uniform way of managing the policies. So they should be okay if we don't give fall back. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420958#comment-15420958 ] Don Bosco Durai commented on HAWQ-256: -- #1. The group "public" is virtual. It is similar to * or all. All users are part of public and you don't need to add users to public nor can you remove users from public. #2. In Ranger 0.6, deny can be used to explicitly deny users or groups. I would say we should target to support Ranger 0.6 and above. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420803#comment-15420803 ] Hubert Zhang commented on HAWQ-256: --- i review the Ranger doc for ranger policy evaluation flow, It said that if no allow-condition matches the request, the access result will be undertermined. In this case, most components will deny the access. However, components like HDFS and YARN fallback to their native ACL to determine the access So I think it's also reasonable for HAWQ to follow this way. No record leads to unknown return value, and let HAWQ to handle the behaviour. Also, this suggest only works in Ranger0.6+ > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420594#comment-15420594 ] Lili Ma commented on HAWQ-256: -- [~bosco] [~vVineet], Could you help confirm whether these APIs definitions are OK? Thanks :) > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419784#comment-15419784 ] Don Bosco Durai commented on HAWQ-256: -- + [~madhan.neethiraj] In the Ranger case, this might not be feasible. Because we go by "permissive" model, which means if there are no permissions, then it is "deny". So by absence of a permission, it will be difficult to determine "no privilege" v/s "deny". My suggestion would be for admins who want to set the default behaviors, can do it in Ranger itself. E.g. They can pick the resources (Database, table,etc) and give the desired permission to group "public". Which means, all users will at least get the permissions set in this policy. And they can have different defaults for different resources. It will be easy to manage these centrally, than trying to set the defaults in other config files or mechanism. This might be a better option, because now, the policies (including defaults) are in one place and it is easy to audit who set the default policies and how any end user got access to the resource. Would this be okay for the users? > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418598#comment-15418598 ] Hubert Zhang commented on HAWQ-256: --- [~bosco] [~vineetgoel] [~lilima][~wlin] We revisited the HAWQ aclcheck related code, and found that in HAWQ if there is no acl information stored in an object(database, table...) catalog, different objects will have different default behaviours to do acl check. But in our original REST-API design, the checkPrivileges API return bool(allow or deny) which lacks a state of unknown to do default acl check. Here 'unknown' means there is no record in ranger about this request. So we propose two ways to handle this issue. 1 Set the type of return value of checkPrivileges to integer, 0 for deny, 1 for allow, 2 for unknown 2 Add another REST-API, isPrivilegeExist(). Any suggestion for which solution is better ? > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[
https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411542#comment-15411542
]
Hubert Zhang commented on HAWQ-256:
---
Agree. We can use JSON array to represent it.
{
“requestor” : “u1”,
[
{
“resource” : {“TABLE”: “t1”, “DATABASE”: “db1”},
“privilege” : [“select”, "insert"]
},
{
“resource” : {“TABLE”: “t2”, “DATABASE”: “db1”},
“privilege” : [“select”]
}
]
}
> Integrate Security with Apache Ranger
> -
>
> Key: HAWQ-256
> URL: https://issues.apache.org/jira/browse/HAWQ-256
> Project: Apache HAWQ
> Issue Type: New Feature
> Components: PXF, Security
>Reporter: Michael Andre Pearce (IG)
>Assignee: Lili Ma
> Fix For: backlog
>
> Attachments: HAWQRangerSupportDesign.pdf
>
>
> Integrate security with Apache Ranger for a unified Hadoop security solution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411518#comment-15411518 ] Lili Ma commented on HAWQ-256: -- Another thing is for HAWQ sync with LDAP. By our investigation, HAWQ needs to run "create role" command for user registered in LDAP. [~teaandcoffee], do you think providing a script for this is acceptable? Or we need to create a backend process to do the user information sync automatically? cc [~wenlin] for this discussion too. Thanks > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[
https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411509#comment-15411509
]
Lili Ma commented on HAWQ-256:
--
[~bosco],
Will you confirm with folks in Ranger team for the API? Thanks
For the API interface, the grant/revoke part, we just design the API
corresponding to SQL grant/revoke syntax.
For the check privilege part, I think your advise is sensible. We can provide
the function for multiple resource check in one time. What about we change it
to following format, say, allowing one requestor to check multiple resources,
and for single resource, allowing multiple operations check?
{code}
{
“requestor” : “u1”,
{
{
“resource” : {“TABLE”: “t1”, “DATABASE”: “db1”},
“privilege” : “select”, "insert"
},
{
“resource” : {“TABLE”: “t2”, “DATABASE”: “db1”},
“privilege” : “select”
}
}
}
{code}
[~hubertzhang], your thoughts?
> Integrate Security with Apache Ranger
> -
>
> Key: HAWQ-256
> URL: https://issues.apache.org/jira/browse/HAWQ-256
> Project: Apache HAWQ
> Issue Type: New Feature
> Components: PXF, Security
>Reporter: Michael Andre Pearce (IG)
>Assignee: Lili Ma
> Fix For: backlog
>
> Attachments: HAWQRangerSupportDesign.pdf
>
>
> Integrate security with Apache Ranger for a unified Hadoop security solution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15409208#comment-15409208 ] Don Bosco Durai commented on HAWQ-256: -- [~lilima], this is looking good. Just couple of comments: 1. 4.3 is not needed. The policies will be cached in Ranger REST Service 2. Section 6: REST API interface - We should get reviewed with few folks in the Ranger team. There could be already something which we could use and extend 3. Section 6 - We should review the Hive model. SQL command might have multiple resources and different actions on them. E.g. Join, CTAS, etc. So single resource might not work. It needs to be be complex object > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > Attachments: HAWQRangerSupportDesign.pdf > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384859#comment-15384859 ] Don Bosco Durai commented on HAWQ-256: -- [~hubertzhang], you are correct. When Ranger is used for authorization, then anything internal/local to Hive (e.g. internal users or roles) are not used. The intention is to keep users and groups consistent across the entire eco-system. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384853#comment-15384853 ] Don Bosco Durai commented on HAWQ-256: -- I don't know the internals of HAWQ to comment much, but I feel, this is a broader discussion and we should probably create another JIRA to handle this. If HAWQ is replicating the users only to give access permission, then when using Ranger, it doesn't have to, because Ranger already syncs with AD/LDAP to manage the policies. So HAWQ only needs to authenticate the user and send the username during authorization call. But since I don't know the internals, so can't suggest much here. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382037#comment-15382037 ] Hubert Zhang commented on HAWQ-256: --- [~bosco] [~Lili Ma] I skim the Authorizer code in Hive and Ranger. I found that the behaviour of RangerHiveAuthorizer is limited. In detail, RangerHiveAuthorizer is subclass of HiveAuthorizer(abstract interface), but it only implemented a subset of member functions. I conclude them as follows: Implemented in ranger: grantPrivileges revokePrivileges checkPrivileges applyRowFilterAndColumnMasking needTransform filterListCmdObjects Not implemented in ranger: createRole dropRole getPrincipalGrantInfoForRole getRoleGrantInfoForPrincipal grantRole revokeRole getAllRoles showPrivileges getCurrentRoleNames setCurrentRole applyAuthorizationConfigPolicy getHiveAuthorizationTranslator So could I get a conclusion that when user config Hive with Ranger enabled, uses cannot create or drop roles in Hive? > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381878#comment-15381878 ] Michael Andre Pearce (IG) commented on HAWQ-256: Hi Guys, As original person to raise this and an end user of both systems. We use AD/LDAP for central user creation and use role groups in AD for user security. Atm meant we have to manually create / sync up users with the DB, we end up having to rely on third party scripts :( urg. The intent was that actually we could do away with bits like: https://github.com/larskanis/pg-ldap-sync We use Ranger for a central way to auth and apply policies to all components in Hadoop and use Kerberos to auth the user. The idea or intent in the original ticket was for the user creation and ability to secure/control tables, schema, dbs via group policies via Ranger very much like the way it works with Hive. The discussion seems to be going very much away from this, as such would end up not making this feature useful for us. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381729#comment-15381729 ] Don Bosco Durai commented on HAWQ-256: -- [~hubertzhang], thanks. The JDBC call should suffice for lookup. Ranger only supports one set of users and it is generally what is supported by Hadoop. The source is either AD/LDAP or linux users. In the case of Hive, if Ranger is used, then Hive's internal users or roles are not used, instead Hadoop Common is used to get users and groups. This keeps the users and groups consistent across all components. I would prefer the same behavior for HAWQ. But for any reason HAWQ needs to support it's own users/groups, then they need to be populated in Ranger also. Since Ranger doesn't have namespace for users, it can't do conflict resolution. So we will have to do what you suggest. It would be good if we defer it and see if users really needs it? Because, users using Ranger prefer uniform user and groups. Thanks > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381250#comment-15381250 ] Don Bosco Durai commented on HAWQ-256: -- You got these two correct. We need to add one more to the list. In RangerAdmin UI, when you create a policy, we do auto-suggest by doing a lookup in the databases and schemas in the component. So on the Ranger Admin side, we will need to write the code to query HAWQ. This is generally the existing APIs provided by the components, so in the case HAWQ it would be JDBC or other any other API supported by HAWQ. We just need to track this for completeness purpose and I don't anticipate any work from the HAWQ side. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378792#comment-15378792 ] Lili Ma commented on HAWQ-256: -- [~bosco] Thanks. Things are getting more clear now. So for the interaction between HAWQ and Ranger, I think there are mainly two parts: 1. Set policy. When HAWQ users invoke GRANT SQL in HAWQ, need pass that command to Ranger to set the policy. 2.Check Authorization. When HAWQ user want to operate on some objects, need contact Ranger to check whether the user has the privilege. Both these two parts of interaction rely on Ranger Plugin. What we need do next is detailing down the interface for interaction and designing the HAWQ own side implementation. Please suggest if I missed something. Thanks > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378322#comment-15378322 ] Don Bosco Durai commented on HAWQ-256: -- 1. The "Add New User" in Ranger is just to add user in the Ranger DB. The users and groups in Ranger are used to help create policies in Ranger. It is not used as source of truth by the component for users or groups. The main reason being, Ranger doesn't do authentication. So you need to rely on AD/LDAP or use local user/password. 2. In the Ranger integration, the policies are stored in the Ranger DB. Ranger provides UI and REST APIs to create the policies. In Hive and HBase, the grant from their CLI calls our plugin running within their process, which in turn calls Ranger REST API. In the case of HAWQ, the C++ client might make the REST API to the proxy Ranger Server to set the policies. 3. The model we suggest is to abstract the authorization layer. The default behavior is the component natively implementation. And those working in a bigger eco-system can alternatively use Ranger or anyone implementing the component's interface. So for native implementation, technically nothing should change. You still will be saving the ACLs the way you are currently storing and using it. When the user choose Ranger as the option, the policies will be stored in Ranger DB in Ranger format and the Ranger implementation will pull the policies and enforce it. So any ACLs stored in the component native storage will not be used. 5. Same as #2. In addition to Ranger UI and REST API, users can also set policies via native component CLI commands. This is primarily for backward compatibility. However, since Ranger support additional conditions, generally it is not possible to set these conditions via native CLI grant commands. Looking forward for the design document. Thanks > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376472#comment-15376472 ] Lili Ma commented on HAWQ-256: -- [~bosco] Thanks for your answer :) 1. Yes, it's good for Ranger to import user list from component. Why I expose this question is that I noticed that Ranger has provided a function "Add New User" under tab "Settings/Users/Groups". Does it mean Ranger also supports creating user in Ranger itself? 2. Grant privilege from just one side is relatively easy and clear. What we need to discuss is which side we allow granting privilege, HAWQ, or Ranger? As you said, HAWQ side is a good choice since there's no change in admin behavior. 3. I also thinks it would be simple if we don't consider Ranger down or Ranger not exist problem. What about the scenarios that user don't intend to install Ranger? Are users are all fine with Ranger? Currently the ACL information is stored in HAWQ catalog. Shall we remove the catalog information if we provide Ranger support? 4. Yes, LDAP/AD is a potential good solution for us :) 5. So In Hive and HBase, the grant operations are all done in the database side instead of Ranger side. Right? In this page it seems that Ranger admin console also supports creating a new policy from UI? Please correct me if my understanding is wrong. Actually, we are investigating and aiming at drafting a design doc. Will attach the design doc to this JIRA once done. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376324#comment-15376324 ] Don Bosco Durai commented on HAWQ-256: -- [~lilima], thanks for listing down the questions. 1. Ranger uses the user from the component. If it is from LDAP/AD or Linux user, then it gets it from there. But if we are creating (only) in HAWQ, then it needs to be imported into Ranger also. Ranger gives different ways of loading users into it's database. API, file import, LDAP/AD, etc. 2. We shouldn't mix and match. In the Hive, HBase, etc, it is either the component or Ranger. Not both. This will be easy for the users to understand and manage 3. If Ranger is down, we should consider as failure. I don't think we should over-engineer this part. We should make the REST API server HA, so minimize the issue of Ranger down case. 4. Refer to my #1 response. I will prefer LDAP/AD, because that is the source of truth for all users across the Hadoop eco system. Also, pretty common in enterprises 5. In Hive and HBase, the grant calls are sent to Ranger, so from the user perspective, there is no change in admin behavior. What is the process in HAWQ? Do we create a design document and review it? We will have to do it for defining the REST APIs and documenting the request flow, etc. Thanks > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376246#comment-15376246 ] Hubert Zhang commented on HAWQ-256: --- Agreed with [~Lili Ma] . We need to look into what's the detailed behaviour of Hive/Hbase interacting with Ranger. I'll add some detailed information after investigation. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lili Ma > Fix For: backlog > > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059737#comment-15059737 ] Lei Chang commented on HAWQ-256: Nice. A proposal helps a lot on starting the technical discussion. And it would be very nice to have a ranger plugins in c or c++ to make it work natively in HAWQ. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lei Chang > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058761#comment-15058761 ] Goden Yao commented on HAWQ-256: Added linked Ranger JIRA > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lei Chang > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger
[ https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058849#comment-15058849 ] Don Bosco Durai commented on HAWQ-256: -- [~lei_chang], if you are going to be working on this. I can help you. > Integrate Security with Apache Ranger > - > > Key: HAWQ-256 > URL: https://issues.apache.org/jira/browse/HAWQ-256 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF, Security >Reporter: Michael Andre Pearce (IG) >Assignee: Lei Chang > > Integrate security with Apache Ranger for a unified Hadoop security solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
