[jira] [Commented] (RANGER-1810) Ranger supports plugin to enable, monitor and manage apache Sqoop2

2017-10-20 Thread Don Bosco Durai (JIRA)

[ 
https://issues.apache.org/jira/browse/RANGER-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16213752#comment-16213752
 ] 

Don Bosco Durai commented on RANGER-1810:
-

[~zhangqiang2] I am excited to see the support for Sqoop in Ranger. Thanks for 
working on it. [~coheigea], thanks for all your review and feedbacks :-)

> Ranger supports plugin to enable, monitor and manage apache Sqoop2
> --
>
> Key: RANGER-1810
> URL: https://issues.apache.org/jira/browse/RANGER-1810
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>Affects Versions: 1.0.0, master
>Reporter: Qiang Zhang
>Assignee: Qiang Zhang
>  Labels: newbie, patch
> Attachments: 
> 0001-RANGER-1810-Ranger-supports-plugin-to-enable-monitor.patch, 
> 1_SqoopServiceManager.jpg, 2_EditSqoopService.jpg, 3_ListSqoopPolicies.jpg, 
> 4_EditSqoopPolicy.jpg, 5_SqoopAuditLog.jpg, 6_SqoopPlugins.jpg
>
>
> Apache Sqoop is a tool designed for efficiently transferring bulk data 
> between Apache Hadoop and structured datastores such as relational databases. 
> You can use Sqoop to import data from external structured datastores into 
> Hadoop Distributed File System or related systems like Hive and HBase. 
> Conversely, Sqoop can be used to extract data from Hadoop and export it to 
> external structured datastores such as relational databases and enterprise 
> data warehouses.It successfully graduated from the Incubator in March of 2012 
> and is now a Top-Level Apache project.  
> The Ranger will further expand the influence in the hadoop ecosystem if it 
> supports sqoop authorization. So we should develop sqoop plugin to enable, 
> monitor and manage apache Sqoop2.
> Our test specialists have rigorously tested this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (RANGER-1847) Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN

2017-10-20 Thread Don Bosco Durai (JIRA)

[ 
https://issues.apache.org/jira/browse/RANGER-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16213751#comment-16213751
 ] 

Don Bosco Durai commented on RANGER-1847:
-

[~rollantz], good it seems you were able to resolve it. Will you be able to 
document your experience? This will help others in the future. If you can, I 
can provide write access to Ranger Wiki or if you can blog somewhere, we can 
put that link in our Ranger Wiki.

Thanks

Bosco

> Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN
> -
>
> Key: RANGER-1847
> URL: https://issues.apache.org/jira/browse/RANGER-1847
> Project: Ranger
>  Issue Type: Test
>  Components: plugins
>Affects Versions: 0.6.3, 0.7.1
> Environment: ubuntu stand-alone hobby environment
>Reporter: Ronald van de Kuil
>Priority: Minor
>
> I am such a NOOB hobby-ing away. And I like it. ;)
> I figured I would give it a try to setup Kafka to use the 
> sasl.enabled.mechanisms of type PLAIN with ranger to do the authorisation and 
> the auditing (instead of GSSAPI).
> I got it to work pretty far. KafkaServer gets into state SaslAuthenticated 
> with Zookeeper. 
> Next it loads the ranger kafka plugin. Then the RangerKafkaAuthorizer 
> complains about Kerberos. 
> I then updated the CLASSPATH and it complains about something else.
> I am not sure how to classify this issue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (RANGER-1851) Enhance Ranger Hive Plugin to support authorization for KILL QUERY command

2017-10-20 Thread Don Bosco Durai (JIRA)

[ 
https://issues.apache.org/jira/browse/RANGER-1851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16213749#comment-16213749
 ] 

Don Bosco Durai commented on RANGER-1851:
-

I feel this is a good thing. We should consider this as a design pattern for 
our other services where we need actions at non-resource level.

Seems we have started having multiple cases, where the resources are at the 
same level (e.g. database/URL in Hive and now Service). Can we associate which 
"actions" are applicable at each top level? This will give a good experience 
for the users. This doesn't have to be part of this JIRA.


> Enhance Ranger Hive Plugin to support authorization for KILL QUERY command
> --
>
> Key: RANGER-1851
> URL: https://issues.apache.org/jira/browse/RANGER-1851
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: master, 0.7.1
>Reporter: Ramesh Mani
>Assignee: Ramesh Mani
>Priority: Critical
>
> With the HIVE-17483 JIRA,  Hive has introduced a way to kill query  and 
> in hive its a privileged  action for Hive Admin Role. In order for the Ranger 
> Hive Authorizer to support authorization, we need to enhance the ranger hive 
> authorizer. Current Hive implementation is to Kill Query in a HiveService 
> which can be LLAP / HIVESERVER2 , later these HIVE SERVICEs can be grouped 
> into NAME SPACEs and kill query can be run against them. When 
> HiveServer2/LLAP Ranger Plugin sends the request to Ranger for Authorization, 
> it will be sending the HIVE SERVICE in the context with the COMMAND that is 
> executed.  
> With all the details proposal is to have 
> 1) In Ranger Hive Service Definition, we will have a new Resource "Hive 
> Service" to authorize.
> 2) In Ranger Hive Permission Model, we will have a new Permission "Service 
> Admin" to group Kill Query operation.
> - "Service Admin"  permission will enable hive ranger plugin to isolate 
> various admin operations in this case "Kill Query" and in future if hive 
> introduces other operations which are done at "HIVE SERVICE level" , group 
> them under this and authorize.
>- "Service Admin" won't be able to do  DATABASE / TABLE / COLUMN 
> operations as this will all be taken care by the existing 
> DATABASE/TABLE/COLUMN level permission model.
> [~madhan.neethiraj] [~vperiasamy][~thejas][~bosco][~sneethiraj]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (RANGER-1852) some groups missed to be sync if they are syncd from openldap If deltasync is enabled

2017-10-20 Thread Deepak Sharma (JIRA)
Deepak Sharma created RANGER-1852:
-

 Summary: some groups missed to be sync if they are syncd from 
openldap If deltasync is enabled
 Key: RANGER-1852
 URL: https://issues.apache.org/jira/browse/RANGER-1852
 Project: Ranger
  Issue Type: Bug
  Components: Ranger
Affects Versions: 0.7.2
Reporter: Deepak Sharma


some groups missed to be sync if they are syncd from openldap in a scenario 
when users are added to the ldap at the time when usersync is happening or if 
it is in progress or within a minute of usersync cycle.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


Re: Review Request 63142: Ranger admin support hdfs HA configuration when creating hdfs service.

2017-10-20 Thread Alejandro Fernandez

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63142/#review188855
---


Ship it!




Ship It!

- Alejandro Fernandez


On Oct. 20, 2017, 6:18 a.m., Qiang Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63142/
> ---
> 
> (Updated Oct. 20, 2017, 6:18 a.m.)
> 
> 
> Review request for ranger, Alok Lal, Ankita Sinha, Don Bosco Durai, Colm O 
> hEigeartaigh, Gautam Borad, Madhan Neethiraj, pengjianhua, Ramesh Mani, 
> Selvamohan Neethiraj, Velmurugan Periasamy, and Qiang Zhang.
> 
> 
> Bugs: RANGER-1844
> https://issues.apache.org/jira/browse/RANGER-1844
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> In ranger admin, when creating a hdfs service, if hdfs cluster is in HA 
> mode,you have to config a lot of configs in hdfs_dev,such as:
> Namenode URL *=hdfs://hdfscluster
> ===Add New Configurations===
> dfs.nameservices=hdfscluster
> dfs.client.failover.proxy.provider.hdfscluster=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> dfs.ha.namenodes.hdfscluster=nn1,nn2
> dfs.namenode.rpc-address.hdfscluster.nn1=hdfs://10.43.159.240:9000
> dfs.namenode.rpc-address.hdfscluster.nn2=hdfs://10.43.159.245:9000
> ===End of add New Configurations===
> And other big data components such as hbase,hive and so on can support HA 
> without config
> lots of "Add New Configurations",it is easy to config a url. like zk queue 
> configuration in hbase ,like jdbc url in hive. In hdfs service, only need to 
> config "fs.default.name" :
> Namenode URL *=hdfs://hdfscluster ?old? 
> Namenode URL *=hdfs://dap230-183:9000,hdfs://dap229-183:9000 ?new?
> 
> 
> Diffs
> -
> 
>   
> hdfs-agent/src/main/java/org/apache/ranger/services/hdfs/client/HdfsClient.java
>  c252213f 
>   security-admin/src/main/webapp/scripts/models/BackboneFormDataType.js 
> 3f8697e7 
>   security-admin/src/main/webapp/scripts/modules/globalize/message/en.js 
> 811db0c9 
> 
> 
> Diff: https://reviews.apache.org/r/63142/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Qiang Zhang
> 
>



[jira] [Updated] (RANGER-1851) Enhance Ranger Hive Plugin to support authorization for KILL QUERY command

2017-10-20 Thread Ramesh Mani (JIRA)

 [ 
https://issues.apache.org/jira/browse/RANGER-1851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramesh Mani updated RANGER-1851:

Description: 
With the HIVE-17483 JIRA,  Hive has introduced a way to kill query  and in 
hive its a privileged  action for Hive Admin Role. In order for the Ranger Hive 
Authorizer to support authorization, we need to enhance the ranger hive 
authorizer. Current Hive implementation is to Kill Query in a HiveService which 
can be LLAP / HIVESERVER2 , later these HIVE SERVICEs can be grouped into NAME 
SPACEs and kill query can be run against them. When HiveServer2/LLAP Ranger 
Plugin sends the request to Ranger for Authorization, it will be sending the 
HIVE SERVICE in the context with the COMMAND that is executed.  
With all the details proposal is to have 
1) In Ranger Hive Service Definition, we will have a new Resource "Hive 
Service" to authorize.
2) In Ranger Hive Permission Model, we will have a new Permission "Service 
Admin" to group Kill Query operation.
- "Service Admin"  permission will enable hive ranger plugin to isolate 
various admin operations in this case "Kill Query" and in future if hive 
introduces other operations which are done at "HIVE SERVICE level" , group them 
under this and authorize.
   - "Service Admin" won't be able to do  DATABASE / TABLE / COLUMN operations 
as this will all be taken care by the existing DATABASE/TABLE/COLUMN level 
permission model.

[~madhan.neethiraj] [~vperiasamy][~thejas][~bosco][~sneethiraj]

  was:
With the HIVE-17483 JIRA,  Hive has introduced a way to kill query  and in 
hive its a privileged  action for Hive Admin Role. In order for the Ranger Hive 
Authorizer to support authorization, we need to enhance the ranger hive 
authorizer. Current Hive implementation is to Kill Query in a HiveService which 
can be LLAP / HIVESERVER2 , later these HIVE SERVICEs can be grouped into NAME 
SPACEs and kill query can be run against them. When HiveServer2/LLAP Ranger 
Plugin sends the request to Ranger for Authorization, it will be sending the 
HIVE SERVICE in the context with the COMMAND that is executed.  
With all the details proposal is to have 
1) In Ranger Hive Service Definition, we will have a new Resource "Hive 
Service" to authorize.
2) In Ranger Hive Permission Model, we will have a new Permission "Service 
Admin" to group Kill Query operation.
- "Service Admin"  permission will enable hive ranger plugin to isolate 
various admin operations in this case "Kill Query" and in future if hive 
introduces other operations which are done at "HIVE SERVICE level" , group them 
under this and authorize.
   - "Service Admin" won't be able to do  DATABASE / TABLE / COLUMN operations 
as this will all be taken care by the existing DATABASE/TABLE/COLUMN level 
permission model.


> Enhance Ranger Hive Plugin to support authorization for KILL QUERY command
> --
>
> Key: RANGER-1851
> URL: https://issues.apache.org/jira/browse/RANGER-1851
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: master, 0.7.1
>Reporter: Ramesh Mani
>Assignee: Ramesh Mani
>Priority: Critical
>
> With the HIVE-17483 JIRA,  Hive has introduced a way to kill query  and 
> in hive its a privileged  action for Hive Admin Role. In order for the Ranger 
> Hive Authorizer to support authorization, we need to enhance the ranger hive 
> authorizer. Current Hive implementation is to Kill Query in a HiveService 
> which can be LLAP / HIVESERVER2 , later these HIVE SERVICEs can be grouped 
> into NAME SPACEs and kill query can be run against them. When 
> HiveServer2/LLAP Ranger Plugin sends the request to Ranger for Authorization, 
> it will be sending the HIVE SERVICE in the context with the COMMAND that is 
> executed.  
> With all the details proposal is to have 
> 1) In Ranger Hive Service Definition, we will have a new Resource "Hive 
> Service" to authorize.
> 2) In Ranger Hive Permission Model, we will have a new Permission "Service 
> Admin" to group Kill Query operation.
> - "Service Admin"  permission will enable hive ranger plugin to isolate 
> various admin operations in this case "Kill Query" and in future if hive 
> introduces other operations which are done at "HIVE SERVICE level" , group 
> them under this and authorize.
>- "Service Admin" won't be able to do  DATABASE / TABLE / COLUMN 
> operations as this will all be taken care by the existing 
> DATABASE/TABLE/COLUMN level permission model.
> [~madhan.neethiraj] [~vperiasamy][~thejas][~bosco][~sneethiraj]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (RANGER-1851) Enhance Ranger Hive Plugin to support authorization for KILL QUERY command

2017-10-20 Thread Ramesh Mani (JIRA)

 [ 
https://issues.apache.org/jira/browse/RANGER-1851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramesh Mani updated RANGER-1851:

Description: 
With the HIVE-17483 JIRA,  Hive has introduced a way to kill query  and in 
hive its a privileged  action for Hive Admin Role. In order for the Ranger Hive 
Authorizer to support authorization, we need to enhance the ranger hive 
authorizer. Current Hive implementation is to Kill Query in a HiveService which 
can be LLAP / HIVESERVER2 , later these HIVE SERVICEs can be grouped into NAME 
SPACEs and kill query can be run against them. When HiveServer2/LLAP Ranger 
Plugin sends the request to Ranger for Authorization, it will be sending the 
HIVE SERVICE in the context with the COMMAND that is executed.  
With all the details proposal is to have 
1) In Ranger Hive Service Definition, we will have a new Resource "Hive 
Service" to authorize.
2) In Ranger Hive Permission Model, we will have a new Permission "Service 
Admin" to group Kill Query operation.
- "Service Admin"  permission will enable hive ranger plugin to isolate 
various admin operations in this case "Kill Query" and in future if hive 
introduces other operations which are done at "HIVE SERVICE level" , group them 
under this and authorize.
   - "Service Admin" won't be able to do  DATABASE / TABLE / COLUMN operations 
as this will all be taken care by the existing DATABASE/TABLE/COLUMN level 
permission model.

  was:
With the HIVE-17483 JIRA,  Hive has introduced a way to kill query  and in 
hive its a privileged  action for Hive Admin Role. In order for the Ranger Hive 
Authorizer to support authorization, we need to enhance the ranger hive 
authorizer. Current Hive implementation is to Kill Query in a HiveService which 
can be LLAP / HIVESERVER2 , later these HIVE SERVICEs can be grouped into NAME 
SPACEs and kill query can be run against them. When HiveServer2/LLAP Ranger 
Plugin sends the request to Ranger for Authorization, it will be sending the 
HIVE SERVICE in the context with they COMMAND.  
With all the details proposal is to have 
1) In Ranger Hive Service Definition, we will have a new Resource "Hive 
Service" to authorize.
2) In Ranger Hive Permission Model, we will have a new Permission "Service 
Admin" to group Kill Query operation.
- "Service Admin"  permission will enable hive ranger plugin to isolate 
various admin operations in this case "Kill Query" and in future if hive 
introduces other operations which are done at "HIVE SERVICE level" , group them 
under this and authorize.
   - "Service Admin" won't be able to do  DATABASE / TABLE / COLUMN operations 
as this will all be taken care by the existing DATABASE/TABLE/COLUMN level 
permission model.


> Enhance Ranger Hive Plugin to support authorization for KILL QUERY command
> --
>
> Key: RANGER-1851
> URL: https://issues.apache.org/jira/browse/RANGER-1851
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: master, 0.7.1
>Reporter: Ramesh Mani
>Assignee: Ramesh Mani
>Priority: Critical
>
> With the HIVE-17483 JIRA,  Hive has introduced a way to kill query  and 
> in hive its a privileged  action for Hive Admin Role. In order for the Ranger 
> Hive Authorizer to support authorization, we need to enhance the ranger hive 
> authorizer. Current Hive implementation is to Kill Query in a HiveService 
> which can be LLAP / HIVESERVER2 , later these HIVE SERVICEs can be grouped 
> into NAME SPACEs and kill query can be run against them. When 
> HiveServer2/LLAP Ranger Plugin sends the request to Ranger for Authorization, 
> it will be sending the HIVE SERVICE in the context with the COMMAND that is 
> executed.  
> With all the details proposal is to have 
> 1) In Ranger Hive Service Definition, we will have a new Resource "Hive 
> Service" to authorize.
> 2) In Ranger Hive Permission Model, we will have a new Permission "Service 
> Admin" to group Kill Query operation.
> - "Service Admin"  permission will enable hive ranger plugin to isolate 
> various admin operations in this case "Kill Query" and in future if hive 
> introduces other operations which are done at "HIVE SERVICE level" , group 
> them under this and authorize.
>- "Service Admin" won't be able to do  DATABASE / TABLE / COLUMN 
> operations as this will all be taken care by the existing 
> DATABASE/TABLE/COLUMN level permission model.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (RANGER-1851) Enhance Ranger Hive Plugin to support authorization for KILL QUERY command

2017-10-20 Thread Ramesh Mani (JIRA)
Ramesh Mani created RANGER-1851:
---

 Summary: Enhance Ranger Hive Plugin to support authorization for 
KILL QUERY command
 Key: RANGER-1851
 URL: https://issues.apache.org/jira/browse/RANGER-1851
 Project: Ranger
  Issue Type: Bug
  Components: Ranger
Affects Versions: 0.7.1, master
Reporter: Ramesh Mani
Assignee: Ramesh Mani
Priority: Critical


With the HIVE-17483 JIRA,  Hive has introduced a way to kill query  and in 
hive its a privileged  action for Hive Admin Role. In order for the Ranger Hive 
Authorizer to support authorization, we need to enhance the ranger hive 
authorizer. Current Hive implementation is to Kill Query in a HiveService which 
can be LLAP / HIVESERVER2 , later these HIVE SERVICEs can be grouped into NAME 
SPACEs and kill query can be run against them. When HiveServer2/LLAP Ranger 
Plugin sends the request to Ranger for Authorization, it will be sending the 
HIVE SERVICE in the context with they COMMAND.  
With all the details proposal is to have 
1) In Ranger Hive Service Definition, we will have a new Resource "Hive 
Service" to authorize.
2) In Ranger Hive Permission Model, we will have a new Permission "Service 
Admin" to group Kill Query operation.
- "Service Admin"  permission will enable hive ranger plugin to isolate 
various admin operations in this case "Kill Query" and in future if hive 
introduces other operations which are done at "HIVE SERVICE level" , group them 
under this and authorize.
   - "Service Admin" won't be able to do  DATABASE / TABLE / COLUMN operations 
as this will all be taken care by the existing DATABASE/TABLE/COLUMN level 
permission model.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (RANGER-1850) Impersonation/proxy user support for gaiandb ranger plugin

2017-10-20 Thread Nigel Jones (JIRA)
Nigel Jones created RANGER-1850:
---

 Summary: Impersonation/proxy user support for gaiandb ranger plugin
 Key: RANGER-1850
 URL: https://issues.apache.org/jira/browse/RANGER-1850
 Project: Ranger
  Issue Type: Sub-task
  Components: plugins
Reporter: Nigel Jones


Applications/users could connect to gaianDB using their own authentication 
information - for example userid/password in the simple case. Here the ranger 
plugin will use that id for policy checks.

However in a multi tiered architecture a service id (aka non personal account) 
may be used, and somehow the user to be impersonated is passed via an 
additional property. This has a number of implications to the system 
configuration, derby/gaiandb configuration & the plugin implementation. 

Opening this Jira as a placeholder and will add a document soon (++days) on the 
same to capture some of the discussion around this area in recent days.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


Review Request 63182: RANGER-1849 - Remove PasswordGenerator

2017-10-20 Thread Colm O hEigeartaigh

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63182/
---

Review request for ranger.


Bugs: RANGER-1849
https://issues.apache.org/jira/browse/RANGER-1849


Repository: ranger


Description
---

PasswordGenerator is not used in any of the scripts and should be removed.


Diffs
-

  
agents-installer/src/main/java/org/apache/ranger/utils/install/PasswordGenerator.java
 a829957a 


Diff: https://reviews.apache.org/r/63182/diff/1/


Testing
---


Thanks,

Colm O hEigeartaigh



[jira] [Updated] (RANGER-1849) Remove PasswordGenerator

2017-10-20 Thread Colm O hEigeartaigh (JIRA)

 [ 
https://issues.apache.org/jira/browse/RANGER-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colm O hEigeartaigh updated RANGER-1849:

Attachment: 0001-RANGER-1849-Remove-PasswordGenerator.patch

> Remove PasswordGenerator
> 
>
> Key: RANGER-1849
> URL: https://issues.apache.org/jira/browse/RANGER-1849
> Project: Ranger
>  Issue Type: Improvement
>  Components: plugins
>Reporter: Colm O hEigeartaigh
>Assignee: Colm O hEigeartaigh
>Priority: Trivial
> Fix For: 1.0.0
>
> Attachments: 0001-RANGER-1849-Remove-PasswordGenerator.patch
>
>
> PasswordGenerator is not used in any of the scripts and should be removed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (RANGER-1849) Remove PasswordGenerator

2017-10-20 Thread Colm O hEigeartaigh (JIRA)
Colm O hEigeartaigh created RANGER-1849:
---

 Summary: Remove PasswordGenerator
 Key: RANGER-1849
 URL: https://issues.apache.org/jira/browse/RANGER-1849
 Project: Ranger
  Issue Type: Improvement
  Components: plugins
Reporter: Colm O hEigeartaigh
Assignee: Colm O hEigeartaigh
Priority: Trivial
 Fix For: 1.0.0


PasswordGenerator is not used in any of the scripts and should be removed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


Re: Review Request 62710: RANGER-1810:Ranger supports plugin to enable, monitor and manage apache Sqoop2

2017-10-20 Thread Colm O hEigeartaigh

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62710/#review188844
---



Looks good thanks. Just three more changes:

a) Add a "hover over" hint for "Sqoop URL" in the Admin Console (e.g. should be 
"http://localhost:12000";)
b) Remove "password" from the UI as it's not used
c) Remove "password" from SqoopClient as it's not used.

- Colm O hEigeartaigh


On Oct. 20, 2017, 7:16 a.m., Qiang Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62710/
> ---
> 
> (Updated Oct. 20, 2017, 7:16 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Don Bosco Durai, Colm O 
> hEigeartaigh, Gautam Borad, Madhan Neethiraj, pengjianhua, Ramesh Mani, 
> Selvamohan Neethiraj, sam  rome, Venkat Ranganathan, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-1810
> https://issues.apache.org/jira/browse/RANGER-1810
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Apache Sqoop is a tool designed for efficiently transferring bulk data 
> between Apache Hadoop and structured datastores such as relational databases. 
> You can use Sqoop to import data from external structured datastores into 
> Hadoop Distributed File System or related systems like Hive and HBase. 
> Conversely, Sqoop can be used to extract data from Hadoop and export it to 
> external structured datastores such as relational databases and enterprise 
> data warehouses.It successfully graduated from the Incubator in March of 2012 
> and is now a Top-Level Apache project.
> The Ranger will further expand the influence in the hadoop ecosystem if it 
> supports sqoop authorization. So we should develop sqoop plugin to enable, 
> monitor and manage apache Sqoop2.
> 
> Our test specialists have rigorously tested this feature.
> 
> 
> Diffs
> -
> 
>   agents-common/scripts/enable-agent.sh d31a264 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/store/EmbeddedServiceDefsUtil.java
>  9463ab8 
>   agents-common/src/main/resources/service-defs/ranger-servicedef-sqoop.json 
> PRE-CREATION 
>   plugin-sqoop/.gitignore PRE-CREATION 
>   plugin-sqoop/conf/ranger-policymgr-ssl-changes.cfg PRE-CREATION 
>   plugin-sqoop/conf/ranger-policymgr-ssl.xml PRE-CREATION 
>   plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg PRE-CREATION 
>   plugin-sqoop/conf/ranger-sqoop-audit.xml PRE-CREATION 
>   plugin-sqoop/conf/ranger-sqoop-security-changes.cfg PRE-CREATION 
>   plugin-sqoop/conf/ranger-sqoop-security.xml PRE-CREATION 
>   plugin-sqoop/pom.xml PRE-CREATION 
>   plugin-sqoop/scripts/install.properties PRE-CREATION 
>   
> plugin-sqoop/src/main/java/org/apache/ranger/authorization/sqoop/authorizer/RangerSqoopAuthorizer.java
>  PRE-CREATION 
>   
> plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/RangerServiceSqoop.java
>  PRE-CREATION 
>   
> plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/client/SqoopClient.java
>  PRE-CREATION 
>   
> plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/client/SqoopResourceMgr.java
>  PRE-CREATION 
>   
> plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/client/json/model/SqoopConnectorResponse.java
>  PRE-CREATION 
>   
> plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/client/json/model/SqoopConnectorsResponse.java
>  PRE-CREATION 
>   pom.xml 3958014 
>   ranger-sqoop-plugin-shim/.gitignore PRE-CREATION 
>   ranger-sqoop-plugin-shim/pom.xml PRE-CREATION 
>   
> ranger-sqoop-plugin-shim/src/main/java/org/apache/ranger/authorization/sqoop/authorizer/RangerSqoopAuthorizer.java
>  PRE-CREATION 
>   src/main/assembly/admin-web.xml 4dc52fd 
>   src/main/assembly/plugin-sqoop.xml PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62710/diff/3/
> 
> 
> Testing
> ---
> 
> Our test specialists have rigorously tested this feature.
> 
> 
> Thanks,
> 
> Qiang Zhang
> 
>



[jira] [Commented] (RANGER-1847) Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN

2017-10-20 Thread Ronald van de Kuil (JIRA)

[ 
https://issues.apache.org/jira/browse/RANGER-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212687#comment-16212687
 ] 

Ronald van de Kuil commented on RANGER-1847:


Thank you Velmurugan. Kafka seems to be a happy camper now. 

> Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN
> -
>
> Key: RANGER-1847
> URL: https://issues.apache.org/jira/browse/RANGER-1847
> Project: Ranger
>  Issue Type: Test
>  Components: plugins
>Affects Versions: 0.6.3, 0.7.1
> Environment: ubuntu stand-alone hobby environment
>Reporter: Ronald van de Kuil
>Priority: Minor
>
> I am such a NOOB hobby-ing away. And I like it. ;)
> I figured I would give it a try to setup Kafka to use the 
> sasl.enabled.mechanisms of type PLAIN with ranger to do the authorisation and 
> the auditing (instead of GSSAPI).
> I got it to work pretty far. KafkaServer gets into state SaslAuthenticated 
> with Zookeeper. 
> Next it loads the ranger kafka plugin. Then the RangerKafkaAuthorizer 
> complains about Kerberos. 
> I then updated the CLASSPATH and it complains about something else.
> I am not sure how to classify this issue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (RANGER-1847) Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN

2017-10-20 Thread Ronald van de Kuil (JIRA)

[ 
https://issues.apache.org/jira/browse/RANGER-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212684#comment-16212684
 ] 

Ronald van de Kuil commented on RANGER-1847:


I am such a NOOB, ;)

> Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN
> -
>
> Key: RANGER-1847
> URL: https://issues.apache.org/jira/browse/RANGER-1847
> Project: Ranger
>  Issue Type: Test
>  Components: plugins
>Affects Versions: 0.6.3, 0.7.1
> Environment: ubuntu stand-alone hobby environment
>Reporter: Ronald van de Kuil
>Priority: Minor
>
> I am such a NOOB hobby-ing away. And I like it. ;)
> I figured I would give it a try to setup Kafka to use the 
> sasl.enabled.mechanisms of type PLAIN with ranger to do the authorisation and 
> the auditing (instead of GSSAPI).
> I got it to work pretty far. KafkaServer gets into state SaslAuthenticated 
> with Zookeeper. 
> Next it loads the ranger kafka plugin. Then the RangerKafkaAuthorizer 
> complains about Kerberos. 
> I then updated the CLASSPATH and it complains about something else.
> I am not sure how to classify this issue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (RANGER-1847) Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN

2017-10-20 Thread Ronald van de Kuil (JIRA)

[ 
https://issues.apache.org/jira/browse/RANGER-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212683#comment-16212683
 ] 

Ronald van de Kuil commented on RANGER-1847:


All that was left was to register kafkadev in the admin console.

> Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN
> -
>
> Key: RANGER-1847
> URL: https://issues.apache.org/jira/browse/RANGER-1847
> Project: Ranger
>  Issue Type: Test
>  Components: plugins
>Affects Versions: 0.6.3, 0.7.1
> Environment: ubuntu stand-alone hobby environment
>Reporter: Ronald van de Kuil
>Priority: Minor
>
> I am such a NOOB hobby-ing away. And I like it. ;)
> I figured I would give it a try to setup Kafka to use the 
> sasl.enabled.mechanisms of type PLAIN with ranger to do the authorisation and 
> the auditing (instead of GSSAPI).
> I got it to work pretty far. KafkaServer gets into state SaslAuthenticated 
> with Zookeeper. 
> Next it loads the ranger kafka plugin. Then the RangerKafkaAuthorizer 
> complains about Kerberos. 
> I then updated the CLASSPATH and it complains about something else.
> I am not sure how to classify this issue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (RANGER-1847) Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN

2017-10-20 Thread Velmurugan Periasamy (JIRA)

[ 
https://issues.apache.org/jira/browse/RANGER-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212679#comment-16212679
 ] 

Velmurugan Periasamy commented on RANGER-1847:
--

>> kafkadev not found

Have you created kafka repository in ranger? 

> Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN
> -
>
> Key: RANGER-1847
> URL: https://issues.apache.org/jira/browse/RANGER-1847
> Project: Ranger
>  Issue Type: Test
>  Components: plugins
>Affects Versions: 0.6.3, 0.7.1
> Environment: ubuntu stand-alone hobby environment
>Reporter: Ronald van de Kuil
>Priority: Minor
>
> I am such a NOOB hobby-ing away. And I like it. ;)
> I figured I would give it a try to setup Kafka to use the 
> sasl.enabled.mechanisms of type PLAIN with ranger to do the authorisation and 
> the auditing (instead of GSSAPI).
> I got it to work pretty far. KafkaServer gets into state SaslAuthenticated 
> with Zookeeper. 
> Next it loads the ranger kafka plugin. Then the RangerKafkaAuthorizer 
> complains about Kerberos. 
> I then updated the CLASSPATH and it complains about something else.
> I am not sure how to classify this issue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (RANGER-1847) Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN

2017-10-20 Thread Ronald van de Kuil (JIRA)

[ 
https://issues.apache.org/jira/browse/RANGER-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212678#comment-16212678
 ] 

Ronald van de Kuil commented on RANGER-1847:


Ahh, ... now I remember

> Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN
> -
>
> Key: RANGER-1847
> URL: https://issues.apache.org/jira/browse/RANGER-1847
> Project: Ranger
>  Issue Type: Test
>  Components: plugins
>Affects Versions: 0.6.3, 0.7.1
> Environment: ubuntu stand-alone hobby environment
>Reporter: Ronald van de Kuil
>Priority: Minor
>
> I am such a NOOB hobby-ing away. And I like it. ;)
> I figured I would give it a try to setup Kafka to use the 
> sasl.enabled.mechanisms of type PLAIN with ranger to do the authorisation and 
> the auditing (instead of GSSAPI).
> I got it to work pretty far. KafkaServer gets into state SaslAuthenticated 
> with Zookeeper. 
> Next it loads the ranger kafka plugin. Then the RangerKafkaAuthorizer 
> complains about Kerberos. 
> I then updated the CLASSPATH and it complains about something else.
> I am not sure how to classify this issue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (RANGER-1847) Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN

2017-10-20 Thread Ronald van de Kuil (JIRA)

[ 
https://issues.apache.org/jira/browse/RANGER-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212674#comment-16212674
 ] 

Ronald van de Kuil commented on RANGER-1847:


I did reinstall and this time I also installed usersync. This got rid of a few 
errors.

The kafka server says:

[2017-10-14 13:25:46,192] ERROR Error getting policies. secureMode=false, 
user=kafka (auth:SIMPLE), 
response={"httpStatusCode":400,"statusCode":1,"msgDesc":"Service:kafkadev not 
found","messageList":[{"name":"DATA_NOT_FOUND","rbKey":"xa.error.data_not_found","message":"Data
 not found"}]}, serviceName=kafkadev 

Ranger says:

ranger_admin.log:2017-10-14 13:38:17,786 [http-bio-6080-exec-10] INFO  
org.apache.ranger.common.RESTErrorUtil (RESTErrorUtil.java:336) - Operation 
error. 
response=VXResponse={org.apache.ranger.view.VXResponse@71383623statusCode={1} 
msgDesc={Service:kafkadev not found} 
messageList={[VXMessage={org.apache.ranger.view.VXMessage@40cfff48name={DATA_NOT_FOUND}
 rbKey={xa.error.data_not_found} message={Data not found} objectId={null} 
fieldName={null} }]} }



> Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN
> -
>
> Key: RANGER-1847
> URL: https://issues.apache.org/jira/browse/RANGER-1847
> Project: Ranger
>  Issue Type: Test
>  Components: plugins
>Affects Versions: 0.6.3, 0.7.1
> Environment: ubuntu stand-alone hobby environment
>Reporter: Ronald van de Kuil
>Priority: Minor
>
> I am such a NOOB hobby-ing away. And I like it. ;)
> I figured I would give it a try to setup Kafka to use the 
> sasl.enabled.mechanisms of type PLAIN with ranger to do the authorisation and 
> the auditing (instead of GSSAPI).
> I got it to work pretty far. KafkaServer gets into state SaslAuthenticated 
> with Zookeeper. 
> Next it loads the ranger kafka plugin. Then the RangerKafkaAuthorizer 
> complains about Kerberos. 
> I then updated the CLASSPATH and it complains about something else.
> I am not sure how to classify this issue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (RANGER-1672) Ranger supports plugin to enable, monitor and manage apache kylin

2017-10-20 Thread Velmurugan Periasamy (JIRA)

[ 
https://issues.apache.org/jira/browse/RANGER-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212669#comment-16212669
 ] 

Velmurugan Periasamy commented on RANGER-1672:
--

[~zhangqiang2] - what is your apache cwiki id? I have given permissions to 
Qiang Zhang (matrixz83) to add pages in 
https://cwiki.apache.org/confluence/display/RANGER/Index

> Ranger supports plugin to enable, monitor and manage apache kylin
> -
>
> Key: RANGER-1672
> URL: https://issues.apache.org/jira/browse/RANGER-1672
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>Reporter: Qiang Zhang
>Assignee: Qiang Zhang
>  Labels: newbie, patch
> Attachments: 
> 0001-RANGER-1672-Ranger-supports-plugin-to-enable-monitor.patch, 
> KylinAuditLog.jpg, KylinPlugins.jpg, KylinPolicies.jpg, 
> KylinServiceEntry.jpg, NewKylinPolicy.jpg, NewKylinService.jpg
>
>
> Apache Kylin is an open source Distributed Analytics Engine designed to 
> provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop 
> supporting extremely large datasets, original contributed from eBay Inc. 
> Apache Kylin lets user query massive data set at sub-second latency in 3 
> steps.
> 1. Identify a Star Schema on Hadoop.
> 2. Build Cube from the identified tables.
> 3. Query with ANSI-SQL and get results in sub-second, via ODBC, JDBC or 
> RESTful API.
> We should support that using Ranger to control kylin's access rights for 
> project and cube.
> Specific implementation plan is as following:
> On the ranger website, administrators can configure policies to control user 
> access to projects and cube permissions.
> Kylin provides an abstract class and authorization interfaces for use by the 
> ranger plugin. kylin instantiates ranger plugin’s implementation class when 
> starting(this class extends the abstract class provided by kylin).
> Ranger plugin periodically polls ranger admin, updates the policy to the 
> local, and updates project and cube access rights based on policy information.
> In the Kylin side:
> 1. Kylin provides an abstract class that enables the ranger plugin's 
> implementation class to extend.
> 2. Add configuration item. 1) ranger authorization switch, 2) ranger plugin 
> implementation class's name.
> 3. Instantiate the ranger plugin implementation class when starting kylin.
> 4. kylin provides authorization interfaces for ranger plugin calls.
> 5. According to the ranger authorization configuration item, hide kylin's 
> authorization management page.
> 6. Using ranger manager access rights of the kylin does not affect kylin's 
> existing permissions functions and logic.
> In the Ranger side:
> 1. Ranger plugin will periodically polls ranger admin, updates the policy to 
> the local.
> 2. The ranger plugin invoking the authorization interfaces provided by kylin 
> to updates the project and cube access rights based on the policy information.
> reference link:https://issues.apache.org/jira/browse/KYLIN-2703



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


Re: Review Request 63147: RANGER-1845 - Add support to configure JWT signature algorithms

2017-10-20 Thread Velmurugan Periasamy

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63147/#review188835
---


Ship it!




Ship It!

- Velmurugan Periasamy


On Oct. 19, 2017, 1:18 p.m., Colm O hEigeartaigh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63147/
> ---
> 
> (Updated Oct. 19, 2017, 1:18 p.m.)
> 
> 
> Review request for ranger.
> 
> 
> Bugs: RANGER-1845
> https://issues.apache.org/jira/browse/RANGER-1845
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> The next Knox release will feature the ability to change the signature 
> algorithm from the default RS256. This task is to add support to specify a 
> signature algorithm in Ranger, which the received token must match. The new 
> configuration parameter is "ranger.sso.expected.sigalg" with a default value 
> of "RS256".
> 
> 
> Diffs
> -
> 
>   
> security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerSSOAuthenticationFilter.java
>  5e4207c9 
>   
> security-admin/src/main/java/org/apache/ranger/security/web/filter/SSOAuthenticationProperties.java
>  b8246a9a 
> 
> 
> Diff: https://reviews.apache.org/r/63147/diff/1/
> 
> 
> Testing
> ---
> 
> Tested it with Knox.
> 
> 
> Thanks,
> 
> Colm O hEigeartaigh
> 
>



[jira] [Updated] (RANGER-1827) micro benchmark for policy evaluation

2017-10-20 Thread Endre Kovacs (JIRA)

 [ 
https://issues.apache.org/jira/browse/RANGER-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Endre Kovacs updated RANGER-1827:
-
Attachment: (was: 
0001-RANGER-1827-microbenchmark-for-RangerPolicyEngine.patch)

> micro benchmark for policy evaluation
> -
>
> Key: RANGER-1827
> URL: https://issues.apache.org/jira/browse/RANGER-1827
> Project: Ranger
>  Issue Type: Test
>  Components: Ranger
>Affects Versions: master
>Reporter: Endre Kovacs
>Assignee: Endre Kovacs
>Priority: Minor
>  Labels: performance, test
> Fix For: 1.0.0
>
> Attachments: 
> 0001-RANGER-1827-microbenchmark-for-RangerPolicyEngine.patch, 
> performance-chart-trie-enabled.html, performance-chart.html, 
> policy-evaluation-performance.png
>
>
> implement micro benchmark testing the performance of RangerPolicyEngine at 
> different load of # of policies and # of concurrent users



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (RANGER-1827) micro benchmark for policy evaluation

2017-10-20 Thread Endre Kovacs (JIRA)

 [ 
https://issues.apache.org/jira/browse/RANGER-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Endre Kovacs updated RANGER-1827:
-
Attachment: 0001-RANGER-1827-microbenchmark-for-RangerPolicyEngine.patch

> micro benchmark for policy evaluation
> -
>
> Key: RANGER-1827
> URL: https://issues.apache.org/jira/browse/RANGER-1827
> Project: Ranger
>  Issue Type: Test
>  Components: Ranger
>Affects Versions: master
>Reporter: Endre Kovacs
>Assignee: Endre Kovacs
>Priority: Minor
>  Labels: performance, test
> Fix For: 1.0.0
>
> Attachments: 
> 0001-RANGER-1827-microbenchmark-for-RangerPolicyEngine.patch, 
> performance-chart-trie-enabled.html, performance-chart.html, 
> policy-evaluation-performance.png
>
>
> implement micro benchmark testing the performance of RangerPolicyEngine at 
> different load of # of policies and # of concurrent users



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


Re: Review Request 62850: RANGER-1827: microbenchmark for policy evaluation

2017-10-20 Thread Endre Zoltan Kovacs via Review Board

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62850/
---

(Updated Oct. 20, 2017, 12:08 p.m.)


Review request for ranger.


Changes
---

added license header to html template file to fix build


Bugs: RANGER-1827
https://issues.apache.org/jira/browse/RANGER-1827


Repository: ranger


Description
---

created a microbenchmark for the policy evaluation engine


Diffs (updated)
-

  
agents-common/src/main/java/org/apache/ranger/plugin/util/PerfDataRecorder.java 
25f533476 
  ranger-tools/pom.xml ff37fb3eb 
  ranger-tools/src/main/java/org/apache/ranger/policyengine/PerfTestClient.java 
e6095cba2 
  
ranger-tools/src/main/java/org/apache/ranger/policyengine/RangerAccessRequestDeserializer.java
 PRE-CREATION 
  
ranger-tools/src/main/java/org/apache/ranger/policyengine/RangerResourceDeserializer.java
 PRE-CREATION 
  
ranger-tools/src/main/java/org/apache/ranger/policyengine/perftest/v2/RangerPolicyFactory.java
 PRE-CREATION 
  
ranger-tools/src/test/java/org/apache/ranger/policyengine/RangerPolicyEnginePerformanceTest.java
 PRE-CREATION 
  ranger-tools/src/test/resources/log4j.properties 4ea9d854e 
  ranger-tools/src/test/resources/testdata/performance-chart.template 
PRE-CREATION 
  ranger-tools/src/test/resources/testdata/single-policy-template.json 
PRE-CREATION 
  ranger-tools/src/test/resources/testdata/single-request-template.json 
PRE-CREATION 


Diff: https://reviews.apache.org/r/62850/diff/7/

Changes: https://reviews.apache.org/r/62850/diff/6-7/


Testing
---

A parameterized JUnit test that tests the performance of RangerPolicyEngine 
under increasing load of number of policies and concurrent calls.
a cross product of the input parameters are generated and fed into the test 
method.

This microbenchmark includes a warm-up phase so that any of the JIT performance 
optimizations happen before the measurement of the policy engine's performance.


File Attachments


0001-RANGER-1827-microbenchmark-for-RangerPolicyEngine.patch
  
https://reviews.apache.org/media/uploaded/files/2017/10/10/3fc881bb-c8ea-427b-a6b4-8d1236159c82__0001-RANGER-1827-microbenchmark-for-RangerPolicyEngine.patch


Thanks,

Endre Zoltan Kovacs



[jira] [Created] (RANGER-1848) Implement getLinkList/getJobList in SqoopClient for Ranger Sqoop2 plugin

2017-10-20 Thread Qiang Zhang (JIRA)
Qiang Zhang created RANGER-1848:
---

 Summary: Implement getLinkList/getJobList in SqoopClient for 
Ranger Sqoop2 plugin
 Key: RANGER-1848
 URL: https://issues.apache.org/jira/browse/RANGER-1848
 Project: Ranger
  Issue Type: Sub-task
  Components: admin
Affects Versions: 1.0.0, master
Reporter: Qiang Zhang
Assignee: Qiang Zhang


Implement getLinkList/getJobList in SqoopClient for Ranger Sqoop2 plugin.
It affects link and job resource lookup when creating sqoop policy in 
RangerAdmin.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


Re: Review Request 62710: RANGER-1810:Ranger supports plugin to enable, monitor and manage apache Sqoop2

2017-10-20 Thread Qiang Zhang

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62710/
---

(Updated 十月 20, 2017, 7:16 a.m.)


Review request for ranger, Ankita Sinha, Don Bosco Durai, Colm O hEigeartaigh, 
Gautam Borad, Madhan Neethiraj, pengjianhua, Ramesh Mani, Selvamohan Neethiraj, 
sam  rome, Venkat Ranganathan, and Velmurugan Periasamy.


Changes
---

Update sqoop2 from 1.99.6 to 1.99.7 as it is the latest version


Bugs: RANGER-1810
https://issues.apache.org/jira/browse/RANGER-1810


Repository: ranger


Description
---

Apache Sqoop is a tool designed for efficiently transferring bulk data between 
Apache Hadoop and structured datastores such as relational databases. You can 
use Sqoop to import data from external structured datastores into Hadoop 
Distributed File System or related systems like Hive and HBase. Conversely, 
Sqoop can be used to extract data from Hadoop and export it to external 
structured datastores such as relational databases and enterprise data 
warehouses.It successfully graduated from the Incubator in March of 2012 and is 
now a Top-Level Apache project.
The Ranger will further expand the influence in the hadoop ecosystem if it 
supports sqoop authorization. So we should develop sqoop plugin to enable, 
monitor and manage apache Sqoop2.

Our test specialists have rigorously tested this feature.


Diffs (updated)
-

  agents-common/scripts/enable-agent.sh d31a264 
  
agents-common/src/main/java/org/apache/ranger/plugin/store/EmbeddedServiceDefsUtil.java
 9463ab8 
  agents-common/src/main/resources/service-defs/ranger-servicedef-sqoop.json 
PRE-CREATION 
  plugin-sqoop/.gitignore PRE-CREATION 
  plugin-sqoop/conf/ranger-policymgr-ssl-changes.cfg PRE-CREATION 
  plugin-sqoop/conf/ranger-policymgr-ssl.xml PRE-CREATION 
  plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg PRE-CREATION 
  plugin-sqoop/conf/ranger-sqoop-audit.xml PRE-CREATION 
  plugin-sqoop/conf/ranger-sqoop-security-changes.cfg PRE-CREATION 
  plugin-sqoop/conf/ranger-sqoop-security.xml PRE-CREATION 
  plugin-sqoop/pom.xml PRE-CREATION 
  plugin-sqoop/scripts/install.properties PRE-CREATION 
  
plugin-sqoop/src/main/java/org/apache/ranger/authorization/sqoop/authorizer/RangerSqoopAuthorizer.java
 PRE-CREATION 
  
plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/RangerServiceSqoop.java
 PRE-CREATION 
  
plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/client/SqoopClient.java
 PRE-CREATION 
  
plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/client/SqoopResourceMgr.java
 PRE-CREATION 
  
plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/client/json/model/SqoopConnectorResponse.java
 PRE-CREATION 
  
plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/client/json/model/SqoopConnectorsResponse.java
 PRE-CREATION 
  pom.xml 3958014 
  ranger-sqoop-plugin-shim/.gitignore PRE-CREATION 
  ranger-sqoop-plugin-shim/pom.xml PRE-CREATION 
  
ranger-sqoop-plugin-shim/src/main/java/org/apache/ranger/authorization/sqoop/authorizer/RangerSqoopAuthorizer.java
 PRE-CREATION 
  src/main/assembly/admin-web.xml 4dc52fd 
  src/main/assembly/plugin-sqoop.xml PRE-CREATION 


Diff: https://reviews.apache.org/r/62710/diff/3/

Changes: https://reviews.apache.org/r/62710/diff/2-3/


Testing
---

Our test specialists have rigorously tested this feature.


Thanks,

Qiang Zhang



[jira] [Comment Edited] (RANGER-1810) Ranger supports plugin to enable, monitor and manage apache Sqoop2

2017-10-20 Thread Qiang Zhang (JIRA)

[ 
https://issues.apache.org/jira/browse/RANGER-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186959#comment-16186959
 ] 

Qiang Zhang edited comment on RANGER-1810 at 10/20/17 7:13 AM:
---

Review Requet:
https://reviews.apache.org/r/62710/

Solution patch, please see attachment:
[patch|https://issues.apache.org/jira/secure/attachment/12893196/0001-RANGER-1810-Ranger-supports-plugin-to-enable-monitor.patch]

Implementation details:
[1_SqoopServiceManager.jpg|https://issues.apache.org/jira/secure/attachment/12893195/1_SqoopServiceManager.jpg]
[2_EditSqoopService.jpg|https://issues.apache.org/jira/secure/attachment/12893194/2_EditSqoopService.jpg]
[3_ListSqoopPolicies.jpg|https://issues.apache.org/jira/secure/attachment/12893193/3_ListSqoopPolicies.jpg]
[4_EditSqoopPolicy.jpg|https://issues.apache.org/jira/secure/attachment/12893192/4_EditSqoopPolicy.jpg]
[5_SqoopAuditLog.jpg|https://issues.apache.org/jira/secure/attachment/12893191/5_SqoopAuditLog.jpg]
[6_SqoopPlugins.jpg|https://issues.apache.org/jira/secure/attachment/12893190/6_SqoopPlugins.jpg]


was (Author: zhangqiang2):
Review Requet:
https://reviews.apache.org/r/62710/

Solution patch, please see attachment:
[patch|https://issues.apache.org/jira/secure/attachment/12891671/0001-RANGER-1810-Ranger-supports-plugin-to-enable-monitor.patch]

Implementation details:
[SqoopServiceManager|https://issues.apache.org/jira/secure/attachment/12889840/SqoopServiceManager.jpg]
[EditSqoopService|https://issues.apache.org/jira/secure/attachment/12889839/EditSqoopService.jpg]
[ListSqoopPolicies|https://issues.apache.org/jira/secure/attachment/12889838/ListSqoopPolicies.jpg]
[EditSqoopPolicy|https://issues.apache.org/jira/secure/attachment/12889837/EditSqoopPolicy.jpg]
[SqoopAuditLog|https://issues.apache.org/jira/secure/attachment/12889836/SqoopAuditLog.jpg]
[SqoopPlugins|https://issues.apache.org/jira/secure/attachment/12889835/SqoopPlugins.jpg]

> Ranger supports plugin to enable, monitor and manage apache Sqoop2
> --
>
> Key: RANGER-1810
> URL: https://issues.apache.org/jira/browse/RANGER-1810
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>Affects Versions: 1.0.0, master
>Reporter: Qiang Zhang
>Assignee: Qiang Zhang
>  Labels: newbie, patch
> Attachments: 
> 0001-RANGER-1810-Ranger-supports-plugin-to-enable-monitor.patch, 
> 1_SqoopServiceManager.jpg, 2_EditSqoopService.jpg, 3_ListSqoopPolicies.jpg, 
> 4_EditSqoopPolicy.jpg, 5_SqoopAuditLog.jpg, 6_SqoopPlugins.jpg
>
>
> Apache Sqoop is a tool designed for efficiently transferring bulk data 
> between Apache Hadoop and structured datastores such as relational databases. 
> You can use Sqoop to import data from external structured datastores into 
> Hadoop Distributed File System or related systems like Hive and HBase. 
> Conversely, Sqoop can be used to extract data from Hadoop and export it to 
> external structured datastores such as relational databases and enterprise 
> data warehouses.It successfully graduated from the Incubator in March of 2012 
> and is now a Top-Level Apache project.  
> The Ranger will further expand the influence in the hadoop ecosystem if it 
> supports sqoop authorization. So we should develop sqoop plugin to enable, 
> monitor and manage apache Sqoop2.
> Our test specialists have rigorously tested this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (RANGER-1810) Ranger supports plugin to enable, monitor and manage apache Sqoop2

2017-10-20 Thread Qiang Zhang (JIRA)

 [ 
https://issues.apache.org/jira/browse/RANGER-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qiang Zhang updated RANGER-1810:

Attachment: 0001-RANGER-1810-Ranger-supports-plugin-to-enable-monitor.patch
1_SqoopServiceManager.jpg
2_EditSqoopService.jpg
3_ListSqoopPolicies.jpg
4_EditSqoopPolicy.jpg
5_SqoopAuditLog.jpg
6_SqoopPlugins.jpg

> Ranger supports plugin to enable, monitor and manage apache Sqoop2
> --
>
> Key: RANGER-1810
> URL: https://issues.apache.org/jira/browse/RANGER-1810
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>Affects Versions: 1.0.0, master
>Reporter: Qiang Zhang
>Assignee: Qiang Zhang
>  Labels: newbie, patch
> Attachments: 
> 0001-RANGER-1810-Ranger-supports-plugin-to-enable-monitor.patch, 
> 1_SqoopServiceManager.jpg, 2_EditSqoopService.jpg, 3_ListSqoopPolicies.jpg, 
> 4_EditSqoopPolicy.jpg, 5_SqoopAuditLog.jpg, 6_SqoopPlugins.jpg
>
>
> Apache Sqoop is a tool designed for efficiently transferring bulk data 
> between Apache Hadoop and structured datastores such as relational databases. 
> You can use Sqoop to import data from external structured datastores into 
> Hadoop Distributed File System or related systems like Hive and HBase. 
> Conversely, Sqoop can be used to extract data from Hadoop and export it to 
> external structured datastores such as relational databases and enterprise 
> data warehouses.It successfully graduated from the Incubator in March of 2012 
> and is now a Top-Level Apache project.  
> The Ranger will further expand the influence in the hadoop ecosystem if it 
> supports sqoop authorization. So we should develop sqoop plugin to enable, 
> monitor and manage apache Sqoop2.
> Our test specialists have rigorously tested this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (RANGER-1810) Ranger supports plugin to enable, monitor and manage apache Sqoop2

2017-10-20 Thread Qiang Zhang (JIRA)

 [ 
https://issues.apache.org/jira/browse/RANGER-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qiang Zhang updated RANGER-1810:

Attachment: (was: ListSqoopPolicies.jpg)

> Ranger supports plugin to enable, monitor and manage apache Sqoop2
> --
>
> Key: RANGER-1810
> URL: https://issues.apache.org/jira/browse/RANGER-1810
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>Affects Versions: 1.0.0, master
>Reporter: Qiang Zhang
>Assignee: Qiang Zhang
>  Labels: newbie, patch
>
> Apache Sqoop is a tool designed for efficiently transferring bulk data 
> between Apache Hadoop and structured datastores such as relational databases. 
> You can use Sqoop to import data from external structured datastores into 
> Hadoop Distributed File System or related systems like Hive and HBase. 
> Conversely, Sqoop can be used to extract data from Hadoop and export it to 
> external structured datastores such as relational databases and enterprise 
> data warehouses.It successfully graduated from the Incubator in March of 2012 
> and is now a Top-Level Apache project.  
> The Ranger will further expand the influence in the hadoop ecosystem if it 
> supports sqoop authorization. So we should develop sqoop plugin to enable, 
> monitor and manage apache Sqoop2.
> Our test specialists have rigorously tested this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (RANGER-1810) Ranger supports plugin to enable, monitor and manage apache Sqoop2

2017-10-20 Thread Qiang Zhang (JIRA)

 [ 
https://issues.apache.org/jira/browse/RANGER-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qiang Zhang updated RANGER-1810:

Attachment: (was: EditSqoopService.jpg)

> Ranger supports plugin to enable, monitor and manage apache Sqoop2
> --
>
> Key: RANGER-1810
> URL: https://issues.apache.org/jira/browse/RANGER-1810
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>Affects Versions: 1.0.0, master
>Reporter: Qiang Zhang
>Assignee: Qiang Zhang
>  Labels: newbie, patch
>
> Apache Sqoop is a tool designed for efficiently transferring bulk data 
> between Apache Hadoop and structured datastores such as relational databases. 
> You can use Sqoop to import data from external structured datastores into 
> Hadoop Distributed File System or related systems like Hive and HBase. 
> Conversely, Sqoop can be used to extract data from Hadoop and export it to 
> external structured datastores such as relational databases and enterprise 
> data warehouses.It successfully graduated from the Incubator in March of 2012 
> and is now a Top-Level Apache project.  
> The Ranger will further expand the influence in the hadoop ecosystem if it 
> supports sqoop authorization. So we should develop sqoop plugin to enable, 
> monitor and manage apache Sqoop2.
> Our test specialists have rigorously tested this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (RANGER-1810) Ranger supports plugin to enable, monitor and manage apache Sqoop2

2017-10-20 Thread Qiang Zhang (JIRA)

 [ 
https://issues.apache.org/jira/browse/RANGER-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qiang Zhang updated RANGER-1810:

Attachment: (was: SqoopAuditLog.jpg)

> Ranger supports plugin to enable, monitor and manage apache Sqoop2
> --
>
> Key: RANGER-1810
> URL: https://issues.apache.org/jira/browse/RANGER-1810
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>Affects Versions: 1.0.0, master
>Reporter: Qiang Zhang
>Assignee: Qiang Zhang
>  Labels: newbie, patch
>
> Apache Sqoop is a tool designed for efficiently transferring bulk data 
> between Apache Hadoop and structured datastores such as relational databases. 
> You can use Sqoop to import data from external structured datastores into 
> Hadoop Distributed File System or related systems like Hive and HBase. 
> Conversely, Sqoop can be used to extract data from Hadoop and export it to 
> external structured datastores such as relational databases and enterprise 
> data warehouses.It successfully graduated from the Incubator in March of 2012 
> and is now a Top-Level Apache project.  
> The Ranger will further expand the influence in the hadoop ecosystem if it 
> supports sqoop authorization. So we should develop sqoop plugin to enable, 
> monitor and manage apache Sqoop2.
> Our test specialists have rigorously tested this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (RANGER-1810) Ranger supports plugin to enable, monitor and manage apache Sqoop2

2017-10-20 Thread Qiang Zhang (JIRA)

 [ 
https://issues.apache.org/jira/browse/RANGER-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qiang Zhang updated RANGER-1810:

Attachment: (was: 
0001-RANGER-1810-Ranger-supports-plugin-to-enable-monitor.patch)

> Ranger supports plugin to enable, monitor and manage apache Sqoop2
> --
>
> Key: RANGER-1810
> URL: https://issues.apache.org/jira/browse/RANGER-1810
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>Affects Versions: 1.0.0, master
>Reporter: Qiang Zhang
>Assignee: Qiang Zhang
>  Labels: newbie, patch
>
> Apache Sqoop is a tool designed for efficiently transferring bulk data 
> between Apache Hadoop and structured datastores such as relational databases. 
> You can use Sqoop to import data from external structured datastores into 
> Hadoop Distributed File System or related systems like Hive and HBase. 
> Conversely, Sqoop can be used to extract data from Hadoop and export it to 
> external structured datastores such as relational databases and enterprise 
> data warehouses.It successfully graduated from the Incubator in March of 2012 
> and is now a Top-Level Apache project.  
> The Ranger will further expand the influence in the hadoop ecosystem if it 
> supports sqoop authorization. So we should develop sqoop plugin to enable, 
> monitor and manage apache Sqoop2.
> Our test specialists have rigorously tested this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (RANGER-1810) Ranger supports plugin to enable, monitor and manage apache Sqoop2

2017-10-20 Thread Qiang Zhang (JIRA)

 [ 
https://issues.apache.org/jira/browse/RANGER-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qiang Zhang updated RANGER-1810:

Attachment: (was: EditSqoopPolicy.jpg)

> Ranger supports plugin to enable, monitor and manage apache Sqoop2
> --
>
> Key: RANGER-1810
> URL: https://issues.apache.org/jira/browse/RANGER-1810
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>Affects Versions: 1.0.0, master
>Reporter: Qiang Zhang
>Assignee: Qiang Zhang
>  Labels: newbie, patch
>
> Apache Sqoop is a tool designed for efficiently transferring bulk data 
> between Apache Hadoop and structured datastores such as relational databases. 
> You can use Sqoop to import data from external structured datastores into 
> Hadoop Distributed File System or related systems like Hive and HBase. 
> Conversely, Sqoop can be used to extract data from Hadoop and export it to 
> external structured datastores such as relational databases and enterprise 
> data warehouses.It successfully graduated from the Incubator in March of 2012 
> and is now a Top-Level Apache project.  
> The Ranger will further expand the influence in the hadoop ecosystem if it 
> supports sqoop authorization. So we should develop sqoop plugin to enable, 
> monitor and manage apache Sqoop2.
> Our test specialists have rigorously tested this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (RANGER-1810) Ranger supports plugin to enable, monitor and manage apache Sqoop2

2017-10-20 Thread Qiang Zhang (JIRA)

 [ 
https://issues.apache.org/jira/browse/RANGER-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qiang Zhang updated RANGER-1810:

Attachment: (was: SqoopPlugins.jpg)

> Ranger supports plugin to enable, monitor and manage apache Sqoop2
> --
>
> Key: RANGER-1810
> URL: https://issues.apache.org/jira/browse/RANGER-1810
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>Affects Versions: 1.0.0, master
>Reporter: Qiang Zhang
>Assignee: Qiang Zhang
>  Labels: newbie, patch
>
> Apache Sqoop is a tool designed for efficiently transferring bulk data 
> between Apache Hadoop and structured datastores such as relational databases. 
> You can use Sqoop to import data from external structured datastores into 
> Hadoop Distributed File System or related systems like Hive and HBase. 
> Conversely, Sqoop can be used to extract data from Hadoop and export it to 
> external structured datastores such as relational databases and enterprise 
> data warehouses.It successfully graduated from the Incubator in March of 2012 
> and is now a Top-Level Apache project.  
> The Ranger will further expand the influence in the hadoop ecosystem if it 
> supports sqoop authorization. So we should develop sqoop plugin to enable, 
> monitor and manage apache Sqoop2.
> Our test specialists have rigorously tested this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (RANGER-1810) Ranger supports plugin to enable, monitor and manage apache Sqoop2

2017-10-20 Thread Qiang Zhang (JIRA)

 [ 
https://issues.apache.org/jira/browse/RANGER-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qiang Zhang updated RANGER-1810:

Attachment: (was: SqoopServiceManager.jpg)

> Ranger supports plugin to enable, monitor and manage apache Sqoop2
> --
>
> Key: RANGER-1810
> URL: https://issues.apache.org/jira/browse/RANGER-1810
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>Affects Versions: 1.0.0, master
>Reporter: Qiang Zhang
>Assignee: Qiang Zhang
>  Labels: newbie, patch
>
> Apache Sqoop is a tool designed for efficiently transferring bulk data 
> between Apache Hadoop and structured datastores such as relational databases. 
> You can use Sqoop to import data from external structured datastores into 
> Hadoop Distributed File System or related systems like Hive and HBase. 
> Conversely, Sqoop can be used to extract data from Hadoop and export it to 
> external structured datastores such as relational databases and enterprise 
> data warehouses.It successfully graduated from the Incubator in March of 2012 
> and is now a Top-Level Apache project.  
> The Ranger will further expand the influence in the hadoop ecosystem if it 
> supports sqoop authorization. So we should develop sqoop plugin to enable, 
> monitor and manage apache Sqoop2.
> Our test specialists have rigorously tested this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


Re: Review Request 62710: RANGER-1810:Ranger supports plugin to enable, monitor and manage apache Sqoop2

2017-10-20 Thread Qiang Zhang


> On 十月 16, 2017, 12:51 p.m., Colm O hEigeartaigh wrote:
> > In addition to comment above, here are some initial remarks:
> > 
> > a) We should update to 1.99.7 as it is the latest version
> > b) getLinkList/getJobList are not implemented in SqoopClient.
> > c) There is a problem with the Sqoop REST API that I haven't been able to 
> > figure out. "http://localhost:12000/sqoop/v1/connectors"; returns a 404 with 
> > 1.99.7 (but http://localhost:12000/sqoop/v1/connector/kafka-connector works 
> > fine), so I can't get "Test Connection" to work. Same goes for the v1/jobs 
> > + v1/links mentioned in the REST API.
> > d) It was not picking up conf/ranger-sqoop-security.xml + hence the plugin 
> > wasn't working. I had to copy ranger-sqoop-security.xml to the root 
> > directory of the Sqoop installation. Any idea why this is?
> > e) After changing the policy in Ranger and having it synced down, it only 
> > "applies" once I exited the sqoop shell and restarted it. Any idea why this 
> > is? I guess it is caching the old policy somehow?

a)OK, thanks. I have updated my sqoop from 1.99.6 to 1.99.7.

b)Yes, I will give another patch to implement getLinkList/getJobList after this 
patch recevied,
because there is some work and optimization to do for connector/link/job, and 
it only affects resource lookup.
We are now focusing on Ranger's access control over Sqoop.

c)Yes, the Sqoop REST API is different between two versions.
1.99.6 : http://localhost:12000/sqoop/v1/connectors
1.99.7 : http://localhost:12000/sqoop/v1/connector/all
Same goes for the jobs and links.

d)I think it's because the conf directory is not in sqoop's classpath,
so ranger configuration files can not be read in JVM,
add conf to sqoop classpath can resolve this problem.
Please add the follow configuration to function sqoop_server_classpath_set of 
script bin/sqoop.sh:
CLASSPATH="$CLASSPATH":"$sqoop_home/conf"

e)Yes, the connector will be cached in sqoop client, 
when Ranger change the user's connector permissions,
the show connector command is not the latest result.

Execute the follow command can clear sqoop client cache:
set server --host zdh-11 --port 12000 --webapp sqoop

Please refer the following sqoop client code which cache connectors:
org.apache.sqoop.client.SqoopClient.getConnector(String)
public MConnector getConnector(String connectorName) {
  // Firstly try if we have this connector already in cache
  MConnector connector = getConnectorFromCache(connectorName);
  if(connector != null) return connector;

  // If the connector wasn't in cache and we have all connectors,
  // it simply do not exists.
  if(isAllConnectors) return null;

  // Retrieve all connectors from server
  getConnectors();
  return getConnectorFromCache(connectorName);
}


- Qiang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62710/#review188127
---


On 十月 12, 2017, 9:02 a.m., Qiang Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62710/
> ---
> 
> (Updated 十月 12, 2017, 9:02 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Don Bosco Durai, Colm O 
> hEigeartaigh, Gautam Borad, Madhan Neethiraj, pengjianhua, Ramesh Mani, 
> Selvamohan Neethiraj, sam  rome, Venkat Ranganathan, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-1810
> https://issues.apache.org/jira/browse/RANGER-1810
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Apache Sqoop is a tool designed for efficiently transferring bulk data 
> between Apache Hadoop and structured datastores such as relational databases. 
> You can use Sqoop to import data from external structured datastores into 
> Hadoop Distributed File System or related systems like Hive and HBase. 
> Conversely, Sqoop can be used to extract data from Hadoop and export it to 
> external structured datastores such as relational databases and enterprise 
> data warehouses.It successfully graduated from the Incubator in March of 2012 
> and is now a Top-Level Apache project.
> The Ranger will further expand the influence in the hadoop ecosystem if it 
> supports sqoop authorization. So we should develop sqoop plugin to enable, 
> monitor and manage apache Sqoop2.
> 
> Our test specialists have rigorously tested this feature.
> 
> 
> Diffs
> -
> 
>   agents-common/scripts/enable-agent.sh d31a264 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/store/EmbeddedServiceDefsUtil.java
>  9463ab8 
>   agents-common/src/main/resources/service-defs/ranger-servicedef-sqoop.json 
> PRE-CREATION 
>   plugin-sqoop/.gitignore PRE-CREATION 
>   plugin-sqoop/conf/ranger-policymgr-ssl-changes.cfg PRE-CREATION 
>   plugin-sqoop/conf/ranger-policymgr-ssl.xml PRE-CREATION 
>   plugin-sqoo

Re: Review Request 62710: RANGER-1810:Ranger supports plugin to enable, monitor and manage apache Sqoop2

2017-10-20 Thread Qiang Zhang


> On 十月 12, 2017, 3:33 p.m., Colm O hEigeartaigh wrote:
> > I think there is a problem with the installation scripts. The lib and conf 
> > dirs are defined as:
> > 
> > HCOMPONENT_LIB_DIR=${HCOMPONENT_INSTALL_DIR}/server/webapps/sqoop/WEB-INF/lib
> > HCOMPONENT_CONF_DIR=${HCOMPONENT_INSTALL_DIR}/server/conf
> > 
> > However these are not the standard locations if you download a Sqoop2 
> > server (instead just $sqoop_home/server/lib + $sqoop_home/conf).

OK, thanks.


- Qiang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62710/#review187801
---


On 十月 12, 2017, 9:02 a.m., Qiang Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62710/
> ---
> 
> (Updated 十月 12, 2017, 9:02 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Don Bosco Durai, Colm O 
> hEigeartaigh, Gautam Borad, Madhan Neethiraj, pengjianhua, Ramesh Mani, 
> Selvamohan Neethiraj, sam  rome, Venkat Ranganathan, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-1810
> https://issues.apache.org/jira/browse/RANGER-1810
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Apache Sqoop is a tool designed for efficiently transferring bulk data 
> between Apache Hadoop and structured datastores such as relational databases. 
> You can use Sqoop to import data from external structured datastores into 
> Hadoop Distributed File System or related systems like Hive and HBase. 
> Conversely, Sqoop can be used to extract data from Hadoop and export it to 
> external structured datastores such as relational databases and enterprise 
> data warehouses.It successfully graduated from the Incubator in March of 2012 
> and is now a Top-Level Apache project.
> The Ranger will further expand the influence in the hadoop ecosystem if it 
> supports sqoop authorization. So we should develop sqoop plugin to enable, 
> monitor and manage apache Sqoop2.
> 
> Our test specialists have rigorously tested this feature.
> 
> 
> Diffs
> -
> 
>   agents-common/scripts/enable-agent.sh d31a264 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/store/EmbeddedServiceDefsUtil.java
>  9463ab8 
>   agents-common/src/main/resources/service-defs/ranger-servicedef-sqoop.json 
> PRE-CREATION 
>   plugin-sqoop/.gitignore PRE-CREATION 
>   plugin-sqoop/conf/ranger-policymgr-ssl-changes.cfg PRE-CREATION 
>   plugin-sqoop/conf/ranger-policymgr-ssl.xml PRE-CREATION 
>   plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg PRE-CREATION 
>   plugin-sqoop/conf/ranger-sqoop-audit.xml PRE-CREATION 
>   plugin-sqoop/conf/ranger-sqoop-security-changes.cfg PRE-CREATION 
>   plugin-sqoop/conf/ranger-sqoop-security.xml PRE-CREATION 
>   plugin-sqoop/pom.xml PRE-CREATION 
>   plugin-sqoop/scripts/install.properties PRE-CREATION 
>   
> plugin-sqoop/src/main/java/org/apache/ranger/authorization/sqoop/authorizer/RangerSqoopAuthorizer.java
>  PRE-CREATION 
>   
> plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/RangerServiceSqoop.java
>  PRE-CREATION 
>   
> plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/client/SqoopClient.java
>  PRE-CREATION 
>   
> plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/client/SqoopResourceMgr.java
>  PRE-CREATION 
>   
> plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/client/json/model/SqoopConnectorResponse.java
>  PRE-CREATION 
>   
> plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/client/json/model/SqoopConnectorsResponse.java
>  PRE-CREATION 
>   pom.xml 3958014 
>   ranger-sqoop-plugin-shim/.gitignore PRE-CREATION 
>   ranger-sqoop-plugin-shim/pom.xml PRE-CREATION 
>   
> ranger-sqoop-plugin-shim/src/main/java/org/apache/ranger/authorization/sqoop/authorizer/RangerSqoopAuthorizer.java
>  PRE-CREATION 
>   src/main/assembly/admin-web.xml 0e97818 
>   src/main/assembly/plugin-sqoop.xml PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62710/diff/2/
> 
> 
> Testing
> ---
> 
> Our test specialists have rigorously tested this feature.
> 
> 
> Thanks,
> 
> Qiang Zhang
> 
>