Re: Review Request 68286: RANGER-2186: Increment service-specific policy and tag versions after update transaction is committed

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68286/#review207252
---




security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
Lines 2821 (patched)


INCREMENT_POLICY_VERSION ==> POLICY_VERSION
INCREMENT_TAG_VERSION==> TAG_VERSION



security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
Line 2864 (original), 2856 (patched)


Instead of creating multiple Runnable objects, one for each 
referringService, consider a single runnable that loops through 
referringServices.

Also, consider adding VERSION_TYPE_INCREMENT_POLICY_AND_TAG_VERSION so that 
both can be updated in a single call.

  final VERION_TYPE versionToUpdate;

  if (filterForServicePlugin && isTagVersionUpdateNeeded) {
versionToUpdate = VERSION_TYPE.INCREMENT_POLICY_AND_TAG_VERSION;
  } else {
versionToUpdate = VERSION_TYPE.INCREMENT_POLICY_VERSION;
  }

  commitWork = new Runnable() {
@Override
public void run() {
  for(XXService referringService : referringServices) {
persistVersionChange(daoMgr, referringService.getId(), 
versionToUpdate);
  }
}
  };

  transactionSynchronizationAdapter.executeOnTransactionCommit(commitWork);



security-admin/src/main/java/org/apache/ranger/common/db/RangerTransactionSynchronizationAdapter.java
Lines 171 (patched)


Consider avoiding afterCommit() method at the base class. Instead:
- have afterCompletion(status) call this method - only when status == 
STATUS_COMMITTED
- clear RUNNABLES_AFTER_COMMIT in afterCompletion(status). Currently this 
list might not be cleared if the transaction fails in commit.



security-admin/src/main/java/org/apache/ranger/db/XXServiceVersionInfoDao.java
Line 124 (original), 130 (patched)


Instead of creating multiple Runnable objects, one for each 
serviceVersionInfo, consider handling all updates within a single Runnable.


- Madhan Neethiraj


On Aug. 10, 2018, 12:58 a.m., Abhay Kulkarni wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68286/
> ---
> 
> (Updated Aug. 10, 2018, 12:58 a.m.)
> 
> 
> Review request for ranger, Madhan Neethiraj, Ramesh Mani, and Velmurugan 
> Periasamy.
> 
> 
> Bugs: RANGER-2186
> https://issues.apache.org/jira/browse/RANGER-2186
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Policy updates to different policies within a service, when successful, 
> update the service's policy version. If the update transactions are 
> concurrent, and executed on different ranger-admin servers (in HA 
> configuration), then it is possible that policy-version of the transaction 
> that commits later overwrites policy-version of earlier transaction, 
> effectively losing track of the first change.
> 
> If policy-version is updated after update to policy is committed, then the 
> window of such loss is greatly reduced.
> 
> Similar considerations apply for tag updates.
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/store/AbstractServiceStore.java
>  69ded6dc8 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 
> 0773616f9 
>   
> security-admin/src/main/java/org/apache/ranger/common/db/RangerTransactionSynchronizationAdapter.java
>  2a62fb408 
>   
> security-admin/src/main/java/org/apache/ranger/db/XXServiceVersionInfoDao.java
>  e1003297a 
>   security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 
> cb496ea8b 
> 
> 
> Diff: https://reviews.apache.org/r/68286/diff/1/
> 
> 
> Testing
> ---
> 
> Passed all unit tests
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>

[jira] [Comment Edited] (RANGER-2165) Address JPA Cache issue when policies Create, Update and Delete are done via REST API in Apache Ranger admin

[Abhay Kulkarni (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-2165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564439#comment-16564439
 ] 

Abhay Kulkarni edited comment on RANGER-2165 at 8/14/18 5:50 PM:
-

Committed to master.

[https://git-wip-us.apache.org/repos/asf?p=ranger.git;a=commit;h=edefd567ca2afead045f486ccadc3f30925f877e]

and

https://git-wip-us.apache.org/repos/asf?p=ranger.git;a=commit;h=6eac3dfc6bc72ae940cf846a941721e5e75ea812


was (Author: abhayk):
Committed to master.

https://git-wip-us.apache.org/repos/asf?p=ranger.git;a=commit;h=edefd567ca2afead045f486ccadc3f30925f877e

> Address JPA Cache issue when policies Create, Update and Delete are done via 
> REST API in Apache Ranger admin
> 
>
> Key: RANGER-2165
> URL: https://issues.apache.org/jira/browse/RANGER-2165
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: master, 1.1.0
>Reporter: Ramesh Mani
>Assignee: Abhay Kulkarni
>Priority: Critical
> Fix For: 2.0.0
>
>
> Address JPA Cache issue when policies Create, Update and Delete are done via 
> REST API in Apache Ranger admin
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68177: RANGER-2188: Support multiple threads to build Trie and on-lookup post-setup for Trie nodes

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68177/#review207238
---


Ship it!




Ship It!

- Madhan Neethiraj


On Aug. 14, 2018, 5:12 p.m., Abhay Kulkarni wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68177/
> ---
> 
> (Updated Aug. 14, 2018, 5:12 p.m.)
> 
> 
> Review request for ranger, Madhan Neethiraj, Ramesh Mani, and Velmurugan 
> Periasamy.
> 
> 
> Bugs: RANGER-2188
> https://issues.apache.org/jira/browse/RANGER-2188
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Time for building a Trie index for resources may become a bottleneck when 
> dealing with large number of resources. It is desirable to build different, 
> non-overlapping parts of Trie structure using multiple threads to reduce 
> overall build time if configured to do so.
> 
> Also, instead of building out all Trie nodes completely (that is, by 
> propagating wildcard evaluators all the way to all leaves of Trie tree) at 
> the initialization time, it is optimal to do so as a Trie-Node is accessed 
> for the first time during resource lookup.
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java
>  b12d8ff11 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineOptions.java
>  b76820cd2 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java
>  088b729fb 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/util/RangerResourceTrie.java
>  10c6faa92 
>   
> agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java
>  1d9b865b0 
> 
> 
> Diff: https://reviews.apache.org/r/68177/diff/5/
> 
> 
> Testing
> ---
> 
> Passes all unit tests
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>

Re: Review Request 68177: RANGER-2188: Support multiple threads to build Trie and on-lookup post-setup for Trie nodes

[Abhay Kulkarni]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68177/
---

(Updated Aug. 14, 2018, 5:12 p.m.)


Review request for ranger, Madhan Neethiraj, Ramesh Mani, and Velmurugan 
Periasamy.


Changes
---

Addressed review comment


Bugs: RANGER-2188
https://issues.apache.org/jira/browse/RANGER-2188


Repository: ranger


Description
---

Time for building a Trie index for resources may become a bottleneck when 
dealing with large number of resources. It is desirable to build different, 
non-overlapping parts of Trie structure using multiple threads to reduce 
overall build time if configured to do so.

Also, instead of building out all Trie nodes completely (that is, by 
propagating wildcard evaluators all the way to all leaves of Trie tree) at the 
initialization time, it is optimal to do so as a Trie-Node is accessed for the 
first time during resource lookup.


Diffs (updated)
-

  
agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java
 b12d8ff11 
  
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineOptions.java
 b76820cd2 
  
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java
 088b729fb 
  
agents-common/src/main/java/org/apache/ranger/plugin/util/RangerResourceTrie.java
 10c6faa92 
  
agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java
 1d9b865b0 


Diff: https://reviews.apache.org/r/68177/diff/5/

Changes: https://reviews.apache.org/r/68177/diff/4-5/


Testing
---

Passes all unit tests


Thanks,

Abhay Kulkarni

[jira] [Commented] (RANGER-2114) Internal Exception: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Data too long for column 'content' at row 1

[Fatima Amjad Khan (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579682#comment-16579682
 ] 

Fatima Amjad Khan commented on RANGER-2114:
---

Committed on [Master| 
https://github.com/apache/ranger/commit/ecbc7a66f124395f9ddf42d0a60d4ead9e6e0542]

> Internal Exception: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Data 
> too long for column 'content' at row 1
> 
>
> Key: RANGER-2114
> URL: https://issues.apache.org/jira/browse/RANGER-2114
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Reporter: Alexander Posledov
>Assignee: Fatima Amjad Khan
>Priority: Major
> Attachments: RANGER-2114.patch
>
>
> An attempt to delete a service definition ends with:
>  
> {code:java}
> // Internal Exception: com.mysql.jdbc.MysqlDataTruncation: Data truncation: 
> Data too long for column 'content' at row 1
> Error Code: 1406
> Call: INSERT INTO x_data_hist (action, content, CREATE_TIME, from_time, 
> obj_class_type, obj_guid, obj_id, obj_name, to_time, UPDATE_TIME, version) 
> VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
> bind = [11 parameters bound]
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68293: RANGER-2114 : Internal Exception: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Data too long for column 'content' at row 1

[Gautam Borad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68293/#review207230
---


Ship it!




Ship It!

- Gautam Borad


On Aug. 10, 2018, 1:22 p.m., Fatima Khan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68293/
> ---
> 
> (Updated Aug. 10, 2018, 1:22 p.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Don Bosco Durai, Gautam Borad, Abhay 
> Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Selvamohan Neethiraj, 
> Sailaja Polavarapu, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2114
> https://issues.apache.org/jira/browse/RANGER-2114
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> An attempt to delete a service definition ends with:
>  
> // Internal Exception: com.mysql.jdbc.MysqlDataTruncation: Data truncation: 
> Data too long for column 'content' at row 1
> Error Code: 1406
> Call: INSERT INTO x_data_hist (action, content, CREATE_TIME, from_time, 
> obj_class_type, obj_guid, obj_id, obj_name, to_time, UPDATE_TIME, version) 
> VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
> bind = [11 parameters bound]
> 
> 
> Diffs
> -
> 
>   security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 9d200ba 
>   security-admin/db/mysql/patches/034-x_data_histContentSize.sql PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/68293/diff/1/
> 
> 
> Testing
> ---
> 
> Tested the following
> 1.In fresh install and upgrade scenario.
> 2.Checked the data type of x_data_hist is changed from text to mediumtext.
> 
> 
> Thanks,
> 
> Fatima Khan
> 
>

Re: Review Request 68128: RANGER-2170:Ranger supports plugin to enable, monitor and manage Elasticsearch

[Qiang Zhang]

> On 八月 10, 2018, 12:50 p.m., Zsombor Gegesy wrote:
> > plugin-elasticsearch/src/main/java/org/apache/ranger/authorization/elasticsearch/authorizer/RangerElasticsearchAuthorizer.java
> > Lines 99 (patched)
> > 
> >
> > I don't get it, why this clientIPAddress is used. This is not a client 
> > IP address, but the current ElasticServer's address. This could confuse the 
> > users. Why not leaving just empty, if ES won't provide it automatically?

OK, I will get client IP address from rest request.


> On 八月 10, 2018, 12:50 p.m., Zsombor Gegesy wrote:
> > plugin-elasticsearch/src/main/java/org/apache/ranger/services/elasticsearch/client/ElasticsearchClient.java
> > Lines 139 (patched)
> > 
> >
> > I guess, joptsimple.internal.Strings.EMPTY is just "". I don't think, 
> > it is good idea to depend on a library just for this - and this library is 
> > not directly included in the pom.

OK, thanks.


> On 八月 10, 2018, 12:50 p.m., Zsombor Gegesy wrote:
> > ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/RangerElasticserachPlugin.java
> > Lines 50 (patched)
> > 
> >
> > RangerElasticserachPlugin -> RangerElasticsearchPlugin

OK, thanks.


> On 八月 10, 2018, 12:50 p.m., Zsombor Gegesy wrote:
> > ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/action/filter/RangerSecurityActionFilter.java
> > Lines 76 (patched)
> > 
> >
> > Isn't it possible to get the user's IP address from 
> > action.remoteAddress()? I'm not too familiar with ES inner workings.

Thanks, I find the user's IP address can be getted from RestRequest in the 
following method:
org.apache.ranger.authorization.elasticsearch.plugin.rest.filter.RangerSecurityRestFilter.handleRequest(RestRequest,
 RestChannel, NodeClient)


> On 八月 10, 2018, 12:50 p.m., Zsombor Gegesy wrote:
> > ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/action/filter/RangerSecurityActionFilter.java
> > Lines 88 (patched)
> > 
> >
> > Isn't it a security problem, if no user name is there, the plugin 
> > allows everything?

1.It is designed for elasticsearch internal call, such as elasticsearch start 
up with no user name, 
otherwise elasticsearch cluster could not work.

2.Restful API call with no user name would throw exception.
Please refer the following test:
curl -X GET "localhost:9200/twitter/_stats?pretty"
{
  "error" : {
"root_cause" : [
  {
"type" : "status_exception",
"reason" : "Error: User is null, the request requires user 
authentication."
  }
],
"type" : "status_exception",
"reason" : "Error: User is null, the request requires user authentication."
  },
  "status" : 401
}

3.May be there is better design in the future,
to do more rigorous verification for elasticsearch cluster.


- Qiang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68128/#review207073
---


On 八月 14, 2018, 8:46 a.m., Qiang Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68128/
> ---
> 
> (Updated 八月 14, 2018, 8:46 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Don Bosco Durai, Colm O 
> hEigeartaigh, Gautam Borad, Madhan Neethiraj, pengjianhua, Ramesh Mani, 
> Selvamohan Neethiraj, sam  rome, Venkat Ranganathan, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2170
> https://issues.apache.org/jira/browse/RANGER-2170
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Elasticsearch is a distributed, RESTful search and analytics engine capable 
> of solving a growing number of use cases. 
> Like Apache Solr, it is also an index server based on Lucence.
> Ranger supports plugin to enable, monitor and manage Elasticsearch,
> to control index security of Elasticsearch.
> 
> As there is X-Pack plugin for the Elasticsearch, but it is not free.
> X-Pack is an Elastic Stack extension that bundles security, alerting, 
> monitoring, reporting, 
> and graph capabilities into one easy-to-install package.
> We refer to the Indices Privileges design of X-Pack,
> by keeping the permissions consistent,
> to make user use ranger Elasticsearch plugin easily.
> Reference X-Pack Indices Privileges:
> https://www.elastic.co/guide/en/x-pack/current/security-privileges.html
> 
> Here we 

[jira] [Comment Edited] (RANGER-2170) Ranger supports plugin to enable, monitor and manage Elasticsearch

[Qiang Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563379#comment-16563379
 ] 

Qiang Zhang edited comment on RANGER-2170 at 8/14/18 8:47 AM:
--

Review Requet:
https://reviews.apache.org/r/68128/

Solution patch, please see attachment:
[patch|https://issues.apache.org/jira/secure/attachment/12935505/0001-RANGER-2170-Ranger-supports-plugin-to-enable-monitor.patch]

Implementation details:
[1_ElasticSearchServiceEntry|https://issues.apache.org/jira/secure/attachment/12933730/1_ElasticSearchServiceEntry.jpg]
[2_EditElasticSearchService|https://issues.apache.org/jira/secure/attachment/12933731/2_EditElasticSearchService.jpg]
[3_ElasticSearchPolicies|https://issues.apache.org/jira/secure/attachment/12933732/3_ElasticSearchPolicies.jpg]
[4_EditElasticSearchPolicy|https://issues.apache.org/jira/secure/attachment/12933733/4_EditElasticSearchPolicy.jpg]
[5_ElasticSearchAuditLog|https://issues.apache.org/jira/secure/attachment/12933734/5_ElasticSearchAuditLog.jpg]
[6_ElasticSearchPlugins|https://issues.apache.org/jira/secure/attachment/12933735/6_ElasticSearchPlugins.jpg]
[7_ElasticSearchPluginStatus.jpg|https://issues.apache.org/jira/secure/attachment/12933736/7_ElasticSearchPluginStatus.jpg]


was (Author: zhangqiang2):
Review Requet:
https://reviews.apache.org/r/68128/

Solution patch, please see attachment:
[patch|https://issues.apache.org/jira/secure/attachment/12933727/0001-RANGER-2170-Ranger-supports-plugin-to-enable-monitor.patch]

Implementation details:
[1_ElasticSearchServiceEntry|https://issues.apache.org/jira/secure/attachment/12933730/1_ElasticSearchServiceEntry.jpg]
[2_EditElasticSearchService|https://issues.apache.org/jira/secure/attachment/12933731/2_EditElasticSearchService.jpg]
[3_ElasticSearchPolicies|https://issues.apache.org/jira/secure/attachment/12933732/3_ElasticSearchPolicies.jpg]
[4_EditElasticSearchPolicy|https://issues.apache.org/jira/secure/attachment/12933733/4_EditElasticSearchPolicy.jpg]
[5_ElasticSearchAuditLog|https://issues.apache.org/jira/secure/attachment/12933734/5_ElasticSearchAuditLog.jpg]
[6_ElasticSearchPlugins|https://issues.apache.org/jira/secure/attachment/12933735/6_ElasticSearchPlugins.jpg]
[7_ElasticSearchPluginStatus.jpg|https://issues.apache.org/jira/secure/attachment/12933736/7_ElasticSearchPluginStatus.jpg]

> Ranger supports plugin to enable, monitor and manage Elasticsearch
> --
>
> Key: RANGER-2170
> URL: https://issues.apache.org/jira/browse/RANGER-2170
> Project: Ranger
>  Issue Type: New Feature
>  Components: Ranger
>Affects Versions: master
>Reporter: Qiang Zhang
>Assignee: Qiang Zhang
>Priority: Major
>  Labels: new-feature, patch
> Attachments: 
> 0001-RANGER-2170-Ranger-supports-plugin-to-enable-monitor.patch, 
> 1_ElasticSearchServiceEntry.jpg, 2_EditElasticSearchService.jpg, 
> 3_ElasticSearchPolicies.jpg, 4_EditElasticSearchPolicy.jpg, 
> 5_ElasticSearchAuditLog.jpg, 6_ElasticSearchPlugins.jpg, 
> 7_ElasticSearchPluginStatus.jpg
>
>
> Elasticsearch is a distributed, RESTful search and analytics engine capable 
> of solving a growing number of use cases. 
> Like Apache Solr, it is also an index server based on Lucence.
> Ranger supports plugin to enable, monitor and manage Elasticsearch,
> to control index security of Elasticsearch.
> As there is X-Pack plugin for the Elasticsearch, but it is not free.
> X-Pack is an Elastic Stack extension that bundles security, alerting, 
> monitoring, reporting, 
> and graph capabilities into one easy-to-install package.
> We refer to the Indices Privileges design of X-Pack,
> by keeping the permissions consistent,
> to make user use ranger Elasticsearch plugin easily.
> Reference X-Pack Indices Privileges:
> https://www.elastic.co/guide/en/x-pack/current/security-privileges.html
> Here we develop Ranger Elasticsearch plugin, based on Elasticsearch version 
> 6.2.2.
> Elasticsearch 6.2.2 was released in February 20, 2018, reference 
> release-notes:
> https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.2.html
> Not like other system, Elasticsearch has no basic authentication, 
> it uses X-pack plugin to support basic authentication, 
> role-based access control, SSL/TLS encryption, LDAP and so on.
> Not like X-pack, our Ranger Elasticsearch plugin is designed to do 
> authorization,
> it is to control index of Elasticsearch without authentication,
> this plugin should work with other Elasticsearch plugin to authenticate users.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (RANGER-2170) Ranger supports plugin to enable, monitor and manage Elasticsearch

[Qiang Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qiang Zhang updated RANGER-2170:

Attachment: (was: 
0001-RANGER-2170-Ranger-supports-plugin-to-enable-monitor.patch)

> Ranger supports plugin to enable, monitor and manage Elasticsearch
> --
>
> Key: RANGER-2170
> URL: https://issues.apache.org/jira/browse/RANGER-2170
> Project: Ranger
>  Issue Type: New Feature
>  Components: Ranger
>Affects Versions: master
>Reporter: Qiang Zhang
>Assignee: Qiang Zhang
>Priority: Major
>  Labels: new-feature, patch
> Attachments: 
> 0001-RANGER-2170-Ranger-supports-plugin-to-enable-monitor.patch, 
> 1_ElasticSearchServiceEntry.jpg, 2_EditElasticSearchService.jpg, 
> 3_ElasticSearchPolicies.jpg, 4_EditElasticSearchPolicy.jpg, 
> 5_ElasticSearchAuditLog.jpg, 6_ElasticSearchPlugins.jpg, 
> 7_ElasticSearchPluginStatus.jpg
>
>
> Elasticsearch is a distributed, RESTful search and analytics engine capable 
> of solving a growing number of use cases. 
> Like Apache Solr, it is also an index server based on Lucence.
> Ranger supports plugin to enable, monitor and manage Elasticsearch,
> to control index security of Elasticsearch.
> As there is X-Pack plugin for the Elasticsearch, but it is not free.
> X-Pack is an Elastic Stack extension that bundles security, alerting, 
> monitoring, reporting, 
> and graph capabilities into one easy-to-install package.
> We refer to the Indices Privileges design of X-Pack,
> by keeping the permissions consistent,
> to make user use ranger Elasticsearch plugin easily.
> Reference X-Pack Indices Privileges:
> https://www.elastic.co/guide/en/x-pack/current/security-privileges.html
> Here we develop Ranger Elasticsearch plugin, based on Elasticsearch version 
> 6.2.2.
> Elasticsearch 6.2.2 was released in February 20, 2018, reference 
> release-notes:
> https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.2.html
> Not like other system, Elasticsearch has no basic authentication, 
> it uses X-pack plugin to support basic authentication, 
> role-based access control, SSL/TLS encryption, LDAP and so on.
> Not like X-pack, our Ranger Elasticsearch plugin is designed to do 
> authorization,
> it is to control index of Elasticsearch without authentication,
> this plugin should work with other Elasticsearch plugin to authenticate users.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (RANGER-2170) Ranger supports plugin to enable, monitor and manage Elasticsearch

[Qiang Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qiang Zhang updated RANGER-2170:

Attachment: 0001-RANGER-2170-Ranger-supports-plugin-to-enable-monitor.patch

> Ranger supports plugin to enable, monitor and manage Elasticsearch
> --
>
> Key: RANGER-2170
> URL: https://issues.apache.org/jira/browse/RANGER-2170
> Project: Ranger
>  Issue Type: New Feature
>  Components: Ranger
>Affects Versions: master
>Reporter: Qiang Zhang
>Assignee: Qiang Zhang
>Priority: Major
>  Labels: new-feature, patch
> Attachments: 
> 0001-RANGER-2170-Ranger-supports-plugin-to-enable-monitor.patch, 
> 1_ElasticSearchServiceEntry.jpg, 2_EditElasticSearchService.jpg, 
> 3_ElasticSearchPolicies.jpg, 4_EditElasticSearchPolicy.jpg, 
> 5_ElasticSearchAuditLog.jpg, 6_ElasticSearchPlugins.jpg, 
> 7_ElasticSearchPluginStatus.jpg
>
>
> Elasticsearch is a distributed, RESTful search and analytics engine capable 
> of solving a growing number of use cases. 
> Like Apache Solr, it is also an index server based on Lucence.
> Ranger supports plugin to enable, monitor and manage Elasticsearch,
> to control index security of Elasticsearch.
> As there is X-Pack plugin for the Elasticsearch, but it is not free.
> X-Pack is an Elastic Stack extension that bundles security, alerting, 
> monitoring, reporting, 
> and graph capabilities into one easy-to-install package.
> We refer to the Indices Privileges design of X-Pack,
> by keeping the permissions consistent,
> to make user use ranger Elasticsearch plugin easily.
> Reference X-Pack Indices Privileges:
> https://www.elastic.co/guide/en/x-pack/current/security-privileges.html
> Here we develop Ranger Elasticsearch plugin, based on Elasticsearch version 
> 6.2.2.
> Elasticsearch 6.2.2 was released in February 20, 2018, reference 
> release-notes:
> https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.2.html
> Not like other system, Elasticsearch has no basic authentication, 
> it uses X-pack plugin to support basic authentication, 
> role-based access control, SSL/TLS encryption, LDAP and so on.
> Not like X-pack, our Ranger Elasticsearch plugin is designed to do 
> authorization,
> it is to control index of Elasticsearch without authentication,
> this plugin should work with other Elasticsearch plugin to authenticate users.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68128: RANGER-2170:Ranger supports plugin to enable, monitor and manage Elasticsearch

[Qiang Zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68128/
---

(Updated 八月 14, 2018, 8:46 a.m.)


Review request for ranger, Ankita Sinha, Don Bosco Durai, Colm O hEigeartaigh, 
Gautam Borad, Madhan Neethiraj, pengjianhua, Ramesh Mani, Selvamohan Neethiraj, 
sam  rome, Venkat Ranganathan, and Velmurugan Periasamy.


Changes
---

Changes:
1.Get client IP address from rest request for audit log.
2.Fix spelling error for "RangerElasticsearchPlugin".
3.Remove joptsimple.internal.Strings.EMPTY, and use 
org.apache.commons.lang.StringUtils.EMPTY instead.


Bugs: RANGER-2170
https://issues.apache.org/jira/browse/RANGER-2170


Repository: ranger


Description
---

Elasticsearch is a distributed, RESTful search and analytics engine capable of 
solving a growing number of use cases. 
Like Apache Solr, it is also an index server based on Lucence.
Ranger supports plugin to enable, monitor and manage Elasticsearch,
to control index security of Elasticsearch.

As there is X-Pack plugin for the Elasticsearch, but it is not free.
X-Pack is an Elastic Stack extension that bundles security, alerting, 
monitoring, reporting, 
and graph capabilities into one easy-to-install package.
We refer to the Indices Privileges design of X-Pack,
by keeping the permissions consistent,
to make user use ranger Elasticsearch plugin easily.
Reference X-Pack Indices Privileges:
https://www.elastic.co/guide/en/x-pack/current/security-privileges.html

Here we develop Ranger Elasticsearch plugin, based on Elasticsearch version 
6.2.2.
Elasticsearch 6.2.2 was released in February 20, 2018, reference release-notes:
https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.2.html
Not like other system, Elasticsearch has no basic authentication, 
it uses X-pack plugin to support basic authentication, 
role-based access control, SSL/TLS encryption, LDAP and so on.
Not like X-pack, our Ranger Elasticsearch plugin is designed to do 
authorization,
it is to control index of Elasticsearch without authentication,
this plugin should work with other Elasticsearch plugin to authenticate users.


Diffs (updated)
-

  agents-common/scripts/enable-agent.sh ce0dc8c 
  agents-common/src/main/java/org/apache/ranger/plugin/client/BaseClient.java 
e654f2b 
  
agents-common/src/main/java/org/apache/ranger/plugin/store/EmbeddedServiceDefsUtil.java
 5e74da8 
  
agents-common/src/main/resources/service-defs/ranger-servicedef-elasticsearch.json
 PRE-CREATION 
  plugin-elasticsearch/.gitignore PRE-CREATION 
  plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg PRE-CREATION 
  plugin-elasticsearch/conf/ranger-elasticsearch-audit.xml PRE-CREATION 
  plugin-elasticsearch/conf/ranger-elasticsearch-security-changes.cfg 
PRE-CREATION 
  plugin-elasticsearch/conf/ranger-elasticsearch-security.xml PRE-CREATION 
  plugin-elasticsearch/conf/ranger-policymgr-ssl-changes.cfg PRE-CREATION 
  plugin-elasticsearch/conf/ranger-policymgr-ssl.xml PRE-CREATION 
  plugin-elasticsearch/pom.xml PRE-CREATION 
  plugin-elasticsearch/scripts/install.properties PRE-CREATION 
  
plugin-elasticsearch/src/main/java/org/apache/ranger/authorization/elasticsearch/authorizer/RangerElasticsearchAuthorizer.java
 PRE-CREATION 
  
plugin-elasticsearch/src/main/java/org/apache/ranger/services/elasticsearch/RangerServiceElasticsearch.java
 PRE-CREATION 
  
plugin-elasticsearch/src/main/java/org/apache/ranger/services/elasticsearch/client/ElasticsearchClient.java
 PRE-CREATION 
  
plugin-elasticsearch/src/main/java/org/apache/ranger/services/elasticsearch/client/ElasticsearchResourceMgr.java
 PRE-CREATION 
  
plugin-elasticsearch/src/main/java/org/apache/ranger/services/elasticsearch/privilege/IndexPrivilege.java
 PRE-CREATION 
  
plugin-elasticsearch/src/main/java/org/apache/ranger/services/elasticsearch/privilege/IndexPrivilegeUtils.java
 PRE-CREATION 
  pom.xml 8d7ea13 
  ranger-elasticsearch-plugin-shim/.gitignore PRE-CREATION 
  ranger-elasticsearch-plugin-shim/conf/plugin-descriptor.properties 
PRE-CREATION 
  ranger-elasticsearch-plugin-shim/conf/plugin-security.policy PRE-CREATION 
  ranger-elasticsearch-plugin-shim/pom.xml PRE-CREATION 
  
ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/authorizer/RangerElasticsearchAccessControl.java
 PRE-CREATION 
  
ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/authorizer/RangerElasticsearchAuthorizer.java
 PRE-CREATION 
  
ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/RangerElasticsearchPlugin.java
 PRE-CREATION 
  
ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/action/filter/RangerSecurityActionFilter.java
 PRE-CREATION 
  

Re: Review Request 68076: RANGER-2165: Address JPA Cache issue when policies Create, Update and Delete are done via REST API in Apache Ranger admin

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68076/#review207215
---


Ship it!




Ship It!

- Madhan Neethiraj


On Aug. 13, 2018, 7:27 p.m., Abhay Kulkarni wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68076/
> ---
> 
> (Updated Aug. 13, 2018, 7:27 p.m.)
> 
> 
> Review request for ranger, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, 
> Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2165
> https://issues.apache.org/jira/browse/RANGER-2165
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Ensure that any entities cached in JPA layer during policy/tag cache refresh 
> are cleared when refresh is completed.
> 
> 
> Diffs
> -
> 
>   
> security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 
> 3e291d50c 
>   
> security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java 
> 789068fcb 
> 
> 
> Diff: https://reviews.apache.org/r/68076/diff/4/
> 
> 
> Testing
> ---
> 
> Triggered policy/tag downloads from plugins. Ensured that policies/tags are 
> correctly retrieved and downloaded to plugins. Ensured that if policy cache 
> is refreshed as part of policy create/update, then any subsequent database 
> interactions are completed in expected time.
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>

Re: Review Request 68177: RANGER-2188: Support multiple threads to build Trie and on-lookup post-setup for Trie nodes

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68177/#review207213
---




agents-common/src/main/java/org/apache/ranger/plugin/util/RangerResourceTrie.java
Line 348 (original), 351 (patched)


Would this cause token replacement to not work when optWildcar is false? 
Please review.


- Madhan Neethiraj


On Aug. 14, 2018, 1:26 a.m., Abhay Kulkarni wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68177/
> ---
> 
> (Updated Aug. 14, 2018, 1:26 a.m.)
> 
> 
> Review request for ranger, Madhan Neethiraj, Ramesh Mani, and Velmurugan 
> Periasamy.
> 
> 
> Bugs: RANGER-2188
> https://issues.apache.org/jira/browse/RANGER-2188
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Time for building a Trie index for resources may become a bottleneck when 
> dealing with large number of resources. It is desirable to build different, 
> non-overlapping parts of Trie structure using multiple threads to reduce 
> overall build time if configured to do so.
> 
> Also, instead of building out all Trie nodes completely (that is, by 
> propagating wildcard evaluators all the way to all leaves of Trie tree) at 
> the initialization time, it is optimal to do so as a Trie-Node is accessed 
> for the first time during resource lookup.
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java
>  b12d8ff11 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineOptions.java
>  b76820cd2 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java
>  088b729fb 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/util/RangerResourceTrie.java
>  10c6faa92 
>   
> agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java
>  1d9b865b0 
> 
> 
> Diff: https://reviews.apache.org/r/68177/diff/4/
> 
> 
> Testing
> ---
> 
> Passes all unit tests
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>