[jira] [Commented] (RANGER-3935) In hdfs authorizer 'processResult', is accessType and action misused?

[Xuze Yang (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610791#comment-17610791
 ] 

Xuze Yang commented on RANGER-3935:
---

Yes, I think the code is confusing the two.
The action corresponds to FsAction, which is the real operation type of hdfs.
{code:java}
public enum FsAction {
    NONE("---"),
    EXECUTE("--x"),
    WRITE("-w-"),
    WRITE_EXECUTE("-wx"),
    READ("r--"),
    READ_EXECUTE("r-x"),
    READ_WRITE("rw-"),
    ALL("rwx");
    ...
}{code}
The accessType corresponds to the abstract operation type in the ranger policy, 
which is defined in RangerHadoopConstants
{code:java}
public static final String READ_ACCCESS_TYPE = "read";
public static final String WRITE_ACCCESS_TYPE = "write";
public static final String EXECUTE_ACCCESS_TYPE = "execute";{code}
By the way, there is a typo in code, 'ACCCESS' should be 'ACCESS'

> In hdfs authorizer 'processResult', is accessType and action misused?
> -
>
> Key: RANGER-3935
> URL: https://issues.apache.org/jira/browse/RANGER-3935
> Project: Ranger
>  Issue Type: Improvement
>  Components: audit
>Reporter: wangningito
>Priority: Major
>
> [https://github.com/apache/ranger/blob/master/hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java#L1037-L1046]
> I see action is filled into accessType while accessType field is filled with 
> action.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (RANGER-3935) In hdfs authorizer 'processResult', is accessType and action misused?

[wangningito (Jira)]

wangningito created RANGER-3935:
---

 Summary: In hdfs authorizer 'processResult', is accessType and 
action misused?
 Key: RANGER-3935
 URL: https://issues.apache.org/jira/browse/RANGER-3935
 Project: Ranger
  Issue Type: Improvement
  Components: audit
Reporter: wangningito


[https://github.com/apache/ranger/blob/master/hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java#L1037-L1046]



I see action is filled into accessType while accessType field is filled with 
action.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Review Request 73694: RANGER-3500: Ranger policy list doesn't support sorting by field

[Xuze Yang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73694/
---

(Updated 九月 29, 2022, 2:37 a.m.)


Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Mahesh Bandal, 
Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, Vishal Suvagia, and 
Velmurugan Periasamy.


Bugs: RANGER-3500
https://issues.apache.org/jira/browse/RANGER-3500


Repository: ranger


Description
---

modify the getServicePolicies() method to support sorting by field and 
sortType(i.e. asc or desc). The changes are as following:
1. Add sortBy and sortType to searchFilter
2. Support sorting by sortType("desc" means in decreasing order, otherwise in 
increasing order)
3. Delete a redundant line of code


Diffs
-

  
agents-common/src/main/java/org/apache/ranger/plugin/store/AbstractPredicateUtil.java
 38d6b03ea 
  security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 
f13cef71d 


Diff: https://reviews.apache.org/r/73694/diff/1/


Testing
---

1. compile and package the code
2. deploy ranger
3. generate hdfs policies and get the hdfs policyList via postman using url 
like 
"http://192.168.0.12:6080/service/plugins/policies/service/2?sortBy=policyName=desc=default-Hdfs;
4. the return policyList is sorted according our specified sortBy and sortType


File Attachments (updated)


0002-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch
  
https://reviews.apache.org/media/uploaded/files/2022/09/29/61ba9494-1dc8-4b67-80ed-44c7e9f313a1__0002-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch


Thanks,

Xuze Yang

[jira] [Commented] (RANGER-3500) Ranger policy list doesn't support sorting by field

[Xuze Yang (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610773#comment-17610773
 ] 

Xuze Yang commented on RANGER-3500:
---

The changes are as following:
1. Add sortBy and sortType to searchFilter
2. Support sorting by sortType("desc" means in decreasing order, otherwise in 
increasing order)
3. Delete a redundant line of code(In AbstractPredicateUtil#static filed)

When the ranger web ui calls the getServicePolicies(), the sortBy and sortType 
parameters are not passed by default, the change is consistent with the 
previous return logic.

The patch is updated based on ranger-2.3.0.

Please find attached patch and review PR as per your convenience. Thanks.

CC: [~madhan] [~kirbyzhou] [~kulkabhay] [~rmani] [~bpatel] 

 

> Ranger policy list doesn't support sorting by field
> ---
>
> Key: RANGER-3500
> URL: https://issues.apache.org/jira/browse/RANGER-3500
> Project: Ranger
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 2.1.0
>Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
> Attachments: 
> 0001-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch, 
> 0002-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch
>
>
> When getting the ranger policy list, we may want to sort the returned policy 
> list according to certain fields, such as policyId, policyName and etc. But 
> case shows that adding the parameters sortBy and sortType to the url has no 
> effect (eg: 
> [http://192.168.0.12:6080/service/plugins/]policies/service/2?sortBy=policyName=desc=default-Hdfs).
>  I look through the source code and find that code supports sorting by 
> fields, but due to some code bugs, it did not really take effect. 
> The main reason for the problem is that when the SearchFilter is copied 
> deeply, only the params is copied, the sortBy and sortType attributes is 
> omitted. The code show as follows:
> {code:java}
> // ServiceDBStore # getServicePolicies()
> Map paramsCopy = new HashMap<>(filter.getParams()); 
> SearchFilter searchFilter = new SearchFilter(paramsCopy); {code}
>  
> {code:java}
> // SearchFilter.java
> private Map params;
> private int startIndex;
> private int maxRows= Integer.MAX_VALUE;
> private boolean getCount   = true;
> private String  sortBy;
> private String  sortType;
> public SearchFilter(Map values) {
>setParams(values);
> }
> public void setParams(Map params) {
>this.params = params;
> }  {code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (RANGER-3500) Ranger policy list doesn't support sorting by field

[Xuze Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuze Yang updated RANGER-3500:
--
Attachment: 0002-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch

> Ranger policy list doesn't support sorting by field
> ---
>
> Key: RANGER-3500
> URL: https://issues.apache.org/jira/browse/RANGER-3500
> Project: Ranger
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 2.1.0
>Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
> Attachments: 
> 0001-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch, 
> 0002-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch
>
>
> When getting the ranger policy list, we may want to sort the returned policy 
> list according to certain fields, such as policyId, policyName and etc. But 
> case shows that adding the parameters sortBy and sortType to the url has no 
> effect (eg: 
> [http://192.168.0.12:6080/service/plugins/]policies/service/2?sortBy=policyName=desc=default-Hdfs).
>  I look through the source code and find that code supports sorting by 
> fields, but due to some code bugs, it did not really take effect. 
> The main reason for the problem is that when the SearchFilter is copied 
> deeply, only the params is copied, the sortBy and sortType attributes is 
> omitted. The code show as follows:
> {code:java}
> // ServiceDBStore # getServicePolicies()
> Map paramsCopy = new HashMap<>(filter.getParams()); 
> SearchFilter searchFilter = new SearchFilter(paramsCopy); {code}
>  
> {code:java}
> // SearchFilter.java
> private Map params;
> private int startIndex;
> private int maxRows= Integer.MAX_VALUE;
> private boolean getCount   = true;
> private String  sortBy;
> private String  sortType;
> public SearchFilter(Map values) {
>setParams(values);
> }
> public void setParams(Map params) {
>this.params = params;
> }  {code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (RANGER-3500) Ranger policy list doesn't support sorting by field

[Xuze Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuze Yang updated RANGER-3500:
--
Description: 
When getting the ranger policy list, we may want to sort the returned policy 
list according to certain fields, such as policyId, policyName and etc. But 
case shows that adding the parameters sortBy and sortType to the url has no 
effect (eg: 
[http://192.168.0.12:6080/service/plugins/]policies/service/2?sortBy=policyName=desc=default-Hdfs).
 I look through the source code and find that code supports sorting by fields, 
but due to some code bugs, it did not really take effect. 
The main reason for the problem is that when the SearchFilter is copied deeply, 
only the params is copied, the sortBy and sortType attributes is omitted. The 
code show as follows:
{code:java}
// ServiceDBStore # getServicePolicies()
Map paramsCopy = new HashMap<>(filter.getParams()); 
SearchFilter searchFilter = new SearchFilter(paramsCopy); {code}
 
{code:java}
// SearchFilter.java
private Map params;
private int startIndex;
private int maxRows= Integer.MAX_VALUE;
private boolean getCount   = true;
private String  sortBy;
private String  sortType;

public SearchFilter(Map values) {
   setParams(values);
}

public void setParams(Map params) {
   this.params = params;
}  {code}
 

  was:
When getting the ranger policy list, we may want to sort the returned policy 
list according to certain fields, such as policyId, policyName and etc. But 
case shows that adding the parameters sortBy and sortType to the url has no 
effect (eg: 
[http://192.168.0.12:6080/service/plugins/]policies/service/2?sortBy=policyName=desc=default-Hdfs).
 I look through the source code and find that code supports sorting by fields, 
but due to some code bugs, it did not really take effect. 
 The main reason for the problem is that when the SearchFilter is copied 
deeply, only the params is copied, the sortBy and sortType attributes is 
omitted. The code show as follows:
{code:java}
Map paramsCopy  = new HashMap<>(filter.getParams());
SearchFilter   searchFilter = new SearchFilter(paramsCopy);
{code}


> Ranger policy list doesn't support sorting by field
> ---
>
> Key: RANGER-3500
> URL: https://issues.apache.org/jira/browse/RANGER-3500
> Project: Ranger
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 2.1.0
>Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
> Attachments: 
> 0001-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch
>
>
> When getting the ranger policy list, we may want to sort the returned policy 
> list according to certain fields, such as policyId, policyName and etc. But 
> case shows that adding the parameters sortBy and sortType to the url has no 
> effect (eg: 
> [http://192.168.0.12:6080/service/plugins/]policies/service/2?sortBy=policyName=desc=default-Hdfs).
>  I look through the source code and find that code supports sorting by 
> fields, but due to some code bugs, it did not really take effect. 
> The main reason for the problem is that when the SearchFilter is copied 
> deeply, only the params is copied, the sortBy and sortType attributes is 
> omitted. The code show as follows:
> {code:java}
> // ServiceDBStore # getServicePolicies()
> Map paramsCopy = new HashMap<>(filter.getParams()); 
> SearchFilter searchFilter = new SearchFilter(paramsCopy); {code}
>  
> {code:java}
> // SearchFilter.java
> private Map params;
> private int startIndex;
> private int maxRows= Integer.MAX_VALUE;
> private boolean getCount   = true;
> private String  sortBy;
> private String  sortType;
> public SearchFilter(Map values) {
>setParams(values);
> }
> public void setParams(Map params) {
>this.params = params;
> }  {code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Review Request 74144: RANGER-3934: optimization in loading of tags into cache

[Abhay Kulkarni]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74144/#review224719
---




agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java
Line 107 (original), 101 (patched)


This conversion will be done every time applyDeltas() is called. This may 
be expensive operation. Is it possible to cache this map in ServiceTags but 
ignore it when serializing ServiceTags object?



agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java
Line 142 (original), 130 (patched)


If any resources are deleted or updated, this code iterates over all 
service-resources. Will this be expensive, as before?


- Abhay Kulkarni


On Sept. 28, 2022, 10:58 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74144/
> ---
> 
> (Updated Sept. 28, 2022, 10:58 p.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Kishor Gollapalliwar, Abhay 
> Kulkarni, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, 
> Subhrat Chaudhary, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-3934
> https://issues.apache.org/jira/browse/RANGER-3934
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> - updated several retrieval of XXService JPA object by replacing with 
> retrival of serviceId
> - replaced several instances of Map with Set, as these 
> instances had same value for key and value
> - avoided expensive calls to RangerTagResourceMapService.getByTagId(tagId) 
> and RangerTagResourceMapService.getTagIdsForResourceId(serviceResourceId)
> - updated several methods in XXTagResourceMapDao to avoid loading of 
> XXTagResourceMap JPA object; instead created XXTagResourceMap object from 
> individial fields retrieved from query
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java
>  8d9241c1c 
>   security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java 
> d8154b7de 
>   
> security-admin/src/main/java/org/apache/ranger/db/XXRMSServiceResourceDao.java
>  afa754ba2 
>   security-admin/src/main/java/org/apache/ranger/db/XXServiceDao.java 
> 3cc3d9cef 
>   security-admin/src/main/java/org/apache/ranger/db/XXTagResourceMapDao.java 
> 3f8b5b718 
>   
> security-admin/src/main/java/org/apache/ranger/patch/PatchForAtlasServiceDefUpdate_J10013.java
>  b0f71e138 
>   security-admin/src/main/java/org/apache/ranger/rest/TagREST.java c7cf3bfb8 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml e4a2354b0 
> 
> 
> Diff: https://reviews.apache.org/r/74144/diff/2/
> 
> 
> Testing
> ---
> 
> - with Ranger database having ~1m service-resources, ~2m tags and delta of 
> 20k resource & 40k tags:
>   -- before these optmizations, tag-cache update from delta didn't complete 
> event after a long time (1h35m)
>   -- with these optmizations, tag-cache update from the same delta completed 
> within 10 minutes
> - there is scope for further optimization 
> RangerServiceTagsDeltaUtil.applyDelta() - as this took almost 97% time in 
> updating the cache (588 seconds); compare this to 
> TagDBStore.getServiceTagsDelta() which completed in 13 seeconds
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 74144: RANGER-3934: optimization in loading of tags into cache

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74144/
---

(Updated Sept. 28, 2022, 10:58 p.m.)


Review request for ranger, Ankita Sinha, Kishor Gollapalliwar, Abhay Kulkarni, 
Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, Subhrat 
Chaudhary, and Velmurugan Periasamy.


Changes
---

addressed review comment to further optimize 
RangerServiceTagsDeltaUtil.applyDelta()


Bugs: RANGER-3934
https://issues.apache.org/jira/browse/RANGER-3934


Repository: ranger


Description
---

- updated several retrieval of XXService JPA object by replacing with retrival 
of serviceId
- replaced several instances of Map with Set, as these 
instances had same value for key and value
- avoided expensive calls to RangerTagResourceMapService.getByTagId(tagId) and 
RangerTagResourceMapService.getTagIdsForResourceId(serviceResourceId)
- updated several methods in XXTagResourceMapDao to avoid loading of 
XXTagResourceMap JPA object; instead created XXTagResourceMap object from 
individial fields retrieved from query


Diffs (updated)
-

  
agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java
 8d9241c1c 
  security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java d8154b7de 
  
security-admin/src/main/java/org/apache/ranger/db/XXRMSServiceResourceDao.java 
afa754ba2 
  security-admin/src/main/java/org/apache/ranger/db/XXServiceDao.java 3cc3d9cef 
  security-admin/src/main/java/org/apache/ranger/db/XXTagResourceMapDao.java 
3f8b5b718 
  
security-admin/src/main/java/org/apache/ranger/patch/PatchForAtlasServiceDefUpdate_J10013.java
 b0f71e138 
  security-admin/src/main/java/org/apache/ranger/rest/TagREST.java c7cf3bfb8 
  security-admin/src/main/resources/META-INF/jpa_named_queries.xml e4a2354b0 


Diff: https://reviews.apache.org/r/74144/diff/2/

Changes: https://reviews.apache.org/r/74144/diff/1-2/


Testing
---

- with Ranger database having ~1m service-resources, ~2m tags and delta of 20k 
resource & 40k tags:
  -- before these optmizations, tag-cache update from delta didn't complete 
event after a long time (1h35m)
  -- with these optmizations, tag-cache update from the same delta completed 
within 10 minutes
- there is scope for further optimization 
RangerServiceTagsDeltaUtil.applyDelta() - as this took almost 97% time in 
updating the cache (588 seconds); compare this to 
TagDBStore.getServiceTagsDelta() which completed in 13 seeconds


Thanks,

Madhan Neethiraj

Re: Review Request 74144: RANGER-3934: optimization in loading of tags into cache

[Madhan Neethiraj]

> On Sept. 28, 2022, 6:12 p.m., Abhay Kulkarni wrote:
> > agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java
> > Line 102 (original), 105 (patched)
> > 
> >
> > This needs a review to reduce the M x N complexity. If there is a map 
> > of  maintained (in memory) for existing 
> > service-tags, then the loop at line 115 may reduce to a map lookup by 
> > resource-id.
> 
> Madhan Neethiraj wrote:
> Yes. Updating this block to replace inner loop with a map look up will 
> result in significant improvement. I was planning to address that in a 
> subsequent patch. Without the current fix, delta processing didn't complete 
> even after 1h35m; with the the delta processing gets to completion within few 
> minutes.
> 
> Also, I think we should review in-place updating of serviceTags instance 
> passed as argument. This would be an issue if that instance is used in 
> another thread while it is being updated.
> 
> Abhay Kulkarni wrote:
> I dont think serviceTags instance passed as an argument will be used in 
> other thread, given that it is called within a lock.tryLock block from 
> getLatestOrCached() function. (Caveat: Of course, there is a case where the 
> tryLock times out while the updates to cached service-tags are going on. It 
> is also called from RangerTagRefresher.populateTag() in the plugins. If the 
> tag download interval is too small, then there is a possibility of two 
> threads accessing service-tags concurrently.)

Consider the following case:
- thread-1 called getLatestOrCached() and obtained ServiceTags instance; and is 
using this instance
- thread-2 calls getLatestOrCached() within a very short period, and there were 
few tag details to be applied. In this case, ServiceTags instance being used by 
thread-1 will be updated by thread-2 - right?


- Madhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74144/#review224714
---


On Sept. 28, 2022, 5:44 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74144/
> ---
> 
> (Updated Sept. 28, 2022, 5:44 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Kishor Gollapalliwar, Abhay 
> Kulkarni, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, 
> Subhrat Chaudhary, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-3934
> https://issues.apache.org/jira/browse/RANGER-3934
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> - updated several retrieval of XXService JPA object by replacing with 
> retrival of serviceId
> - replaced several instances of Map with Set, as these 
> instances had same value for key and value
> - avoided expensive calls to RangerTagResourceMapService.getByTagId(tagId) 
> and RangerTagResourceMapService.getTagIdsForResourceId(serviceResourceId)
> - updated several methods in XXTagResourceMapDao to avoid loading of 
> XXTagResourceMap JPA object; instead created XXTagResourceMap object from 
> individial fields retrieved from query
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java
>  8d9241c1c 
>   security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java 
> d8154b7de 
>   
> security-admin/src/main/java/org/apache/ranger/db/XXRMSServiceResourceDao.java
>  afa754ba2 
>   security-admin/src/main/java/org/apache/ranger/db/XXServiceDao.java 
> 3cc3d9cef 
>   security-admin/src/main/java/org/apache/ranger/db/XXTagResourceMapDao.java 
> 3f8b5b718 
>   
> security-admin/src/main/java/org/apache/ranger/patch/PatchForAtlasServiceDefUpdate_J10013.java
>  b0f71e138 
>   security-admin/src/main/java/org/apache/ranger/rest/TagREST.java c7cf3bfb8 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml e4a2354b0 
> 
> 
> Diff: https://reviews.apache.org/r/74144/diff/1/
> 
> 
> Testing
> ---
> 
> - with Ranger database having ~1m service-resources, ~2m tags and delta of 
> 20k resource & 40k tags:
>   -- before these optmizations, tag-cache update from delta didn't complete 
> event after a long time (1h35m)
>   -- with these optmizations, tag-cache update from the same delta completed 
> within 10 minutes
> - there is scope for further optimization 
> RangerServiceTagsDeltaUtil.applyDelta() - as this took almost 97% time in 
> updating the cache (588 seconds); compare this to 
> TagDBStore.getServiceTagsDelta() which completed in 13 seeconds
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 74144: RANGER-3934: optimization in loading of tags into cache

[Abhay Kulkarni]

> On Sept. 28, 2022, 6:12 p.m., Abhay Kulkarni wrote:
> > agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java
> > Line 102 (original), 105 (patched)
> > 
> >
> > This needs a review to reduce the M x N complexity. If there is a map 
> > of  maintained (in memory) for existing 
> > service-tags, then the loop at line 115 may reduce to a map lookup by 
> > resource-id.
> 
> Madhan Neethiraj wrote:
> Yes. Updating this block to replace inner loop with a map look up will 
> result in significant improvement. I was planning to address that in a 
> subsequent patch. Without the current fix, delta processing didn't complete 
> even after 1h35m; with the the delta processing gets to completion within few 
> minutes.
> 
> Also, I think we should review in-place updating of serviceTags instance 
> passed as argument. This would be an issue if that instance is used in 
> another thread while it is being updated.

I dont think serviceTags instance passed as an argument will be used in other 
thread, given that it is called within a lock.tryLock block from 
getLatestOrCached() function. (Caveat: Of course, there is a case where the 
tryLock times out while the updates to cached service-tags are going on. It is 
also called from RangerTagRefresher.populateTag() in the plugins. If the tag 
download interval is too small, then there is a possibility of two threads 
accessing service-tags concurrently.)


> On Sept. 28, 2022, 6:12 p.m., Abhay Kulkarni wrote:
> > security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java
> > Line 1251 (original), 1250 (patched)
> > 
> >
> > Although this change reduces dependency on JPA cache behavior, it also 
> > removes one consistency check on the sanity of deltas and the state of the 
> > database. Perhaps, performing all of these computations/database 
> > interactions in a dedicated, read-only new transaction will obviate need 
> > for such peep-hole optimizations. This also applies to changes around line 
> > #1269.
> 
> Madhan Neethiraj wrote:
> Yes. Use of read-only transactions can help avoid JPA overheads. However, 
> wouldn't the optimizations introduced in this patch still be helpful to avoid 
> overheads?

Yes, the current optimizations will help, but for the sake of maintainability 
in the face of future changes, the real solution may be to explore using 
separate, read-only transaction for delta creation.


- Abhay


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74144/#review224714
---


On Sept. 28, 2022, 5:44 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74144/
> ---
> 
> (Updated Sept. 28, 2022, 5:44 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Kishor Gollapalliwar, Abhay 
> Kulkarni, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, 
> Subhrat Chaudhary, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-3934
> https://issues.apache.org/jira/browse/RANGER-3934
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> - updated several retrieval of XXService JPA object by replacing with 
> retrival of serviceId
> - replaced several instances of Map with Set, as these 
> instances had same value for key and value
> - avoided expensive calls to RangerTagResourceMapService.getByTagId(tagId) 
> and RangerTagResourceMapService.getTagIdsForResourceId(serviceResourceId)
> - updated several methods in XXTagResourceMapDao to avoid loading of 
> XXTagResourceMap JPA object; instead created XXTagResourceMap object from 
> individial fields retrieved from query
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java
>  8d9241c1c 
>   security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java 
> d8154b7de 
>   
> security-admin/src/main/java/org/apache/ranger/db/XXRMSServiceResourceDao.java
>  afa754ba2 
>   security-admin/src/main/java/org/apache/ranger/db/XXServiceDao.java 
> 3cc3d9cef 
>   security-admin/src/main/java/org/apache/ranger/db/XXTagResourceMapDao.java 
> 3f8b5b718 
>   
> security-admin/src/main/java/org/apache/ranger/patch/PatchForAtlasServiceDefUpdate_J10013.java
>  b0f71e138 
>   security-admin/src/main/java/org/apache/ranger/rest/TagREST.java c7cf3bfb8 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml e4a2354b0 
> 
> 
> Diff: https://reviews.apache.org/r/74144/diff/1/
> 
> 
> Testing
> ---
> 
> - with Ranger database having ~1m service-resources, ~2m tags and delta of 

Re: Review Request 74144: RANGER-3934: optimization in loading of tags into cache

[Madhan Neethiraj]

> On Sept. 28, 2022, 6:12 p.m., Abhay Kulkarni wrote:
> > agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java
> > Line 102 (original), 105 (patched)
> > 
> >
> > This needs a review to reduce the M x N complexity. If there is a map 
> > of  maintained (in memory) for existing 
> > service-tags, then the loop at line 115 may reduce to a map lookup by 
> > resource-id.

Yes. Updating this block to replace inner loop with a map look up will result 
in significant improvement. I was planning to address that in a subsequent 
patch. Without the current fix, delta processing didn't complete even after 
1h35m; with the the delta processing gets to completion within few minutes.

Also, I think we should review in-place updating of serviceTags instance passed 
as argument. This would be an issue if that instance is used in another thread 
while it is being updated.


> On Sept. 28, 2022, 6:12 p.m., Abhay Kulkarni wrote:
> > security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java
> > Lines 1217 (patched)
> > 
> >
> > This change may result in processing the same tag or resource id 
> > multiple times if the same object appears multiple times in the list of 
> > deltas. Does the change cause significant performance improvement? If not, 
> > please consider either reverting it or replacing the Map to a HashSet to 
> > ensure that each object appears at most once in the subsequent loop.

This change replaces use of Map with Set - since the map was 
populated with the same value for key and value. This change doesn't introduce 
any change in behavior.


> On Sept. 28, 2022, 6:12 p.m., Abhay Kulkarni wrote:
> > security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java
> > Line 1251 (original), 1250 (patched)
> > 
> >
> > Although this change reduces dependency on JPA cache behavior, it also 
> > removes one consistency check on the sanity of deltas and the state of the 
> > database. Perhaps, performing all of these computations/database 
> > interactions in a dedicated, read-only new transaction will obviate need 
> > for such peep-hole optimizations. This also applies to changes around line 
> > #1269.

Yes. Use of read-only transactions can help avoid JPA overheads. However, 
wouldn't the optimizations introduced in this patch still be helpful to avoid 
overheads?


> On Sept. 28, 2022, 6:12 p.m., Abhay Kulkarni wrote:
> > security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java
> > Lines 1282 (patched)
> > 
> >
> > Please review if this code fragment is required after reviewing the 
> > following API and its callers. The  changes to each of tags, resources and 
> > tag-resource-mappings may be completely kept track in the change-records 
> > created in the delta-tables.
> > 
> > ---
> > 
> > XXServiceVersionInfoDao.updateTagVersionAndTagUpdateTime().

Earlier version used rangerTagResourceMapService.getTagIdsForResourceId() to 
get IDs of tags associated with a given resources. This patch replaces this 
logic to obtain tag IDs from XXServiceResource.tags field.

It is not clear hwo updateTagVersionAndTagUpdateTime() can help here. Can you 
please add details? Thanks.


- Madhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74144/#review224714
---


On Sept. 28, 2022, 5:44 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74144/
> ---
> 
> (Updated Sept. 28, 2022, 5:44 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Kishor Gollapalliwar, Abhay 
> Kulkarni, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, 
> Subhrat Chaudhary, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-3934
> https://issues.apache.org/jira/browse/RANGER-3934
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> - updated several retrieval of XXService JPA object by replacing with 
> retrival of serviceId
> - replaced several instances of Map with Set, as these 
> instances had same value for key and value
> - avoided expensive calls to RangerTagResourceMapService.getByTagId(tagId) 
> and RangerTagResourceMapService.getTagIdsForResourceId(serviceResourceId)
> - updated several methods in XXTagResourceMapDao to avoid loading of 
> XXTagResourceMap JPA object; instead created XXTagResourceMap object from 
> individial fields retrieved from query
> 
> 
> Diffs
> -
> 

Re: Review Request 74144: RANGER-3934: optimization in loading of tags into cache

[Abhay Kulkarni]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74144/#review224714
---




agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java
Line 102 (original), 105 (patched)


This needs a review to reduce the M x N complexity. If there is a map of 
 maintained (in memory) for existing 
service-tags, then the loop at line 115 may reduce to a map lookup by 
resource-id.



security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java
Lines 1217 (patched)


This change may result in processing the same tag or resource id multiple 
times if the same object appears multiple times in the list of deltas. Does the 
change cause significant performance improvement? If not, please consider 
either reverting it or replacing the Map to a HashSet to ensure that each 
object appears at most once in the subsequent loop.



security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java
Line 1251 (original), 1250 (patched)


Although this change reduces dependency on JPA cache behavior, it also 
removes one consistency check on the sanity of deltas and the state of the 
database. Perhaps, performing all of these computations/database interactions 
in a dedicated, read-only new transaction will obviate need for such peep-hole 
optimizations. This also applies to changes around line #1269.



security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java
Lines 1282 (patched)


Please review if this code fragment is required after reviewing the 
following API and its callers. The  changes to each of tags, resources and 
tag-resource-mappings may be completely kept track in the change-records 
created in the delta-tables.

---

XXServiceVersionInfoDao.updateTagVersionAndTagUpdateTime().


- Abhay Kulkarni


On Sept. 28, 2022, 5:44 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74144/
> ---
> 
> (Updated Sept. 28, 2022, 5:44 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Kishor Gollapalliwar, Abhay 
> Kulkarni, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, 
> Subhrat Chaudhary, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-3934
> https://issues.apache.org/jira/browse/RANGER-3934
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> - updated several retrieval of XXService JPA object by replacing with 
> retrival of serviceId
> - replaced several instances of Map with Set, as these 
> instances had same value for key and value
> - avoided expensive calls to RangerTagResourceMapService.getByTagId(tagId) 
> and RangerTagResourceMapService.getTagIdsForResourceId(serviceResourceId)
> - updated several methods in XXTagResourceMapDao to avoid loading of 
> XXTagResourceMap JPA object; instead created XXTagResourceMap object from 
> individial fields retrieved from query
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java
>  8d9241c1c 
>   security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java 
> d8154b7de 
>   
> security-admin/src/main/java/org/apache/ranger/db/XXRMSServiceResourceDao.java
>  afa754ba2 
>   security-admin/src/main/java/org/apache/ranger/db/XXServiceDao.java 
> 3cc3d9cef 
>   security-admin/src/main/java/org/apache/ranger/db/XXTagResourceMapDao.java 
> 3f8b5b718 
>   
> security-admin/src/main/java/org/apache/ranger/patch/PatchForAtlasServiceDefUpdate_J10013.java
>  b0f71e138 
>   security-admin/src/main/java/org/apache/ranger/rest/TagREST.java c7cf3bfb8 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml e4a2354b0 
> 
> 
> Diff: https://reviews.apache.org/r/74144/diff/1/
> 
> 
> Testing
> ---
> 
> - with Ranger database having ~1m service-resources, ~2m tags and delta of 
> 20k resource & 40k tags:
>   -- before these optmizations, tag-cache update from delta didn't complete 
> event after a long time (1h35m)
>   -- with these optmizations, tag-cache update from the same delta completed 
> within 10 minutes
> - there is scope for further optimization 
> RangerServiceTagsDeltaUtil.applyDelta() - as this took almost 97% time in 
> updating the cache (588 seconds); compare this to 
> TagDBStore.getServiceTagsDelta() which completed in 13 seeconds
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 74147: Ranger KMS generates master key on every restart before inserting into DB

[Ramesh Mani]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74147/#review224715
---




kms/src/main/java/org/apache/hadoop/crypto/key/RangerMasterKey.java
Lines 412 (patched)


can the daoManager be null if so do null check.


- Ramesh Mani


On Sept. 28, 2022, 2:43 p.m., Vikas Kumar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74147/
> ---
> 
> (Updated Sept. 28, 2022, 2:43 p.m.)
> 
> 
> Review request for ranger, Dhaval Shah and Sailaja Polavarapu.
> 
> 
> Bugs: RANGER-3927
> https://issues.apache.org/jira/browse/RANGER-3927
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Issue: Master Key(MK) was being first generated and then it was checking into 
> DB that if MK actually exists on each restart of the KMS process. If exists, 
> it simply returns. The same check could have been added before regenerating 
> the MK to avoid this unnecessary execution.
> 
> FIX: Added the same check before calling the method 
> RangerMasterKey.saveEncryptedMK(). Also removed the check from 
> saveEncryptedMK().
> 
> New logs: If MK doesn't exist, then 
> 
> logger.info("Master Key doesn't exist in DB, Generating the Master Key");
> 
> if MK already exists, then
> 
> logger.debug("Ranger Master Key already exists in the DB, returning.");
> 
> 
> Diffs
> -
> 
>   kms/src/main/java/org/apache/hadoop/crypto/key/RangerMasterKey.java 
> c37e98ee5 
> 
> 
> Diff: https://reviews.apache.org/r/74147/diff/1/
> 
> 
> Testing
> ---
> 
> verified through logs on restart of the KMS service. And I got "Ranger Master 
> Key already exists in the DB, returning." It was not going to generate and 
> save the MK into DB.
> 
> 
> Thanks,
> 
> Vikas Kumar
> 
>

Re: Review Request 74124: RANGER-3913: Reduce number of calls to FilenameUtils.wildcardMatch() when evaluating resource matching

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74124/#review224713
---


Ship it!




Ship It!

- Madhan Neethiraj


On Sept. 27, 2022, 6:13 p.m., Abhay Kulkarni wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74124/
> ---
> 
> (Updated Sept. 27, 2022, 6:13 p.m.)
> 
> 
> Review request for ranger, Madhan Neethiraj, Ramesh Mani, Sailaja Polavarapu, 
> and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-3913
> https://issues.apache.org/jira/browse/RANGER-3913
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> When evaluating resource matching for a path-type resource in a recursive 
> policy containing wildcards, several calls may be made to Hadoop library 
> function FilenameUtils.wildcardMatch(). These calls may be expensive and 
> minimizing such calls may help improve policy evaluation performance.
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/resourcematcher/RangerPathResourceMatcher.java
>  a95b1f73f 
> 
> 
> Diff: https://reviews.apache.org/r/74124/diff/2/
> 
> 
> Testing
> ---
> 
> Compiled clean, ran all unit tests successfully.
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>

[jira] [Commented] (RANGER-3900) Roles deletion Takes time in Apache Ranger when there are more users,groups,roles

[Ramachandran (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610623#comment-17610623
 ] 

Ramachandran commented on RANGER-3900:
--

The review is available here :[https://reviews.apache.org/r/74148]

cc >> [~mad...@apache.org]  [~pradeep]  [~abhay]  [~vel] 

> Roles deletion Takes time in Apache Ranger when there are more 
> users,groups,roles
> -
>
> Key: RANGER-3900
> URL: https://issues.apache.org/jira/browse/RANGER-3900
> Project: Ranger
>  Issue Type: Improvement
>  Components: Ranger
>Affects Versions: 3.0.0
>Reporter: Ramachandran
>Assignee: Ramachandran
>Priority: Major
>
> *{color:#ff}Role Deletion Steps in Apache Ranger:{color}*{color:#ff} 
> {color}
>  
> {code:java}
>  
> 1. Getting XXRole by roleName:
> XXRole xxRole = daoMgr.getXXRole().findByRoleName(roleName); → 1DB Read call
> 2. Ensuring role is not in policy 
> Long roleRefPolicyCount = 
> daoMgr.getXXPolicyRefRole().findRoleRefPolicyCount(roleName); → 1DB Read call
> 3. Ensuring role is not in another role
> Long roleRefRoleCount = 
> daoMgr.getXXRoleRefRole().findRoleRefRoleCount(roleName);→ 1DB Read call
> 4. Fetching RangerRole by id
> RangerRole role = roleService.read(xxRole.getId()); → 1DB Read call
> 5. Dereferencing all the users from the role.
>  for (XXRoleRefUser xxRoleRefUser : xRoleUserDao.findByRoleId(roleId)) { → 
> 1DB Read call
> xRoleUserDao.remove(xxRoleRefUser);   → 1DB Write call for each xxRoleRefUser
> }
> 6. Dereferencing all the groups from the role
> for (XXRoleRefGroup xxRoleRefGroup : xRoleGroupDao.findByRoleId(roleId)) { → 
> 1DB Read call
> xRoleGroupDao.remove(xxRoleRefGroup);  → 1DB Write call for each 
> xxRoleRefGroup
> }
> 7. Dereferencing all other roles from the role.
> for (XXRoleRefRole xxRoleRefRole : xRoleRoleDao.findByRoleId(roleId)) {→ 1DB 
> Read call
> xRoleRoleDao.remove(xxRoleRefRole);→ 1DB Write call for each xxRoleRefRole
> }
> 8. Delete the rangerRole in DB
>   roleService.delete(role);  → 1DB Write call
> 9. Create TrxLog
> bizUtil.createTrxLog(trxLogList) → 1DB Write call
>  
> {code}
>  
> *{color:#ff}Total number of DB calls involved for the below role deletion 
> in Apache Ranger:{color}*
>  
> *Deleting the role which contains more users:*
> 1. role contains 100 users 
> ||DB Write count||DB Read count||DB Batch Write count||Time is taken to 
> delete the role||
> |102|6|0|1.01 seconds|
>  
> 1 role contains 500 users  
> ||DB Write count||DB Read count||DB Batch Write count||Time is taken to 
> delete the role||
> |502|6|0| 2.241 seconds|
> 1 role contains 1000 users  
> ||DB Write count||DB Read count||DB Batch Write count||Time is taken to 
> delete the role||
> |1002|6|0|3.14 seconds|
>  
> *Deleting the user (Which was associated with the role . Now that role got 
> deleted)*
>  
> ||Number of users ||Time is taken to delete each user||
> |1000| 0.8 to 1.5  seconds|
>  
> *Deleting the user (Which was not associated with any role)*
> ||Number of users ||Time is taken to delete each user||
> |1000| 0.8 to 1.2  seconds|
> {color:#FF}Recommendation :{color}
> When we reduce the number of users added to the role -→ Total number of DB 
> calls will be reduced 
> Instead of adding roles to the users, can we add those users into the group. 
> And then link that group to the role
> if we use groups instead of directly adding users into the role, the total 
> number of DB write call will be reduced to 2 from 1002  for 1k users 
>  
> {color:#ff}Proposal{color}:
> {color:#ff}We can try DB batch write call instead of 1 DB write call for 
> every user deference {color}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Review Request 74148: Roles deletion Takes time in Apache Ranger when there are more users, groups, roles

[Ramachandran Krishnan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74148/
---

Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Nikhil P, Pradeep 
Agrawal, Ramesh Mani, Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan 
Periasamy.


Bugs: RANGER-3900
https://issues.apache.org/jira/browse/RANGER-3900


Repository: ranger


Description
---

We used DB batch write call instead of 1 DB write call for every 
user,role,group deference


Diffs
-

  security-admin/src/main/java/org/apache/ranger/biz/RoleRefUpdater.java 
56f7ec4c8 
  security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 
fc56ff88b 
  security-admin/src/main/java/org/apache/ranger/db/XXRoleRefGroupDao.java 
eb470f260 
  security-admin/src/main/java/org/apache/ranger/db/XXRoleRefRoleDao.java 
8f6fc8cfa 
  security-admin/src/main/java/org/apache/ranger/db/XXRoleRefUserDao.java 
ba9fb494b 
  security-admin/src/main/resources/META-INF/jpa_named_queries.xml e4a2354b0 


Diff: https://reviews.apache.org/r/74148/diff/1/


Testing
---

test with 1 Role contains 1k other roles and 1k groups 
while deleting the role it took 11.577 seconds
After fix
it took 1.021 seconds


Thanks,

Ramachandran Krishnan

[jira] [Updated] (RANGER-3900) Roles deletion Takes time in Apache Ranger when there are more users,groups,roles

[Ramachandran (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramachandran updated RANGER-3900:
-
Summary: Roles deletion Takes time in Apache Ranger when there are more 
users,groups,roles  (was: Roles deletion Takes time in Apache Ranger when there 
are more users)

> Roles deletion Takes time in Apache Ranger when there are more 
> users,groups,roles
> -
>
> Key: RANGER-3900
> URL: https://issues.apache.org/jira/browse/RANGER-3900
> Project: Ranger
>  Issue Type: Improvement
>  Components: Ranger
>Affects Versions: 3.0.0
>Reporter: Ramachandran
>Assignee: Ramachandran
>Priority: Major
>
> *{color:#ff}Role Deletion Steps in Apache Ranger:{color}*{color:#ff} 
> {color}
>  
> {code:java}
>  
> 1. Getting XXRole by roleName:
> XXRole xxRole = daoMgr.getXXRole().findByRoleName(roleName); → 1DB Read call
> 2. Ensuring role is not in policy 
> Long roleRefPolicyCount = 
> daoMgr.getXXPolicyRefRole().findRoleRefPolicyCount(roleName); → 1DB Read call
> 3. Ensuring role is not in another role
> Long roleRefRoleCount = 
> daoMgr.getXXRoleRefRole().findRoleRefRoleCount(roleName);→ 1DB Read call
> 4. Fetching RangerRole by id
> RangerRole role = roleService.read(xxRole.getId()); → 1DB Read call
> 5. Dereferencing all the users from the role.
>  for (XXRoleRefUser xxRoleRefUser : xRoleUserDao.findByRoleId(roleId)) { → 
> 1DB Read call
> xRoleUserDao.remove(xxRoleRefUser);   → 1DB Write call for each xxRoleRefUser
> }
> 6. Dereferencing all the groups from the role
> for (XXRoleRefGroup xxRoleRefGroup : xRoleGroupDao.findByRoleId(roleId)) { → 
> 1DB Read call
> xRoleGroupDao.remove(xxRoleRefGroup);  → 1DB Write call for each 
> xxRoleRefGroup
> }
> 7. Dereferencing all other roles from the role.
> for (XXRoleRefRole xxRoleRefRole : xRoleRoleDao.findByRoleId(roleId)) {→ 1DB 
> Read call
> xRoleRoleDao.remove(xxRoleRefRole);→ 1DB Write call for each xxRoleRefRole
> }
> 8. Delete the rangerRole in DB
>   roleService.delete(role);  → 1DB Write call
> 9. Create TrxLog
> bizUtil.createTrxLog(trxLogList) → 1DB Write call
>  
> {code}
>  
> *{color:#ff}Total number of DB calls involved for the below role deletion 
> in Apache Ranger:{color}*
>  
> *Deleting the role which contains more users:*
> 1. role contains 100 users 
> ||DB Write count||DB Read count||DB Batch Write count||Time is taken to 
> delete the role||
> |102|6|0|1.01 seconds|
>  
> 1 role contains 500 users  
> ||DB Write count||DB Read count||DB Batch Write count||Time is taken to 
> delete the role||
> |502|6|0| 2.241 seconds|
> 1 role contains 1000 users  
> ||DB Write count||DB Read count||DB Batch Write count||Time is taken to 
> delete the role||
> |1002|6|0|3.14 seconds|
>  
> *Deleting the user (Which was associated with the role . Now that role got 
> deleted)*
>  
> ||Number of users ||Time is taken to delete each user||
> |1000| 0.8 to 1.5  seconds|
>  
> *Deleting the user (Which was not associated with any role)*
> ||Number of users ||Time is taken to delete each user||
> |1000| 0.8 to 1.2  seconds|
> {color:#FF}Recommendation :{color}
> When we reduce the number of users added to the role -→ Total number of DB 
> calls will be reduced 
> Instead of adding roles to the users, can we add those users into the group. 
> And then link that group to the role
> if we use groups instead of directly adding users into the role, the total 
> number of DB write call will be reduced to 2 from 1002  for 1k users 
>  
> {color:#ff}Proposal{color}:
> {color:#ff}We can try DB batch write call instead of 1 DB write call for 
> every user deference {color}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (RANGER-3929) To support batching in SQL IN clause of SQL queries used in Ranger

[Ramachandran (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramachandran updated RANGER-3929:
-
Attachment: 0001-RANGER-3929-To-support-batching-in-SQL-IN-clause-of-.patch

> To support batching in SQL IN clause of SQL queries used in Ranger
> --
>
> Key: RANGER-3929
> URL: https://issues.apache.org/jira/browse/RANGER-3929
> Project: Ranger
>  Issue Type: Improvement
>  Components: admin
>Affects Versions: 3.0.0
>Reporter: Ramachandran
>Assignee: Ramachandran
>Priority: Major
> Attachments: 
> 0001-RANGER-3929-To-support-batching-in-SQL-IN-clause-of-.patch
>
>
> As of  now, we are not handling the max values that will be passed into the 
> IN clause SQL queries
> Most of the databases which support maximum of 1000 values in the IN clause 
>  
>  
> {code:java}
> getEntityManager()
> .createNamedQuery("XXPolicyRefUser.deleteByIds", tClass)
> .setParameter("ids", ids).executeUpdate();
>  
> {code}
> We will get the issue when the size grows more than 1000 otherwise we will 
> not face any issue.
> We  need to fix this to avoid the potential bug in the later point of time
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Review Request 74131: Logback.xml has been incorrectly modified by RANGER-3704.

[Ramachandran Krishnan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74131/
---

(Updated Sept. 28, 2022, 3:02 p.m.)


Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.


Bugs: RANGER-3775
https://issues.apache.org/jira/browse/RANGER-3775


Repository: ranger


Description
---

{logger name="com.mchange"}
is used by c3p0 and {logger name="jdbc.connection"}

is used by log4jdbc

 

  
  


Diffs (updated)
-

  security-admin/src/main/resources/conf.dist/logback.xml 53cdc49cf 


Diff: https://reviews.apache.org/r/74131/diff/9/

Changes: https://reviews.apache.org/r/74131/diff/8-9/


Testing
---


Thanks,

Ramachandran Krishnan

Re: Review Request 74131: Logback.xml has been incorrectly modified by RANGER-3704.

[Ramachandran Krishnan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74131/
---

(Updated Sept. 28, 2022, 2:54 p.m.)


Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.


Bugs: RANGER-3775
https://issues.apache.org/jira/browse/RANGER-3775


Repository: ranger


Description
---

{logger name="com.mchange"}
is used by c3p0 and {logger name="jdbc.connection"}

is used by log4jdbc

 

  
  


Diffs (updated)
-

  security-admin/src/main/resources/conf.dist/logback.xml 53cdc49cf 


Diff: https://reviews.apache.org/r/74131/diff/8/

Changes: https://reviews.apache.org/r/74131/diff/7-8/


Testing
---


Thanks,

Ramachandran Krishnan

Re: Review Request 74131: Logback.xml has been incorrectly modified by RANGER-3704.

[Ramachandran Krishnan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74131/
---

(Updated Sept. 28, 2022, 2:50 p.m.)


Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.


Bugs: RANGER-3775
https://issues.apache.org/jira/browse/RANGER-3775


Repository: ranger


Description
---

{logger name="com.mchange"}
is used by c3p0 and {logger name="jdbc.connection"}

is used by log4jdbc

 

  
  


Diffs (updated)
-

  security-admin/src/main/resources/conf.dist/logback.xml 53cdc49cf 


Diff: https://reviews.apache.org/r/74131/diff/7/

Changes: https://reviews.apache.org/r/74131/diff/6-7/


Testing
---


Thanks,

Ramachandran Krishnan

[jira] [Commented] (RANGER-3927) Ranger KMS generates master key on every restart before inserting into DB

[Vikas Kumar (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610603#comment-17610603
 ] 

Vikas Kumar commented on RANGER-3927:
-

Review request : https://reviews.apache.org/r/74147/

> Ranger KMS generates master key on every restart before inserting into DB
> -
>
> Key: RANGER-3927
> URL: https://issues.apache.org/jira/browse/RANGER-3927
> Project: Ranger
>  Issue Type: Improvement
>  Components: Ranger
>Reporter: Dhaval Shah
>Assignee: Vikas Kumar
>Priority: Major
> Attachments: 
> 0001-RANGER-3927-Avoid-Master-Key-regeneration-on-every-r.patch
>
>
> Hi,
> At preset Ranger KMS generates Master Key on every restart,
> Ref: 
> 1.) 
> https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java#L194
> 2.) 
> https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/RangerMasterKey.java#L192
> And later check for its existence in DB .
> Ref: 
> 1.) 
> https://github.com/apache/ranger/blob/c3f1c4a6ad2ee0c805ad29f0efb265569e0d2cf0/kms/src/main/java/org/apache/hadoop/crypto/key/RangerMasterKey.java#L356
> We should check for master key existence in DB at early stage i.e. before 
> generation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Review Request 74147: Ranger KMS generates master key on every restart before inserting into DB

[Vikas Kumar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74147/
---

Review request for ranger and Sailaja Polavarapu.


Bugs: RANGER-3927
https://issues.apache.org/jira/browse/RANGER-3927


Repository: ranger


Description
---

Issue: Master Key(MK) was being first generated and then it was checking into 
DB that if MK actually exists on each restart of the KMS process. If exists, it 
simply returns. The same check could have been added before regenerating the MK 
to avoid this unnecessary execution.

FIX: Added the same check before calling the method 
RangerMasterKey.saveEncryptedMK(). Also removed the check from 
saveEncryptedMK().

New logs: If MK doesn't exist, then 

logger.info("Master Key doesn't exist in DB, Generating the Master Key");

if MK already exists, then

logger.debug("Ranger Master Key already exists in the DB, returning.");


Diffs
-

  kms/src/main/java/org/apache/hadoop/crypto/key/RangerMasterKey.java c37e98ee5 


Diff: https://reviews.apache.org/r/74147/diff/1/


Testing
---

verified through logs on restart of the KMS service. And I got "Ranger Master 
Key already exists in the DB, returning." It was not going to generate and save 
the MK into DB.


Thanks,

Vikas Kumar

Review Request 74146: RANGER-3825: Ranger admin user is unable to change another user email after the upgrade

[Pradeep Agrawal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74146/
---

Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Nikhil P, Pradeep 
Agrawal, Ramesh Mani, Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan 
Periasamy.


Bugs: RANGER-3825
https://issues.apache.org/jira/browse/RANGER-3825


Repository: ranger


Description
---

**Problem statement:** Ranger admin user is unable to change another user email 
after the upgrade. This could be due to that stored password can be in md5 
format and the new code checks the sha256 algorithm only.

Request: 
curl --insecure -X POST -H "Accept: application/json" -H "Content-Type: 
application/json" -d '{"status": 1, "oldPassword": "Password@123", "firstName": 
"first_name_test_user_ranger_up_03", "groupPermissions": [], "userSource": 0, 
"lastName": "last_name_test_userranger_up_03", "createDate": null, 
"publicScreenName": "first_name_test_user_ranger_up_03 
last_name_test_userranger_up_03", "emailAddress": "test_user_foqv...@abc.com", 
"updateDate": null, "userRoleList": ["ROLE_USER"], "userPermList": [], "id": 
102, "loginId": "ranger_up_03"}' -u 'admin:admin123' 
http://localhost:6080/service/users/102/emailchange

Response: 
Error message: 
{"statusCode":1,"msgDesc":"serverMsg.userMgrWrongPassword","messageList":[{"name":"OPER_NO_PERMISSION","rbKey":"xa.error.oper_no_permission","message":"User
 doesn't have permission to perform this operation"}]}

**Proposed statement:** the changeEmailAddress method can have additional check 
for md5 encrption if sha256 encryption check is failing.


Diffs
-

  security-admin/src/main/java/org/apache/ranger/biz/UserMgr.java 086c6e5d7 


Diff: https://reviews.apache.org/r/74146/diff/1/


Testing
---

Tested the change with this patch and curl request to update email was 
successful. Ranger build with testcases passed. 

Request: 
curl --insecure -X POST -H "Accept: application/json" -H "Content-Type: 
application/json" -d '{"status": 1, "oldPassword": "Password@123", "firstName": 
"first_name_test_user_ranger_up_03", "groupPermissions": [], "userSource": 0, 
"lastName": "last_name_test_userranger_up_03", "createDate": null, 
"publicScreenName": "first_name_test_user_ranger_up_03 
last_name_test_userranger_up_03", "emailAddress": "test_user_foqv...@abc.com", 
"updateDate": null, "userRoleList": ["ROLE_USER"], "userPermList": [], "id": 
102, "loginId": "ranger_up_03"}' -u 'admin:admin123' 
http://localhost:6080/service/users/102/emailchange

Response: 
{"id":102,"createDate":null,"updateDate":null,"loginId":"ranger_up_03","status":1,"emailAddress":"test_user_foqv...@abc.com","firstName":"first_name_test_user_ranger_up_03","lastName":"last_name_test_userranger_up_03","publicScreenName":"first_name_test_user_ranger_up_03
 
last_name_test_userranger_up_03","userSource":0,"userRoleList":["ROLE_USER"],"userPermList":[],"groupPermissions":[]}


Thanks,

Pradeep Agrawal

[jira] [Commented] (RANGER-3825) Ranger internal user is unable to change his password after the upgrade.

[Pradeep Agrawal (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610583#comment-17610583
 ] 

Pradeep Agrawal commented on RANGER-3825:
-

This Jira commit is causing below problem :

Ranger admin user is unable to change another user email after the upgrade. 

> Ranger internal user is unable to change his password after the upgrade.
> 
>
> Key: RANGER-3825
> URL: https://issues.apache.org/jira/browse/RANGER-3825
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Pradeep Agrawal
>Assignee: Pradeep Agrawal
>Priority: Critical
> Fix For: 3.0.0, 2.4.0
>
> Attachments: 
> 0001-RANGER-3825-Ranger-internal-user-is-unable-to-change.patch
>
>
> Ranger internal user is unable to change his password after the upgrade.
> Workaround : Ranger admin user can change the password of other users.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (RANGER-3825) Ranger internal user is unable to change his password after the upgrade.

[Pradeep Agrawal (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pradeep Agrawal reopened RANGER-3825:
-

> Ranger internal user is unable to change his password after the upgrade.
> 
>
> Key: RANGER-3825
> URL: https://issues.apache.org/jira/browse/RANGER-3825
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Pradeep Agrawal
>Assignee: Pradeep Agrawal
>Priority: Critical
> Fix For: 3.0.0, 2.4.0
>
> Attachments: 
> 0001-RANGER-3825-Ranger-internal-user-is-unable-to-change.patch
>
>
> Ranger internal user is unable to change his password after the upgrade.
> Workaround : Ranger admin user can change the password of other users.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (RANGER-3927) Ranger KMS generates master key on every restart before inserting into DB

[Vikas Kumar (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610564#comment-17610564
 ] 

Vikas Kumar edited comment on RANGER-3927 at 9/28/22 1:22 PM:
--

*Issue:* Master Key(MK) was being first generated and then it was checking into 
DB that if MK actually exists on each restart of the KMS process. If exists, it 
simply returns. The same check could have been added before regenerating the MK 
to avoid this unnecessary execution.

*FIX:* Added the same check before calling the method 
RangerMasterKey.saveEncryptedMK(). Also removed the check from 
saveEncryptedMK().

*New logs:* If MK doesn't exist, then 

logger.info("Master Key doesn't exist in DB, Generating the Master Key");

if MK already exists, then

logger.debug("Ranger Master Key already exists in the DB, returning.");

PFA the patch file.   will update patch once Review request is raised.

 


was (Author: JIRAUSER295683):
*Issue:* Master Key(MK) was being first generated and then it was checking into 
DB that if MK actually exists on each restart of the KMS process. If exists, it 
simply returns. The same check could have been added before regenerating the MK 
to avoid this unnecessary execution.

*FIX:* Added the same check before calling the method 
RangerMasterKey.saveEncryptedMK(). Also removed the check from 
saveEncryptedMK().

*New logs:* If MK doesn't exist, then 

logger.info("Master Key doesn't exist in DB, Generating the Master Key");

if MK already exists, then

logger.debug("Ranger Master Key already exists in the DB, returning.");

PFA the patch file. 
[^0001-RANGER-3927-Avoid-Master-Key-regeneration-on-every-r.patch]

 

> Ranger KMS generates master key on every restart before inserting into DB
> -
>
> Key: RANGER-3927
> URL: https://issues.apache.org/jira/browse/RANGER-3927
> Project: Ranger
>  Issue Type: Improvement
>  Components: Ranger
>Reporter: Dhaval Shah
>Assignee: Vikas Kumar
>Priority: Major
> Attachments: 
> 0001-RANGER-3927-Avoid-Master-Key-regeneration-on-every-r.patch
>
>
> Hi,
> At preset Ranger KMS generates Master Key on every restart,
> Ref: 
> 1.) 
> https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java#L194
> 2.) 
> https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/RangerMasterKey.java#L192
> And later check for its existence in DB .
> Ref: 
> 1.) 
> https://github.com/apache/ranger/blob/c3f1c4a6ad2ee0c805ad29f0efb265569e0d2cf0/kms/src/main/java/org/apache/hadoop/crypto/key/RangerMasterKey.java#L356
> We should check for master key existence in DB at early stage i.e. before 
> generation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (RANGER-3927) Ranger KMS generates master key on every restart before inserting into DB

[Vikas Kumar (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikas Kumar updated RANGER-3927:

Attachment: 0001-RANGER-3927-Avoid-Master-Key-regeneration-on-every-r.patch

> Ranger KMS generates master key on every restart before inserting into DB
> -
>
> Key: RANGER-3927
> URL: https://issues.apache.org/jira/browse/RANGER-3927
> Project: Ranger
>  Issue Type: Improvement
>  Components: Ranger
>Reporter: Dhaval Shah
>Assignee: Vikas Kumar
>Priority: Major
> Attachments: 
> 0001-RANGER-3927-Avoid-Master-Key-regeneration-on-every-r.patch
>
>
> Hi,
> At preset Ranger KMS generates Master Key on every restart,
> Ref: 
> 1.) 
> https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java#L194
> 2.) 
> https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/RangerMasterKey.java#L192
> And later check for its existence in DB .
> Ref: 
> 1.) 
> https://github.com/apache/ranger/blob/c3f1c4a6ad2ee0c805ad29f0efb265569e0d2cf0/kms/src/main/java/org/apache/hadoop/crypto/key/RangerMasterKey.java#L356
> We should check for master key existence in DB at early stage i.e. before 
> generation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (RANGER-3927) Ranger KMS generates master key on every restart before inserting into DB

[Vikas Kumar (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610564#comment-17610564
 ] 

Vikas Kumar commented on RANGER-3927:
-

*Issue:* Master Key(MK) was being first generated and then it was checking into 
DB that if MK actually exists on each restart of the KMS process. If exists, it 
simply returns. The same check could have been added before regenerating the MK 
to avoid this unnecessary execution.

*FIX:* Added the same check before calling the method 
RangerMasterKey.saveEncryptedMK(). Also removed the check from 
saveEncryptedMK().

*New logs:* If MK doesn't exist, then 

logger.info("Master Key doesn't exist in DB, Generating the Master Key");

if MK already exists, then

logger.debug("Ranger Master Key already exists in the DB, returning.");

PFA the patch file. 
[^0001-RANGER-3927-Avoid-Master-Key-regeneration-on-every-r.patch]

 

> Ranger KMS generates master key on every restart before inserting into DB
> -
>
> Key: RANGER-3927
> URL: https://issues.apache.org/jira/browse/RANGER-3927
> Project: Ranger
>  Issue Type: Improvement
>  Components: Ranger
>Reporter: Dhaval Shah
>Assignee: Vikas Kumar
>Priority: Major
> Attachments: 
> 0001-RANGER-3927-Avoid-Master-Key-regeneration-on-every-r.patch
>
>
> Hi,
> At preset Ranger KMS generates Master Key on every restart,
> Ref: 
> 1.) 
> https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java#L194
> 2.) 
> https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/RangerMasterKey.java#L192
> And later check for its existence in DB .
> Ref: 
> 1.) 
> https://github.com/apache/ranger/blob/c3f1c4a6ad2ee0c805ad29f0efb265569e0d2cf0/kms/src/main/java/org/apache/hadoop/crypto/key/RangerMasterKey.java#L356
> We should check for master key existence in DB at early stage i.e. before 
> generation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Review Request 74131: Logback.xml has been incorrectly modified by RANGER-3704.

[Ramachandran Krishnan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74131/
---

(Updated Sept. 28, 2022, 11:19 a.m.)


Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.


Bugs: RANGER-3775
https://issues.apache.org/jira/browse/RANGER-3775


Repository: ranger


Description
---

{logger name="com.mchange"}
is used by c3p0 and {logger name="jdbc.connection"}

is used by log4jdbc

 

  
  


Diffs (updated)
-

  security-admin/src/main/resources/conf.dist/logback.xml 53cdc49cf 


Diff: https://reviews.apache.org/r/74131/diff/6/

Changes: https://reviews.apache.org/r/74131/diff/5-6/


Testing
---


Thanks,

Ramachandran Krishnan

Re: Review Request 74131: Logback.xml has been incorrectly modified by RANGER-3704.

[Ramachandran Krishnan]

> On Sept. 28, 2022, 9:19 a.m., Kirby Zhou wrote:
> > There was an error displaying this diff.
> > 
> > 
> > The patch to 'security-admin/src/main/resources/conf.dist/logback.xml' 
> > didn't apply cleanly. The temporary files have been left in 
> > '/tmp/reviewboard.joRCmj' for debugging purposes.
> > `patch` returned: patch:  malformed patch at line 28: -- 
> > 
> > 
> > 
> > 
> > 
> >  This may be a bug in the software, a temporary outage, or an issue
> >  with the format of your diff.
> > 
> > 
> > 
> > 
> > 
> >  Please try again, and if you still have trouble,
> >  contact support.
> > 
> > 
> >
> >   
> > 
> >   
> >
> > 
> > Details
> 
> Ramachandran Krishnan wrote:
> Is it working or still we are facing issues ?

I was not able to update this diff in reviewboard .But the same patch i updated 
in the ticket  .It seems Kirby Zhou verified that as well.


- Ramachandran


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74131/#review224708
---


On Sept. 23, 2022, 8:37 a.m., Ramachandran Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74131/
> ---
> 
> (Updated Sept. 23, 2022, 8:37 a.m.)
> 
> 
> Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.
> 
> 
> Bugs: RANGER-3775
> https://issues.apache.org/jira/browse/RANGER-3775
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> {logger name="com.mchange"}
> is used by c3p0 and {logger name="jdbc.connection"}
> 
> is used by log4jdbc
> 
>  
> 
>   
>   
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/resources/conf.dist/logback.xml 53cdc49cf 
> 
> 
> Diff: https://reviews.apache.org/r/74131/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ramachandran Krishnan
> 
>

Re: Review Request 74136: RANGER-3685:hive 'show' sql produces excessive audit log

[Xuze Yang]

> On 九月 28, 2022, 9:15 a.m., Kirby Zhou wrote:
> > Why modify filterListCmdObjects() only?

filterListCmdObjects() will filter out any objects that should not be shown to 
the user, from the list of tables or databases coming from a 'show tables' or 
'show databases' command. This patch gives an option to disable record audit 
logs for every tables or databases when execute 'show' sql. checkPrivileges() 
will record one audit log about 'show' sql(accetype is USE), and this audit log 
is valuable, we should keep it.  
I would like to know what else needs to be changed, can you give more hints? 
Thanks


- Xuze


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74136/#review224707
---


On 九月 24, 2022, 2:37 p.m., Xuze Yang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74136/
> ---
> 
> (Updated 九月 24, 2022, 2:37 p.m.)
> 
> 
> Review request for ranger, Kirby Zhou, Abhay Kulkarni, and Madhan Neethiraj.
> 
> 
> Bugs: RANGER-3685
> https://issues.apache.org/jira/browse/RANGER-3685
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> hive show sql produces too many audit logs, which will flood valuable audit 
> logs and take up a lot of storage resources. This patch aims to provide a 
> configuration item to support simplified hive audit logging
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/authorization/hadoop/constants/RangerHadoopConstants.java
>  6675125e1 
>   
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuthorizer.java
>  8f6801be1 
> 
> 
> Diff: https://reviews.apache.org/r/74136/diff/1/
> 
> 
> Testing
> ---
> 
> 1. when configuration "xasecure.hive.simplify.audit.of.hive.show.sql" is not 
> set or set to false. The code behaves the same as before,i.e record audit 
> logs for each databases or tables
> 2. when configuration "xasecure.hive.simplify.audit.of.hive.show.sql" is set 
> to true. It will only record an audit log about USE operation
> 
> 
> Thanks,
> 
> Xuze Yang
> 
>

Re: Review Request 74131: Logback.xml has been incorrectly modified by RANGER-3704.

[Ramachandran Krishnan]

> On Sept. 28, 2022, 9:19 a.m., Kirby Zhou wrote:
> > There was an error displaying this diff.
> > 
> > 
> > The patch to 'security-admin/src/main/resources/conf.dist/logback.xml' 
> > didn't apply cleanly. The temporary files have been left in 
> > '/tmp/reviewboard.joRCmj' for debugging purposes.
> > `patch` returned: patch:  malformed patch at line 28: -- 
> > 
> > 
> > 
> > 
> > 
> >  This may be a bug in the software, a temporary outage, or an issue
> >  with the format of your diff.
> > 
> > 
> > 
> > 
> > 
> >  Please try again, and if you still have trouble,
> >  contact support.
> > 
> > 
> >
> >   
> > 
> >   
> >
> > 
> > Details

Is it working or still we are facing issues ?


- Ramachandran


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74131/#review224708
---


On Sept. 23, 2022, 8:37 a.m., Ramachandran Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74131/
> ---
> 
> (Updated Sept. 23, 2022, 8:37 a.m.)
> 
> 
> Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.
> 
> 
> Bugs: RANGER-3775
> https://issues.apache.org/jira/browse/RANGER-3775
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> {logger name="com.mchange"}
> is used by c3p0 and {logger name="jdbc.connection"}
> 
> is used by log4jdbc
> 
>  
> 
>   
>   
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/resources/conf.dist/logback.xml 53cdc49cf 
> 
> 
> Diff: https://reviews.apache.org/r/74131/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ramachandran Krishnan
> 
>

Re: Review Request 74131: Logback.xml has been incorrectly modified by RANGER-3704.

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74131/#review224708
---



There was an error displaying this diff.


The patch to 'security-admin/src/main/resources/conf.dist/logback.xml' 
didn't apply cleanly. The temporary files have been left in 
'/tmp/reviewboard.joRCmj' for debugging purposes.
`patch` returned: patch:  malformed patch at line 28: -- 





 This may be a bug in the software, a temporary outage, or an issue
 with the format of your diff.





 Please try again, and if you still have trouble,
 contact support.


   
  

  
   

Details

- Kirby Zhou


On 九月 23, 2022, 8:37 a.m., Ramachandran Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74131/
> ---
> 
> (Updated 九月 23, 2022, 8:37 a.m.)
> 
> 
> Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.
> 
> 
> Bugs: RANGER-3775
> https://issues.apache.org/jira/browse/RANGER-3775
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> {logger name="com.mchange"}
> is used by c3p0 and {logger name="jdbc.connection"}
> 
> is used by log4jdbc
> 
>  
> 
>   
>   
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/resources/conf.dist/logback.xml 53cdc49cf 
> 
> 
> Diff: https://reviews.apache.org/r/74131/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ramachandran Krishnan
> 
>

[jira] [Commented] (RANGER-3775) Logback.xml has been incorrectly modified by RANGER-3704.

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610461#comment-17610461
 ] 

kirby zhou commented on RANGER-3775:


It works

> Logback.xml has been incorrectly modified by RANGER-3704.
> -
>
> Key: RANGER-3775
> URL: https://issues.apache.org/jira/browse/RANGER-3775
> Project: Ranger
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 3.0.0
>Reporter: kirby zhou
>Assignee: Ramachandran
>Priority: Critical
> Attachments: 
> 0001-RANGER-3775-Logback.xml-has-been-incorrectly-modifie.patch
>
>
> {code:java}
> git show 361f179249 | filterdiff -i '*/logback.xml'
> diff --git a/security-admin/src/main/webapp/WEB-INF/logback.xml 
> b/security-admin/src/main/webapp/WEB-INF/logback.xml
> index 997f3bc59..53cdc49cf 100644
> --- a/security-admin/src/main/webapp/WEB-INF/logback.xml
> +++ b/security-admin/src/main/webapp/WEB-INF/logback.xml
> @@ -80,7 +80,7 @@
>    
>      
>    
> -  
> +  
>      
>    
>     
> {code}
> These changes seems not related to the issue RANGER-3704.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Review Request 74136: RANGER-3685:hive 'show' sql produces excessive audit log

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74136/#review224707
---



Why modify filterListCmdObjects() only?

- Kirby Zhou


On 九月 24, 2022, 2:37 p.m., Xuze Yang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74136/
> ---
> 
> (Updated 九月 24, 2022, 2:37 p.m.)
> 
> 
> Review request for ranger, Kirby Zhou, Abhay Kulkarni, and Madhan Neethiraj.
> 
> 
> Bugs: RANGER-3685
> https://issues.apache.org/jira/browse/RANGER-3685
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> hive show sql produces too many audit logs, which will flood valuable audit 
> logs and take up a lot of storage resources. This patch aims to provide a 
> configuration item to support simplified hive audit logging
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/authorization/hadoop/constants/RangerHadoopConstants.java
>  6675125e1 
>   
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuthorizer.java
>  8f6801be1 
> 
> 
> Diff: https://reviews.apache.org/r/74136/diff/1/
> 
> 
> Testing
> ---
> 
> 1. when configuration "xasecure.hive.simplify.audit.of.hive.show.sql" is not 
> set or set to false. The code behaves the same as before,i.e record audit 
> logs for each databases or tables
> 2. when configuration "xasecure.hive.simplify.audit.of.hive.show.sql" is set 
> to true. It will only record an audit log about USE operation
> 
> 
> Thanks,
> 
> Xuze Yang
> 
>

Re: Review Request 74134: RANGER-3888, RANGER-3887, RANGER-3886, RANGER-3931: Validity Scheduler , recurrence schedule: added validation

[Dineshkumar Yadav]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74134/
---

(Updated Sept. 28, 2022, 8:20 a.m.)


Review request for ranger, Dhaval Shah, Jayendra Parab, Kishor Gollapalliwar, 
Abhay Kulkarni, Pradeep Agrawal, Ramesh Mani, and Velmurugan Periasamy.


Bugs: RANGER-3886, RANGER-3887, RANGER-3888, and RANGER-3931
https://issues.apache.org/jira/browse/RANGER-3886
https://issues.apache.org/jira/browse/RANGER-3887
https://issues.apache.org/jira/browse/RANGER-3888
https://issues.apache.org/jira/browse/RANGER-3931


Repository: ranger


Description
---

below validation is added for Validity Scheduler and recurrence schedule.
1. When there is no interval but schedule is specified , policy is created
2. Ranger accepts invalid start/endTime
3. Ranger reports overlapping ranges though there are no overlapping values


Diffs
-

  
agents-common/src/main/java/org/apache/ranger/plugin/model/validation/RangerValidityScheduleValidator.java
 2b4b049fa 


Diff: https://reviews.apache.org/r/74134/diff/2/


Testing
---

tested using curl request.


Thanks,

Dineshkumar Yadav