[jira] [Resolved] (RANGER-4713) Alter view needs additional select permission on db which is not required for create view
[ https://issues.apache.org/jira/browse/RANGER-4713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal resolved RANGER-4713. Resolution: Not A Bug > Alter view needs additional select permission on db which is not required for > create view > - > > Key: RANGER-4713 > URL: https://issues.apache.org/jira/browse/RANGER-4713 > Project: Ranger > Issue Type: Bug > Components: Ranger >Reporter: suja s >Assignee: Mahesh Hanumant Bandal >Priority: Major > > STEPS TO REPRODUCE: > Create db dbfortest > Create table tablefortest under dbfortest (create table > dbfortest.tablefortest(id int, name1 string, name2 string)) > Insert rows into dbfortest.tablefortest > User u1 has select access on tablefortest via ranger policy[db=dbfortest, > table=tablefortest, column=*] - policy P1 > User u1 has create and alter permissions via ranger policy [db=dbfortest, > table=viewfortest, column=*] - policy P2 > Connect to beeline as user u1 and execute command 'create view > dbfortest.viewfortest as select id,name1 from dbfortest.tablefortest' > View creation is successful, Ranger access audits show that policy P1 granted > select on tablefortest and policy P2 granted create on viewfortest > Execute command 'alter view dbfortest.viewfortest as select id,name2 from > dbfortest.tablefortest'. > CURRENT BEHAVIOUR: > Alter view command fails with access denied error for user not having select > permissions on database dbfortest -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-4713) Alter view needs additional select permission on db which is not required for create view
[ https://issues.apache.org/jira/browse/RANGER-4713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833137#comment-17833137 ] Mahesh Hanumant Bandal commented on RANGER-4713: [~suja] - This behaviour was added in RANGER-4001. Removing check for [select] access on database during 'ALTERVIEW_AS' operation may lead to security issues. This is working as expected. Also for the 'CREATEVIEW' operation, adding check for [select] access on database will cause change in behaviour. Resolving this JIRA. > Alter view needs additional select permission on db which is not required for > create view > - > > Key: RANGER-4713 > URL: https://issues.apache.org/jira/browse/RANGER-4713 > Project: Ranger > Issue Type: Bug > Components: Ranger >Reporter: suja s >Assignee: Mahesh Hanumant Bandal >Priority: Major > > STEPS TO REPRODUCE: > Create db dbfortest > Create table tablefortest under dbfortest (create table > dbfortest.tablefortest(id int, name1 string, name2 string)) > Insert rows into dbfortest.tablefortest > User u1 has select access on tablefortest via ranger policy[db=dbfortest, > table=tablefortest, column=*] - policy P1 > User u1 has create and alter permissions via ranger policy [db=dbfortest, > table=viewfortest, column=*] - policy P2 > Connect to beeline as user u1 and execute command 'create view > dbfortest.viewfortest as select id,name1 from dbfortest.tablefortest' > View creation is successful, Ranger access audits show that policy P1 granted > select on tablefortest and policy P2 granted create on viewfortest > Execute command 'alter view dbfortest.viewfortest as select id,name2 from > dbfortest.tablefortest'. > CURRENT BEHAVIOUR: > Alter view command fails with access denied error for user not having select > permissions on database dbfortest -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (RANGER-4713) Alter view needs additional select permission on db which is not required for create view
[ https://issues.apache.org/jira/browse/RANGER-4713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal reassigned RANGER-4713: -- Assignee: Mahesh Hanumant Bandal > Alter view needs additional select permission on db which is not required for > create view > - > > Key: RANGER-4713 > URL: https://issues.apache.org/jira/browse/RANGER-4713 > Project: Ranger > Issue Type: Bug > Components: Ranger >Reporter: suja s >Assignee: Mahesh Hanumant Bandal >Priority: Major > > STEPS TO REPRODUCE: > Create db dbfortest > Create table tablefortest under dbfortest (create table > dbfortest.tablefortest(id int, name1 string, name2 string)) > Insert rows into dbfortest.tablefortest > User u1 has select access on tablefortest via ranger policy[db=dbfortest, > table=tablefortest, column=*] - policy P1 > User u1 has create and alter permissions via ranger policy [db=dbfortest, > table=viewfortest, column=*] - policy P2 > Connect to beeline as user u1 and execute command 'create view > dbfortest.viewfortest as select id,name1 from dbfortest.tablefortest' > View creation is successful, Ranger access audits show that policy P1 granted > select on tablefortest and policy P2 granted create on viewfortest > Execute command 'alter view dbfortest.viewfortest as select id,name2 from > dbfortest.tablefortest'. > CURRENT BEHAVIOUR: > Alter view command fails with access denied error for user not having select > permissions on database dbfortest -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (RANGER-3961) AuditFileSpool logs out all events that were not audited successfully
[
https://issues.apache.org/jira/browse/RANGER-3961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahesh Hanumant Bandal resolved RANGER-3961.
Fix Version/s: 3.0.0
Resolution: Fixed
This patch is merged into master branch.
Commit id :
[56b241b1472541c280e19219ef65d73413bef2a2|https://github.com/apache/ranger/commit/56b241b1472541c280e19219ef65d73413bef2a2]
> AuditFileSpool logs out all events that were not audited successfully
> -
>
> Key: RANGER-3961
> URL: https://issues.apache.org/jira/browse/RANGER-3961
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Mahesh Hanumant Bandal
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Fix For: 3.0.0
>
>
> In AuditFileSpool.java from line 901-905, there is this code:
> {code:java}
> } catch (Throwable t) {
>logger.error("Error while sending logs to consumer. provider="
> + queueProvider.getName() + ", consumer="
> + consumerProvider.getName() + ", log=" + lines, t);
> } {code}
> Here the variable *lines* holds all the events that are in current batch,
> that is 1000 by default (~0.5MB of strings). The batch can be configured even
> higher that makes things worse.
> If there is an issue in audit, a lot of huge strings will be logged out.
> Suggesting to remove *lines* variable and/or add lines.size() to know how
> many events were not sent.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (RANGER-3962) Add preload directive to HSTS header
[ https://issues.apache.org/jira/browse/RANGER-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal resolved RANGER-3962. Fix Version/s: 3.0.0 Resolution: Fixed This patch is merged into master branch. Commit : [https://github.com/apache/ranger/commit/1620539bd24ffef43320ce21baca4ff09db82af3] > Add preload directive to HSTS header > > > Key: RANGER-3962 > URL: https://issues.apache.org/jira/browse/RANGER-3962 > Project: Ranger > Issue Type: Improvement > Components: kms, Ranger >Affects Versions: 3.0.0, 2.3.0 >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Major > Fix For: 3.0.0 > > > "Preload" directive is absent in HSTS header. As security its recommended to > have within HSTS header. > ref: > [https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Strict_Transport_Security_Cheat_Sheet.html] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-3962) Add preload directive to HSTS header
Mahesh Hanumant Bandal created RANGER-3962: -- Summary: Add preload directive to HSTS header Key: RANGER-3962 URL: https://issues.apache.org/jira/browse/RANGER-3962 Project: Ranger Issue Type: Improvement Components: kms, Ranger Affects Versions: 2.3.0, 3.0.0 Reporter: Mahesh Hanumant Bandal Assignee: Mahesh Hanumant Bandal "Preload" directive is absent in HSTS header. As security its recommended to have within HSTS header. ref: [https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Strict_Transport_Security_Cheat_Sheet.html] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-3961) AuditFileSpool logs out all events that were not audited successfully
Mahesh Hanumant Bandal created RANGER-3961:
--
Summary: AuditFileSpool logs out all events that were not audited
successfully
Key: RANGER-3961
URL: https://issues.apache.org/jira/browse/RANGER-3961
Project: Ranger
Issue Type: Improvement
Components: Ranger
Affects Versions: 2.3.0, 3.0.0
Reporter: Mahesh Hanumant Bandal
Assignee: Mahesh Hanumant Bandal
In AuditFileSpool.java from line 901-905, there is this code:
{code:java}
} catch (Throwable t) {
logger.error("Error while sending logs to consumer. provider="
+ queueProvider.getName() + ", consumer="
+ consumerProvider.getName() + ", log=" + lines, t);
} {code}
Here the variable *lines* holds all the events that are in current batch, that
is 1000 by default (~0.5MB of strings). The batch can be configured even higher
that makes things worse.
If there is an issue in audit, a lot of huge strings will be logged out.
Suggesting to remove *lines* variable and/or add lines.size() to know how many
events were not sent.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-3518) Limit the query size stored in Audit logs
[ https://issues.apache.org/jira/browse/RANGER-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal updated RANGER-3518: --- Description: Ranger audit stores complete query as it is in audit logs. If the single query size is too large (i.e reqData > 10MB) it consumes large heap size on hive plugin size. When similar 200-300 queries are executed simultaneously, ranger audit takes up more than half of the heap. We need to configure ranger to limit the query size stored in Audit logs. We can set value for config "xasecure.audit.solr.limit.query.req.size" in ranger-hive-audit.xml Recommended value is 1024; however default value is set to Integer.MAX_VALUE was: Ranger audit stores complete query as it is in audit logs. If the single query size is too large (i.e reqData > 10MB) it consumes large heap size on hive plugin size. When similar 200-300 queries are executed simultaneously, ranger audit takes up more than half of the heap. We need to configure ranger to limit the query size stored in Audit logs (defaults to 1024 characters) > Limit the query size stored in Audit logs > - > > Key: RANGER-3518 > URL: https://issues.apache.org/jira/browse/RANGER-3518 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 3.0.0 >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Major > Fix For: 3.0.0 > > > Ranger audit stores complete query as it is in audit logs. If the single > query size is too large (i.e reqData > 10MB) it consumes large heap size on > hive plugin size. When similar 200-300 queries are executed simultaneously, > ranger audit takes up more than half of the heap. > We need to configure ranger to limit the query size stored in Audit logs. We > can set value for config "xasecure.audit.solr.limit.query.req.size" in > ranger-hive-audit.xml Recommended value is 1024; however default value is set > to Integer.MAX_VALUE -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (RANGER-3551) Analyze & optimize module permissions related API
[
https://issues.apache.org/jira/browse/RANGER-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17462642#comment-17462642
]
Mahesh Hanumant Bandal edited comment on RANGER-3551 at 12/20/21, 2:27 PM:
---
Ranger UI side improvements for GET API /permission/\{id} are tracked in
RANGER-3552
was (Author: maheshbandal):
Ranger UI side improvements for GET API /permission/{id} are tracked in
[RANGER-3552|https://issues.apache.org/jira/browse/RANGER-3552]
> Analyze & optimize module permissions related API
> -
>
> Key: RANGER-3551
> URL: https://issues.apache.org/jira/browse/RANGER-3551
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Mahesh Hanumant Bandal
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
>
> In ranger permission tab, edit module permission & user lookup is slower with
> large number of users & groups.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (RANGER-3551) Analyze & optimize module permissions related API
[
https://issues.apache.org/jira/browse/RANGER-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17462642#comment-17462642
]
Mahesh Hanumant Bandal commented on RANGER-3551:
Ranger UI side improvements for GET API /permission/{id} are tracked in
[RANGER-3552|https://issues.apache.org/jira/browse/RANGER-3552]
> Analyze & optimize module permissions related API
> -
>
> Key: RANGER-3551
> URL: https://issues.apache.org/jira/browse/RANGER-3551
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Mahesh Hanumant Bandal
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
>
> In ranger permission tab, edit module permission & user lookup is slower with
> large number of users & groups.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (RANGER-3551) Analyze & optimize module permissions related API
[ https://issues.apache.org/jira/browse/RANGER-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal updated RANGER-3551: --- Affects Version/s: 3.0.0 2.3.0 > Analyze & optimize module permissions related API > - > > Key: RANGER-3551 > URL: https://issues.apache.org/jira/browse/RANGER-3551 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 3.0.0, 2.3.0 >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Major > > In ranger permission tab, edit module permission & user lookup is slower with > large number of users & groups. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (RANGER-3551) Analyze & optimize module permissions related API
Mahesh Hanumant Bandal created RANGER-3551: -- Summary: Analyze & optimize module permissions related API Key: RANGER-3551 URL: https://issues.apache.org/jira/browse/RANGER-3551 Project: Ranger Issue Type: Improvement Components: Ranger Reporter: Mahesh Hanumant Bandal Assignee: Mahesh Hanumant Bandal In ranger permission tab, edit module permission & user lookup is slower with large number of users & groups. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (RANGER-3518) Limit the query size stored in Audit logs
Mahesh Hanumant Bandal created RANGER-3518: -- Summary: Limit the query size stored in Audit logs Key: RANGER-3518 URL: https://issues.apache.org/jira/browse/RANGER-3518 Project: Ranger Issue Type: Improvement Components: Ranger Affects Versions: 3.0.0 Reporter: Mahesh Hanumant Bandal Assignee: Mahesh Hanumant Bandal Ranger audit stores complete query as it is in audit logs. If the single query size is too large (i.e reqData > 10MB) it consumes large heap size on hive plugin size. When similar 200-300 queries are executed simultaneously, ranger audit takes up more than half of the heap. We need to configure ranger to limit the query size stored in Audit logs (defaults to 1024 characters) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (RANGER-3023) Permission tab takes longer time to load with large number of users and group_users data
[
https://issues.apache.org/jira/browse/RANGER-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahesh Hanumant Bandal resolved RANGER-3023.
Fix Version/s: 3.0.0
Resolution: Fixed
committed in master branch, commit id :
[5732060da4c4c88c1fba3c89fa5369ea56c9b942|https://github.com/apache/ranger/commit/5732060da4c4c88c1fba3c89fa5369ea56c9b942]
> Permission tab takes longer time to load with large number of users and
> group_users data
>
>
> Key: RANGER-3023
> URL: https://issues.apache.org/jira/browse/RANGER-3023
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Affects Versions: 2.2.0
>Reporter: Mahesh Hanumant Bandal
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Fix For: 3.0.0
>
>
> GET API /service/xusers/permission takes longer time to load with following
> number of users and group mappings in db.
> {code:java}
> select count(*) from x_user;
> 50052
> select count(*) from x_group_users;
> 500056
> {code}
> Need to have performance optimization on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3211) Upgrade libthrift to 0.14.1
[ https://issues.apache.org/jira/browse/RANGER-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17418175#comment-17418175 ] Mahesh Hanumant Bandal commented on RANGER-3211: HiveMetaStoreClient is not compatible with thrift-0.14.1 and due to this hive resource lookup is not working. This patch needs to be reverted from master branch as well as ranger-2.2 branch. Created a [Revert Review Request|https://reviews.apache.org/r/73596/] for the same. > Upgrade libthrift to 0.14.1 > --- > > Key: RANGER-3211 > URL: https://issues.apache.org/jira/browse/RANGER-3211 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 3.0.0 >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Major > Fix For: 3.0.0 > > > Ranger is pulling in Thrift 0.13.0 and should be upgraded to 0.14.1 for best > practices. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3408) Upgrade commons-io to 2.8.0
Mahesh Hanumant Bandal created RANGER-3408: -- Summary: Upgrade commons-io to 2.8.0 Key: RANGER-3408 URL: https://issues.apache.org/jira/browse/RANGER-3408 Project: Ranger Issue Type: Improvement Components: Ranger Affects Versions: 3.0.0 Reporter: Mahesh Hanumant Bandal Assignee: Mahesh Hanumant Bandal Upgrade commons-io to 2.8.0 for best practices. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3381) Upgrade to junit 4.13.1
Mahesh Hanumant Bandal created RANGER-3381: -- Summary: Upgrade to junit 4.13.1 Key: RANGER-3381 URL: https://issues.apache.org/jira/browse/RANGER-3381 Project: Ranger Issue Type: Improvement Components: Ranger Affects Versions: 3.0.0 Reporter: Mahesh Hanumant Bandal Assignee: Mahesh Hanumant Bandal Fix For: 3.0.0 Upgrade junit version to 4.13.1 for best practices. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (RANGER-3367) [Intermittent] Ranger Admin perf logs are not getting logged after Spring Security upgrade
[ https://issues.apache.org/jira/browse/RANGER-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal resolved RANGER-3367. Resolution: Fixed commit id: master branch - [10d0a833eedba764e60ac1012f8371764a2468dd|https://github.com/apache/ranger/commit/10d0a833eedba764e60ac1012f8371764a2468dd] ranger-2.2 branch - [30730b1f691ba5ff7ebcddab78102c18b4deb734|https://github.com/apache/ranger/commit/30730b1f691ba5ff7ebcddab78102c18b4deb734] > [Intermittent] Ranger Admin perf logs are not getting logged after Spring > Security upgrade > -- > > Key: RANGER-3367 > URL: https://issues.apache.org/jira/browse/RANGER-3367 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 3.0.0 >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Major > Fix For: 3.0.0 > > > When debug logs are enabled, PERF logs are not generated in ranger. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-3367) [Intermittent] Ranger Admin perf logs are not getting logged after Spring Security upgrade
[ https://issues.apache.org/jira/browse/RANGER-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17396548#comment-17396548 ] Mahesh Hanumant Bandal commented on RANGER-3367: After spring upgrade, spring-jcl-5.3.7.jar is loaded into ranger classpath which causes this issue. > [Intermittent] Ranger Admin perf logs are not getting logged after Spring > Security upgrade > -- > > Key: RANGER-3367 > URL: https://issues.apache.org/jira/browse/RANGER-3367 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 3.0.0 >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Major > Fix For: 3.0.0 > > > When debug logs are enabled, PERF logs are not generated in ranger. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3367) [Intermittent] Ranger Admin perf logs are not getting logged after Spring Security upgrade
Mahesh Hanumant Bandal created RANGER-3367: -- Summary: [Intermittent] Ranger Admin perf logs are not getting logged after Spring Security upgrade Key: RANGER-3367 URL: https://issues.apache.org/jira/browse/RANGER-3367 Project: Ranger Issue Type: Improvement Components: Ranger Affects Versions: 3.0.0 Reporter: Mahesh Hanumant Bandal Assignee: Mahesh Hanumant Bandal Fix For: 3.0.0 When debug logs are enabled, PERF logs are not generated in ranger. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (RANGER-3314) [Atlas Ranger Plugin classification auth changes] Importing atlas policy with old schema displays incorrect permissions on UI
[ https://issues.apache.org/jira/browse/RANGER-3314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal reassigned RANGER-3314: -- Assignee: Mahesh Hanumant Bandal > [Atlas Ranger Plugin classification auth changes] Importing atlas policy with > old schema displays incorrect permissions on UI > - > > Key: RANGER-3314 > URL: https://issues.apache.org/jira/browse/RANGER-3314 > Project: Ranger > Issue Type: Bug > Components: plugins >Affects Versions: 2.2.0 >Reporter: Abhishek Shukla >Assignee: Mahesh Hanumant Bandal >Priority: Major > Attachments: permission list.png, policy details.png, policy.json > > > This is with respect to RANGER-3195 where we have moved the Add/Update/Remove > classification permissions to a new classification resource. > > Steps: > * Try to import an atlas resource policy with the old format [Attached test > policy.json] > * Expectations are the permission like Add/Update/Remove classification > should not be shown in permission list as they are not present at Entity > resource level but we are showing them in the UI. [Attached screenshots] > > So these permissions are present in the policy, is this expected? I have not > performed any atlas side validation to check if these permissions will be > enforced. > I think we should not add these permissions even if they are present in the > policy OR we should display some error message during import, displaying > permissions not supported something like that. > > Creating this Jira for more discussion on this. > cc [~nixon] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3303) Improve error handling in Ranger Solr bootstrap
Mahesh Hanumant Bandal created RANGER-3303:
--
Summary: Improve error handling in Ranger Solr bootstrap
Key: RANGER-3303
URL: https://issues.apache.org/jira/browse/RANGER-3303
Project: Ranger
Issue Type: Improvement
Components: Ranger
Affects Versions: 3.0.0
Reporter: Mahesh Hanumant Bandal
Assignee: Mahesh Hanumant Bandal
Fix For: 3.0.0
Handle retries for failure during upload configs and validate {{ranger_audits}}
collection after create collection action.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (RANGER-3211) Upgrade libthrift to 0.14.1
[ https://issues.apache.org/jira/browse/RANGER-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal updated RANGER-3211: --- Fix Version/s: 2.2.0 > Upgrade libthrift to 0.14.1 > --- > > Key: RANGER-3211 > URL: https://issues.apache.org/jira/browse/RANGER-3211 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 3.0.0 >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > Ranger is pulling in Thrift 0.13.0 and should be upgraded to 0.14.1 for best > practices. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3215) Ranger Metric Util consumes more heap-size and does not exit
Mahesh Hanumant Bandal created RANGER-3215: -- Summary: Ranger Metric Util consumes more heap-size and does not exit Key: RANGER-3215 URL: https://issues.apache.org/jira/browse/RANGER-3215 Project: Ranger Issue Type: Bug Components: audit, Ranger Affects Versions: 3.0.0 Reporter: Mahesh Hanumant Bandal Assignee: Mahesh Hanumant Bandal When Ranger Metric Util is called and auditDBType="solr", the process does not exit and keeps consuming more heap-size than normally required. Solr server throws : Java heap space OutOfMemoryError for large number of audits. cc: [~abhayk], [~vel], [~mehul] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3211) Upgrade libthrift to 0.14.1
Mahesh Hanumant Bandal created RANGER-3211: -- Summary: Upgrade libthrift to 0.14.1 Key: RANGER-3211 URL: https://issues.apache.org/jira/browse/RANGER-3211 Project: Ranger Issue Type: Improvement Components: Ranger Affects Versions: 3.0.0 Reporter: Mahesh Hanumant Bandal Assignee: Mahesh Hanumant Bandal Ranger is pulling in Thrift 0.13.0 and should be upgraded to 0.14.1 for best practices. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (RANGER-3204) Upgrade log4j-core to 2.13.3
[ https://issues.apache.org/jira/browse/RANGER-3204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300065#comment-17300065 ] Mahesh Hanumant Bandal edited comment on RANGER-3204 at 3/15/21, 3:49 PM: -- commit id : master branch - [ea68a309da500c8ada275915947de236a8334b83|https://github.com/apache/ranger/commit/ea68a309da500c8ada275915947de236a8334b83] ranger-2.2 branch - [18dca4e396d710a7b8cf4c657cdffa50ef513395|https://github.com/apache/ranger/commit/18dca4e396d710a7b8cf4c657cdffa50ef513395] was (Author: maheshbandal): commit id : master branch - [ea68a309da500c8ada275915947de236a8334b83|https://github.com/apache/ranger/commit/ea68a309da500c8ada275915947de236a8334b83] > Upgrade log4j-core to 2.13.3 > > > Key: RANGER-3204 > URL: https://issues.apache.org/jira/browse/RANGER-3204 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 3.0.0 >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Major > Fix For: 3.0.0 > > > Upgrade log4j-core to 2.13.3 for best practices. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-3199) illegal reflective access operation warning in KMS catalina.out logs
[
https://issues.apache.org/jira/browse/RANGER-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17301720#comment-17301720
]
Mahesh Hanumant Bandal commented on RANGER-3199:
commit on ranger-2.2 branch :
[438d91ba87b798aad28b9bda8f6a316407b16f8e|https://github.com/apache/ranger/commit/438d91ba87b798aad28b9bda8f6a316407b16f8e]
> illegal reflective access operation warning in KMS catalina.out logs
>
>
> Key: RANGER-3199
> URL: https://issues.apache.org/jira/browse/RANGER-3199
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 3.0.0
>Reporter: Mahesh Hanumant Bandal
>Assignee: Mahesh Hanumant Bandal
>Priority: Minor
>
> following logs are observed in catalina.out file of ranger-kms while using
> JDK-11:
> {code:java}
> INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:43
> PM'
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector$1 jaxb-impl-2.2.3-1.jar) to
> method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
> WARNING: Please consider reporting this to the maintainers of
> com.sun.xml.bind.v2.runtime.reflect.opt.Injector$1
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {code}
> Need to fix this warning logs.
> *This fix should be provided from the maintainers of "com.sun.xml.bind" i.e.
> Oracle Corporation. Affected jar is "jaxb-impl-2.2.3-1.jar". This jar is
> inherited from "jersey-json-1.19.3.jar".*
>
> Workaround for this issue :
> * +To avoid this warning logs we can add dependency for jaxb-impl-2.3.3.jar+
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (RANGER-3204) Upgrade log4j-core to 2.13.3
Mahesh Hanumant Bandal created RANGER-3204: -- Summary: Upgrade log4j-core to 2.13.3 Key: RANGER-3204 URL: https://issues.apache.org/jira/browse/RANGER-3204 Project: Ranger Issue Type: Improvement Components: Ranger Affects Versions: 3.0.0 Reporter: Mahesh Hanumant Bandal Assignee: Mahesh Hanumant Bandal Fix For: 3.0.0 Upgrade log4j-core to 2.13.3 for best practices. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3199) illegal reflective access operation warning in KMS catalina.out logs
Mahesh Hanumant Bandal created RANGER-3199:
--
Summary: illegal reflective access operation warning in KMS
catalina.out logs
Key: RANGER-3199
URL: https://issues.apache.org/jira/browse/RANGER-3199
Project: Ranger
Issue Type: Bug
Components: Ranger
Affects Versions: 3.0.0
Reporter: Mahesh Hanumant Bandal
Assignee: Mahesh Hanumant Bandal
following logs are observed in catalina.out file of ranger-kms while using
JDK-11:
{code:java}
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:43
PM'
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by
com.sun.xml.bind.v2.runtime.reflect.opt.Injector$1 jaxb-impl-2.2.3-1.jar) to
method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
WARNING: Please consider reporting this to the maintainers of
com.sun.xml.bind.v2.runtime.reflect.opt.Injector$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
{code}
Need to fix this warning logs.
*This fix should be provided from the maintainers of "com.sun.xml.bind" i.e.
Oracle Corporation. Affected jar is "jaxb-impl-2.2.3-1.jar". This jar is
inherited from "jersey-json-1.19.3.jar".*
Workaround for this issue :
* +To avoid this warning logs we can add dependency for jaxb-impl-2.3.3.jar+
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (RANGER-3176) Ranger HBase resource lookup fails with java.lang.NoClassDefFoundError
Mahesh Hanumant Bandal created RANGER-3176:
--
Summary: Ranger HBase resource lookup fails with
java.lang.NoClassDefFoundError
Key: RANGER-3176
URL: https://issues.apache.org/jira/browse/RANGER-3176
Project: Ranger
Issue Type: Bug
Components: Ranger
Reporter: Mahesh Hanumant Bandal
Assignee: Mahesh Hanumant Bandal
Following exception occurs
{code:java}
2021-02-04 19:46:55,175 [timed-executor-pool-0] ERROR
apache.ranger.services.hbase.client.HBaseClient$2 (HBaseClient.java:279) -
getTableList: Unable to get HBase table List for
[repository:cl1_hbase,table-match:t.*].org.apache.hadoop.hbase.DoNotRetryIOException:
java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer
2021-02-04 19:46:55,176 [timed-executor-pool-0] ERROR
apache.ranger.services.hbase.client.HBaseResourceMgr
(HBaseResourceMgr.java:147) - Unable to get hbase resources.
org.apache.ranger.plugin.client.HadoopException: getTableList: Unable to get
HBase table List for [repository:cl1_hbase,table-match:t.*].
at
org.apache.ranger.services.hbase.client.HBaseClient$2.run(HBaseClient.java:276)
at
org.apache.ranger.services.hbase.client.HBaseClient$2.run(HBaseClient.java:223)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.ranger.services.hbase.client.HBaseClient.getTableList(HBaseClient.java:223)
at
org.apache.ranger.services.hbase.client.HBaseResourceMgr$1.call(HBaseResourceMgr.java:115)
at
org.apache.ranger.services.hbase.client.HBaseResourceMgr$1.call(HBaseResourceMgr.java:112)
at
org.apache.ranger.plugin.util.TimedEventUtil.timedTask(TimedEventUtil.java:40)
at
org.apache.ranger.services.hbase.client.HBaseResourceMgr.getHBaseResource(HBaseResourceMgr.java:140)
at
org.apache.ranger.services.hbase.RangerServiceHBase.lookupResource(RangerServiceHBase.java:119)
at
org.apache.ranger.biz.ServiceMgr$LookupCallable.actualCall(ServiceMgr.java:642)
at
org.apache.ranger.biz.ServiceMgr$LookupCallable.actualCall(ServiceMgr.java:626)
at
org.apache.ranger.biz.ServiceMgr$TimedCallable.call(ServiceMgr.java:608)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.DoNotRetryIOException:
java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:119)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3133)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3125)
at
org.apache.hadoop.hbase.client.HBaseAdmin.listTableDescriptors(HBaseAdmin.java:347)
at
org.apache.hadoop.hbase.client.HBaseAdmin.listTableDescriptors(HBaseAdmin.java:341)
at
org.apache.ranger.services.hbase.client.HBaseClient$2.run(HBaseClient.java:238)
... 16 more
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer
at org.apache.hadoop.hbase.ipc.Call.(Call.java:76)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:410)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:325)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:91)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:576)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java)
at
org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionImplementation.java:1152)
at
org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceStubMaker.makeStubNoRetries(ConnectionImplementation.java:1182)
at
org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionImplementation.java:1198)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1263)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.getMaster(ConnectionImplementation.java:1252)
at
org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:57)
at
org.apache.hadoop.hbase.client.Rpc
[jira] [Created] (RANGER-3168) User/Auditor should have read-only access for Servicedef via PublicAPIsv2 API
Mahesh Hanumant Bandal created RANGER-3168:
--
Summary: User/Auditor should have read-only access for Servicedef
via PublicAPIsv2 API
Key: RANGER-3168
URL: https://issues.apache.org/jira/browse/RANGER-3168
Project: Ranger
Issue Type: Improvement
Components: Ranger
Affects Versions: 3.0.0
Reporter: Mahesh Hanumant Bandal
Assignee: Mahesh Hanumant Bandal
Fix For: 3.0.0
Following GET API's should be accessible for other users.
{code:java}
GET API: /service/public/v2/api/servicedef/
GET API: /service/public/v2/api/servicedef/{id}
GET API: /service/public/v2/api/servicedef/name/{name}
{code}
If a user has permissions on the "Resource Based Policies" module, then he is
be able to view servicedef using Ranger-UI and therefore he can access
servicedef using GET API- /service/public/v2/api/servicedef/
KeyAdmin and KMSAuditor users can only access KMS servicedef.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3155) Roles are not accessible for Admin User through REST API
[
https://issues.apache.org/jira/browse/RANGER-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271135#comment-17271135
]
Mahesh Hanumant Bandal commented on RANGER-3155:
[~rujia1019] - Build succeeded with "mvn clean compile package test", I did not
see any PMD issues though.
*case 1 (role1 exists) :*
curl -X GET -H "Content-Type:application/json" -iku admin:admin123
"http://localhost:6080/service/roles/roles/name/role1?execUser=non-admin";
{code:java}
User non-admin does not have privilege to role role1
1
{code}
*case 2 (role3 does not exist and logged-in user is admin) :*
curl -X GET -H "Content-Type:application/json" -iku admin:admin123
"http://localhost:6080/service/roles/roles/name/role3?execUser=non-admin";
{code:java}
Role with name: role3 does not exist
1
{code}
*case 3 (role3 does not exist and ?execUser=non-admin removed from query param)
:*
curl -X GET -H "Content-Type:application/json" -iku admin:admin123
"http://localhost:6080/service/roles/roles/name/role3";
{code:java}
Role with name: role3 does not exist
1
{code}
*case 4 (role1 exists and logged-in user is non-admin) :*
curl -X GET -H "Content-Type:application/json" -iku non-admin:nonadmin1
"http://localhost:6080/service/roles/roles/name/role1";
{code:java}
User doesn't have permissions to get details for role1
1
{code}
> Roles are not accessible for Admin User through REST API
>
>
> Key: RANGER-3155
> URL: https://issues.apache.org/jira/browse/RANGER-3155
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 3.0.0
>Reporter: Mahesh Hanumant Bandal
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Fix For: 3.0.0
>
>
> This issue is introduced by RANGER-3135.
> Following is current observation :
> I tried to access GET API
> [http://localhost:6080/service/roles/roles/name/role1] with admin user and it
> does not return roles which already exist, whereas select * from x_role
> returns two roles. ie. role1, role2.
> API returns following response :
> {code:java}
>
> Role with name: role1 does not exist
> 1
>
> {code}
> *With admin user it shows same response regardless of role's existence.*
> =
> Resolution :
> Admin user should get roles when accessed via REST API. This JIRA should also
> focus on providing fix for RANGER-3135 where GET API /roles/name/\{name}
> should provide proper message in case of the role does not exist. In case of
> non-admin user, it should deny access to roles.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3155) Roles are not accessible for Admin User through REST API
[
https://issues.apache.org/jira/browse/RANGER-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269326#comment-17269326
]
Mahesh Hanumant Bandal commented on RANGER-3155:
Yes. That is why when logged-in user is non-admin it returns "User doesn't have
permissions to get details for ROLETEST" regardless of role "ROLETEST" exist or
not.
And when logged-in user is an Admin and execUser=non-admin it shows "User
doesn't have permissions to get details for ROLETEST". This is because of
non-admin user.
If you remove query param "?execUser=non-admin" and logged-in user is an Admin
and Role does not exist it will show "Role does not exist"
> Roles are not accessible for Admin User through REST API
>
>
> Key: RANGER-3155
> URL: https://issues.apache.org/jira/browse/RANGER-3155
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 3.0.0
>Reporter: Mahesh Hanumant Bandal
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Fix For: 3.0.0
>
>
> This issue is introduced by RANGER-3135.
> Following is current observation :
> I tried to access GET API
> [http://localhost:6080/service/roles/roles/name/role1] with admin user and it
> does not return roles which already exist, whereas select * from x_role
> returns two roles. ie. role1, role2.
> API returns following response :
> {code:java}
>
> Role with name: role1 does not exist
> 1
>
> {code}
> *With admin user it shows same response regardless of role's existence.*
> =
> Resolution :
> Admin user should get roles when accessed via REST API. This JIRA should also
> focus on providing fix for RANGER-3135 where GET API /roles/name/\{name}
> should provide proper message in case of the role does not exist. In case of
> non-admin user, it should deny access to roles.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (RANGER-3155) Roles are not accessible for Admin User through REST API
[
https://issues.apache.org/jira/browse/RANGER-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269262#comment-17269262
]
Mahesh Hanumant Bandal edited comment on RANGER-3155 at 1/21/21, 12:41 PM:
---
Reverted RANGER-3135 patch as it breaks current functionality to access role
for ranger admin and provided a fix with its root cause to represent proper
message in case of role does not exist. Earlier it was showing "User doesn't
have permissions to get details for role" even if role does not exist.
With this patch we Admin user can access roles and if role does not exist it
shows message as "Role with name: ROLETEST does not exist". For non-admin user
it behaves as it is. It shows "User doesn't have permissions to get details for
ROLETEST".
This fix simply throws an error caught in getRoleIfAccessible() function of
RoleREST.java class to parent method. Parent method prints appropriate message
caught in exception. And reverted changes made in RANGER-3135.
Testing Done :
For admin user :
1. If role is present - API returns role information.
2. If role does not exist - API will return "Role with name: ROLETEST does not
exist.
3. If role exist and ?execUser=non-admin - API will return "User non-admin does
not have privilege to role ROLETEST"
For non-admin user :
1. It shows "User doesn't have permissions to get details for ROLETEST".
CC : [~mehul], [~vel], [~pradeep], [~rujia1019]
was (Author: maheshbandal):
Reverted RANGER-3135 patch as it breaks current functionality to access role
for ranger admin and provided a fix with its root cause to represent proper
message in case of role does not exist. Earlier it was showing "User doesn't
have permissions to get details for role" even if role does not exist.
With this patch we Admin user can access roles and if role does not exist it
shows message as "Role with name: ROLETEST does not exist". For non-admin user
it behaves as it is. It shows "User doesn't have permissions to get details for
ROLETEST".
This fix simply throws an error caught in getRoleIfAccessible() function of
RoleREST.java class to parent method. Parent method prints appropriate message
caught in exception. And reverted changes made in RANGER-3135.
Testing Done :
For admin user :
1. If role is present - API returns role information.
2. If role does not exist - API will return "Role with name: ROLETEST does not
exist.
3. If role exist and ?execUser=non-admin - API will return "User non-admin does
not have privilege to role ROLETEST"
For non-admin user :
1. It shows "User doesn't have permissions to get details for ROLETEST".
> Roles are not accessible for Admin User through REST API
>
>
> Key: RANGER-3155
> URL: https://issues.apache.org/jira/browse/RANGER-3155
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 3.0.0
>Reporter: Mahesh Hanumant Bandal
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Fix For: 3.0.0
>
>
> This issue is introduced by RANGER-3135.
> Following is current observation :
> I tried to access GET API
> [http://localhost:6080/service/roles/roles/name/role1] with admin user and it
> does not return roles which already exist, whereas select * from x_role
> returns two roles. ie. role1, role2.
> API returns following response :
> {code:java}
>
> Role with name: role1 does not exist
> 1
>
> {code}
> *With admin user it shows same response regardless of role's existence.*
> =
> Resolution :
> Admin user should get roles when accessed via REST API. This JIRA should also
> focus on providing fix for RANGER-3135 where GET API /roles/name/\{name}
> should provide proper message in case of the role does not exist. In case of
> non-admin user, it should deny access to roles.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3135) Ranger always ponit out "User does not have permission for this operation" when user try to query a none-exist role
[
https://issues.apache.org/jira/browse/RANGER-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269268#comment-17269268
]
Mahesh Hanumant Bandal commented on RANGER-3135:
Hi [~rujia1019],
Please refer to RANGER-3155. I have updated apache request on the same. It has
fix for current issue. It will also print appropriate message for get API
/roles/name/\{name}.
Thank you.
> Ranger always ponit out "User does not have permission for this operation"
> when user try to query a none-exist role
> ---
>
> Key: RANGER-3135
> URL: https://issues.apache.org/jira/browse/RANGER-3135
> Project: Ranger
> Issue Type: Improvement
> Components: admin
>Affects Versions: 2.0.0, 2.1.0
>Reporter: rujia
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 0001-optimze-log-print-for-querying-roles.patch
>
>
> when i access /role/name/{name}, a Exception "User does not have permission
> for this operation" always been taken, even if this role doesn't exist .
> it is better to prompt user this role is not exist in this case when
> execute-user has admin permission.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3155) Roles are not accessible for Admin User through REST API
[
https://issues.apache.org/jira/browse/RANGER-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269262#comment-17269262
]
Mahesh Hanumant Bandal commented on RANGER-3155:
Reverted RANGER-3135 patch as it breaks current functionality to access role
for ranger admin and provided a fix with its root cause to represent proper
message in case of role does not exist. Earlier it was showing "User doesn't
have permissions to get details for role" even if role does not exist.
With this patch we Admin user can access roles and if role does not exist it
shows message as "Role with name: ROLETEST does not exist". For non-admin user
it behaves as it is. It shows "User doesn't have permissions to get details for
ROLETEST".
This fix simply throws an error caught in getRoleIfAccessible() function of
RoleREST.java class to parent method. Parent method prints appropriate message
caught in exception. And reverted changes made in RANGER-3135.
Testing Done :
For admin user :
1. If role is present - API returns role information.
2. If role does not exist - API will return "Role with name: ROLETEST does not
exist.
3. If role exist and ?execUser=non-admin - API will return "User non-admin does
not have privilege to role ROLETEST"
For non-admin user :
1. It shows "User doesn't have permissions to get details for ROLETEST".
> Roles are not accessible for Admin User through REST API
>
>
> Key: RANGER-3155
> URL: https://issues.apache.org/jira/browse/RANGER-3155
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 3.0.0
>Reporter: Mahesh Hanumant Bandal
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Fix For: 3.0.0
>
>
> This issue is introduced by RANGER-3135.
> Following is current observation :
> I tried to access GET API
> [http://localhost:6080/service/roles/roles/name/role1] with admin user and it
> does not return roles which already exist, whereas select * from x_role
> returns two roles. ie. role1, role2.
> API returns following response :
> {code:java}
>
> Role with name: role1 does not exist
> 1
>
> {code}
> *With admin user it shows same response regardless of role's existence.*
> =
> Resolution :
> Admin user should get roles when accessed via REST API. This JIRA should also
> focus on providing fix for RANGER-3135 where GET API /roles/name/\{name}
> should provide proper message in case of the role does not exist. In case of
> non-admin user, it should deny access to roles.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (RANGER-3155) Roles are not accessible for Admin User through REST API
[
https://issues.apache.org/jira/browse/RANGER-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahesh Hanumant Bandal updated RANGER-3155:
---
Summary: Roles are not accessible for Admin User through REST API (was:
Roles are not accessible for admin User through REST API)
> Roles are not accessible for Admin User through REST API
>
>
> Key: RANGER-3155
> URL: https://issues.apache.org/jira/browse/RANGER-3155
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 3.0.0
>Reporter: Mahesh Hanumant Bandal
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Fix For: 3.0.0
>
>
> This issue is introduced by RANGER-3135.
> Following is current observation :
> I tried to access GET API
> [http://localhost:6080/service/roles/roles/name/role1] with admin user and it
> does not return roles which already exist, whereas select * from x_role
> returns two roles. ie. role1, role2.
> API returns following response :
> {code:java}
>
> Role with name: role1 does not exist
> 1
>
> {code}
> *With admin user it shows same response regardless of role's existence.*
> =
> Resolution :
> Admin user should get roles when accessed via REST API. This JIRA should also
> focus on providing fix for RANGER-3135 where GET API /roles/name/\{name}
> should provide proper message in case of the role does not exist. In case of
> non-admin user, it should deny access to roles.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (RANGER-3155) Roles are not accessible for admin User through REST API
Mahesh Hanumant Bandal created RANGER-3155:
--
Summary: Roles are not accessible for admin User through REST API
Key: RANGER-3155
URL: https://issues.apache.org/jira/browse/RANGER-3155
Project: Ranger
Issue Type: Bug
Components: Ranger
Affects Versions: 3.0.0
Reporter: Mahesh Hanumant Bandal
Assignee: Mahesh Hanumant Bandal
Fix For: 3.0.0
This issue is introduced by RANGER-3135.
Following is current observation :
I tried to access GET API
[http://localhost:6080/service/roles/roles/name/role1] with admin user and it
does not return roles which already exist, whereas select * from x_role returns
two roles. ie. role1, role2.
API returns following response :
{code:java}
Role with name: role1 does not exist
1
{code}
*With admin user it shows same response regardless of role's existence.*
=
Resolution :
Admin user should get roles when accessed via REST API. This JIRA should also
focus on providing fix for RANGER-3135 where GET API /roles/name/\{name} should
provide proper message in case of the role does not exist. In case of non-admin
user, it should deny access to roles.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3136) NullPointException found when import policies form web side and "isOverride" is selected
[ https://issues.apache.org/jira/browse/RANGER-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269180#comment-17269180 ] Mahesh Hanumant Bandal commented on RANGER-3136: NPE issue was already fixed along with [RANGER-3078|https://issues.apache.org/jira/browse/RANGER-3078] and committed on master branch [https://github.com/apache/ranger/commit/208b6df0d42d63eb5834c889afb092b7a69146a4]. [~pradeep], Following is my observation : [deletePoliciesProvidedInServiceMap()|https://github.com/apache/ranger/blob/4b9ab8a509f2e0d9e8e7dd210505fa7f342c2b44/security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java#L2693] function eventually calls [getServicePolicies()|https://github.com/apache/ranger/blob/4ff1c7ea9984812ec2ea02910ff8bb9eb9003f9a/security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java#L2522] function in ServiceDBStore class which takes serchFilter object as an argument. After adding serviceName to serachFilter object at [ServiceREST.java#L2702|https://github.com/apache/ranger/blob/4b9ab8a509f2e0d9e8e7dd210505fa7f342c2b44/security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java#L2702] it will cause no-zone policies to undergo filtering process and code between [ServiceDBStore.java#L2536|https://github.com/apache/ranger/blob/4ff1c7ea9984812ec2ea02910ff8bb9eb9003f9a/security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java#L2536] and [ServiceDBStore.java#L2595|https://github.com/apache/ranger/blob/4ff1c7ea9984812ec2ea02910ff8bb9eb9003f9a/security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java#L2595] will execute. This code seems to be related to resource filter. As of now, it will filter out policies based on ServiceName and return list of RangerPolicy. Currently there is no exception found with/without this patch during import policies with "Override Policy" flag is checked. I am not sure about any other impact of introducing "Service Name" to searchFilter in deletePoliciesProvidedInServiceMap() function. [~rujia1019], can you please provide your analysis on the same? As NPE is already fixed and this patch introduces extra parameter to searchFilter. > NullPointException found when import policies form web side and "isOverride" > is selected > > > Key: RANGER-3136 > URL: https://issues.apache.org/jira/browse/RANGER-3136 > Project: Ranger > Issue Type: Bug > Components: admin >Affects Versions: 2.1.0 >Reporter: rujia >Priority: Major > Fix For: 3.0.0 > > Attachments: > 0001-NUllPointException-occur-when-import-polices-anf-isO.patch, > ServiceDBStore-L2536.png, image-2021-01-21-11-26-15-996.png, > image-2021-01-21-11-27-05-887.png > > > i got NullPointException when i imported policies json file from rangeradmin > web, i selected "isOverride" flag, it is caused by RANGER-3064, serchFilter > will be null when do deletePolices. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-3136) NullPointException found when import policies form web side and "isOverride" is selected
[ https://issues.apache.org/jira/browse/RANGER-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269069#comment-17269069 ] Mahesh Hanumant Bandal commented on RANGER-3136: Adding the "Service Name" to searchFilter changes behaviour of current implementation of filtering service policies. As per the code, I understand that no-zone policies will also undergo filtering process. As you mentioned this issue is already resolved in [RANGER-3078|https://issues.apache.org/jira/browse/RANGER-3078] by [~dineshkumar-yadav]. Can we please revert this changes? CC: [~pradeep], [~mehul], [~vel] > NullPointException found when import policies form web side and "isOverride" > is selected > > > Key: RANGER-3136 > URL: https://issues.apache.org/jira/browse/RANGER-3136 > Project: Ranger > Issue Type: Bug > Components: admin >Affects Versions: 2.1.0 >Reporter: rujia >Priority: Major > Fix For: 3.0.0 > > Attachments: > 0001-NUllPointException-occur-when-import-polices-anf-isO.patch, > ServiceDBStore-L2536.png, image-2021-01-21-11-26-15-996.png, > image-2021-01-21-11-27-05-887.png > > > i got NullPointException when i imported policies json file from rangeradmin > web, i selected "isOverride" flag, it is caused by RANGER-3064, serchFilter > will be null when do deletePolices. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-3135) Ranger always ponit out "User does not have permission for this operation" when user try to query a none-exist role
[
https://issues.apache.org/jira/browse/RANGER-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269051#comment-17269051
]
Mahesh Hanumant Bandal commented on RANGER-3135:
Hi [~rujia1019], any updates for this patch? Please retest and confirm as this
breaks current functionality of ranger.
> Ranger always ponit out "User does not have permission for this operation"
> when user try to query a none-exist role
> ---
>
> Key: RANGER-3135
> URL: https://issues.apache.org/jira/browse/RANGER-3135
> Project: Ranger
> Issue Type: Improvement
> Components: admin
>Affects Versions: 2.0.0, 2.1.0
>Reporter: rujia
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 0001-optimze-log-print-for-querying-roles.patch
>
>
> when i access /role/name/{name}, a Exception "User does not have permission
> for this operation" always been taken, even if this role doesn't exist .
> it is better to prompt user this role is not exist in this case when
> execute-user has admin permission.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3136) NullPointException found when import policies form web side and "isOverride" is selected
[
https://issues.apache.org/jira/browse/RANGER-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269048#comment-17269048
]
Mahesh Hanumant Bandal commented on RANGER-3136:
[~rujia1019] - As per the image that you have shared, I can see that "if"
condition is missing null check for filter.getParams() in your code. Please
refer to this line of code which seems to be missing in your code.
[https://github.com/apache/ranger/blob/4ff1c7ea9984812ec2ea02910ff8bb9eb9003f9a/security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java#L2536]
{code:java}
if(policies != null && filter != null &&
MapUtils.isNotEmpty(filter.getParams())) {
{code}
Please refer to this image -
!ServiceDBStore-L2536.png|width=560,height=287!
Please make sure you have latest code base from [ranger git
repository|https://github.com/apache/ranger].
Thank you.
CC : [~pradeep], [~mehul], [~vel]
> NullPointException found when import policies form web side and "isOverride"
> is selected
>
>
> Key: RANGER-3136
> URL: https://issues.apache.org/jira/browse/RANGER-3136
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 2.1.0
>Reporter: rujia
>Priority: Major
> Fix For: 3.0.0
>
> Attachments:
> 0001-NUllPointException-occur-when-import-polices-anf-isO.patch,
> ServiceDBStore-L2536.png, image-2021-01-21-11-26-15-996.png,
> image-2021-01-21-11-27-05-887.png
>
>
> i got NullPointException when i imported policies json file from rangeradmin
> web, i selected "isOverride" flag, it is caused by RANGER-3064, serchFilter
> will be null when do deletePolices.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (RANGER-3136) NullPointException found when import policies form web side and "isOverride" is selected
[ https://issues.apache.org/jira/browse/RANGER-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal updated RANGER-3136: --- Attachment: ServiceDBStore-L2536.png > NullPointException found when import policies form web side and "isOverride" > is selected > > > Key: RANGER-3136 > URL: https://issues.apache.org/jira/browse/RANGER-3136 > Project: Ranger > Issue Type: Bug > Components: admin >Affects Versions: 2.1.0 >Reporter: rujia >Priority: Major > Fix For: 3.0.0 > > Attachments: > 0001-NUllPointException-occur-when-import-polices-anf-isO.patch, > ServiceDBStore-L2536.png, image-2021-01-21-11-26-15-996.png, > image-2021-01-21-11-27-05-887.png > > > i got NullPointException when i imported policies json file from rangeradmin > web, i selected "isOverride" flag, it is caused by RANGER-3064, serchFilter > will be null when do deletePolices. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-3136) NullPointException found when import policies form web side and "isOverride" is selected
[ https://issues.apache.org/jira/browse/RANGER-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269041#comment-17269041 ] Mahesh Hanumant Bandal commented on RANGER-3136: [~rujia1019] - I have not defined any zones. The steps that I mentioned earlier, I did not created any SecurityZone. So all default policies created with service "hive_one" and "hive_two" are no-zone policies. > NullPointException found when import policies form web side and "isOverride" > is selected > > > Key: RANGER-3136 > URL: https://issues.apache.org/jira/browse/RANGER-3136 > Project: Ranger > Issue Type: Bug > Components: admin >Affects Versions: 2.1.0 >Reporter: rujia >Priority: Major > Fix For: 3.0.0 > > Attachments: > 0001-NUllPointException-occur-when-import-polices-anf-isO.patch, > image-2021-01-21-11-26-15-996.png, image-2021-01-21-11-27-05-887.png > > > i got NullPointException when i imported policies json file from rangeradmin > web, i selected "isOverride" flag, it is caused by RANGER-3064, serchFilter > will be null when do deletePolices. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-3136) NullPointException found when import policies form web side and "isOverride" is selected
[
https://issues.apache.org/jira/browse/RANGER-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268564#comment-17268564
]
Mahesh Hanumant Bandal commented on RANGER-3136:
Hi [~rujia1019] , I tried to reproduce this issue. I took latest code base from
ranger repository [https://github.com/apache/ranger] into my local machine.
I have reverted your patch
[https://github.com/apache/ranger/commit/bd1cf093b7a03431e3bbddf36dd2a565c32dcd40]
in my local repository and build ranger locally.
I have tried following steps to reproduce NPE :
{code:java}
1. Created new hive services named as "hive_one" and "hive_two". These two
services has their default policies.
2. Under "hive_one" service I have created one new policy named as
"test_policy_one".
3. Export policy from service "hive_one" into a json file.
4. Import policies from from json file into service "hive_two" with "Override
Policy" flag is checked.
{code}
I did not see any NullPointerException in logs. I did not see any error while
import policy.
Can you please provide steps to reproduce this issue?
CC: [~pradeep], [~mehul], [~vel]
> NullPointException found when import policies form web side and "isOverride"
> is selected
>
>
> Key: RANGER-3136
> URL: https://issues.apache.org/jira/browse/RANGER-3136
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 2.1.0
>Reporter: rujia
>Priority: Major
> Fix For: 3.0.0
>
> Attachments:
> 0001-NUllPointException-occur-when-import-polices-anf-isO.patch
>
>
> i got NullPointException when i imported policies json file from rangeradmin
> web, i selected "isOverride" flag, it is caused by RANGER-3064, serchFilter
> will be null when do deletePolices.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (RANGER-3135) Ranger always ponit out "User does not have permission for this operation" when user try to query a none-exist role
[
https://issues.apache.org/jira/browse/RANGER-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267925#comment-17267925
]
Mahesh Hanumant Bandal edited comment on RANGER-3135 at 1/19/21, 2:36 PM:
--
Hi [~rujia1019], Thank you for suggesting this improvement.
I tried to access GET API
[http://localhost:6080/service/roles/roles/name/]role1 with admin user and it
does not return roles which already exist, whereas select * from x_role returns
two roles. ie. role1, role2.
API returns following response :
{code:java}
Role with name: role1 does not exist
1
{code}
*With admin user it shows same response regardless of role's existence.*
This patch breaks current functionality of ranger. Can you please retest and
confirm?
CC : [~pradeep], [~mehul], [~vel]
was (Author: maheshbandal):
Hi [~rujia1019], Thank you for suggesting this improvement.
I tried to access GET API
[http://localhost:6080/service/roles/roles/name/]role1 with admin user and it
does not return roles which already exist, whereas select * from x_role returns
two roles. ie. role1, role2.
API returns following response :
{code:java}
Role with name: role1 does not exist
1
{code}
*With admin user it shows same response regardless of role's existence.*
This patch breaks current functionality of ranger. Can you please retest and
confirm?
> Ranger always ponit out "User does not have permission for this operation"
> when user try to query a none-exist role
> ---
>
> Key: RANGER-3135
> URL: https://issues.apache.org/jira/browse/RANGER-3135
> Project: Ranger
> Issue Type: Improvement
> Components: admin
>Affects Versions: 2.0.0, 2.1.0
>Reporter: rujia
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 0001-optimze-log-print-for-querying-roles.patch
>
>
> when i access /role/name/{name}, a Exception "User does not have permission
> for this operation" always been taken, even if this role doesn't exist .
> it is better to prompt user this role is not exist in this case when
> execute-user has admin permission.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3135) Ranger always ponit out "User does not have permission for this operation" when user try to query a none-exist role
[
https://issues.apache.org/jira/browse/RANGER-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267925#comment-17267925
]
Mahesh Hanumant Bandal commented on RANGER-3135:
Hi [~rujia1019], Thank you for suggesting this improvement.
I tried to access GET API
[http://localhost:6080/service/roles/roles/name/]role1 with admin user and it
does not return roles which already exist, whereas select * from x_role returns
two roles. ie. role1, role2.
API returns following response :
{code:java}
Role with name: role1 does not exist
1
{code}
*With admin user it shows same response regardless of role's existence.*
This patch breaks current functionality of ranger. Can you please retest and
confirm?
> Ranger always ponit out "User does not have permission for this operation"
> when user try to query a none-exist role
> ---
>
> Key: RANGER-3135
> URL: https://issues.apache.org/jira/browse/RANGER-3135
> Project: Ranger
> Issue Type: Improvement
> Components: admin
>Affects Versions: 2.0.0, 2.1.0
>Reporter: rujia
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 0001-optimze-log-print-for-querying-roles.patch
>
>
> when i access /role/name/{name}, a Exception "User does not have permission
> for this operation" always been taken, even if this role doesn't exist .
> it is better to prompt user this role is not exist in this case when
> execute-user has admin permission.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (RANGER-3118) Upgrade jackson to 2.11.3
[ https://issues.apache.org/jira/browse/RANGER-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal resolved RANGER-3118. Fix Version/s: 2.2.0 Resolution: Fixed commit id on master branch : [e93af53f9a097170e3e027c44258f8ffe919d28b|https://github.com/apache/ranger/commit/e93af53f9a097170e3e027c44258f8ffe919d28b] commit id on ranger-2.2 branch : [f1d5bf18e6490d7b81d3550c61b08318a71691ec|https://github.com/apache/ranger/commit/f1d5bf18e6490d7b81d3550c61b08318a71691ec] > Upgrade jackson to 2.11.3 > - > > Key: RANGER-3118 > URL: https://issues.apache.org/jira/browse/RANGER-3118 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 3.0.0 >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Minor > Fix For: 3.0.0, 2.2.0 > > > Upgrade jackson version to 2.11.3 for best practices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3119) Upgrade to slf4j 1.7.30
Mahesh Hanumant Bandal created RANGER-3119: -- Summary: Upgrade to slf4j 1.7.30 Key: RANGER-3119 URL: https://issues.apache.org/jira/browse/RANGER-3119 Project: Ranger Issue Type: Improvement Components: Ranger Affects Versions: 3.0.0 Reporter: Mahesh Hanumant Bandal Assignee: Mahesh Hanumant Bandal Fix For: 3.0.0 Upgrade slf4j version to 1.7.30 for best practices. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-3118) Upgrade jackson to 2.11.3
[ https://issues.apache.org/jira/browse/RANGER-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal updated RANGER-3118: --- Issue Type: Improvement (was: Bug) > Upgrade jackson to 2.11.3 > - > > Key: RANGER-3118 > URL: https://issues.apache.org/jira/browse/RANGER-3118 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 3.0.0 >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Minor > Fix For: 3.0.0 > > > Upgrade jackson version to 2.11.3 for best practices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3118) Upgrade jackson to 2.11.3
Mahesh Hanumant Bandal created RANGER-3118: -- Summary: Upgrade jackson to 2.11.3 Key: RANGER-3118 URL: https://issues.apache.org/jira/browse/RANGER-3118 Project: Ranger Issue Type: Bug Components: Ranger Affects Versions: 3.0.0 Reporter: Mahesh Hanumant Bandal Assignee: Mahesh Hanumant Bandal Fix For: 3.0.0 Upgrade jackson version to 2.11.3 for best practices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-3100) Upgrade httpclient version from 4.5.6 to 4.5.13+ due to CVE-2020-13956
[ https://issues.apache.org/jira/browse/RANGER-3100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249546#comment-17249546 ] Mahesh Hanumant Bandal commented on RANGER-3100: [~pradeep], I have updated httpcore version and also made changes into agents-audit/pom.xml. Please refer to this patch [^RANGER-3100-V1.patch] > Upgrade httpclient version from 4.5.6 to 4.5.13+ due to CVE-2020-13956 > -- > > Key: RANGER-3100 > URL: https://issues.apache.org/jira/browse/RANGER-3100 > Project: Ranger > Issue Type: Bug > Components: Ranger >Reporter: Saket Jajoo >Assignee: Pradeep Agrawal >Priority: Major > Attachments: > 0001-RANGER-3100-Upgrade-httpclient-version-from-4.5.6-to.patch, > RANGER-3100-V1.patch > > > Ranger is pulling in [httpclient > 4.5.6|https://github.com/apache/ranger/blob/2f4277eb15f86b82718d05989de5545687c33539/pom.xml#L132]. > Please upgrade to httpclient 4.5.13+. > CVE-2020-13956: https://snyk.io/vuln/SNYK-JAVA-ORGAPACHEHTTPCOMPONENTS-1016906 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-3100) Upgrade httpclient version from 4.5.6 to 4.5.13+ due to CVE-2020-13956
[ https://issues.apache.org/jira/browse/RANGER-3100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal updated RANGER-3100: --- Attachment: RANGER-3100-V1.patch > Upgrade httpclient version from 4.5.6 to 4.5.13+ due to CVE-2020-13956 > -- > > Key: RANGER-3100 > URL: https://issues.apache.org/jira/browse/RANGER-3100 > Project: Ranger > Issue Type: Bug > Components: Ranger >Reporter: Saket Jajoo >Assignee: Pradeep Agrawal >Priority: Major > Attachments: > 0001-RANGER-3100-Upgrade-httpclient-version-from-4.5.6-to.patch, > RANGER-3100-V1.patch > > > Ranger is pulling in [httpclient > 4.5.6|https://github.com/apache/ranger/blob/2f4277eb15f86b82718d05989de5545687c33539/pom.xml#L132]. > Please upgrade to httpclient 4.5.13+. > CVE-2020-13956: https://snyk.io/vuln/SNYK-JAVA-ORGAPACHEHTTPCOMPONENTS-1016906 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-3092) KMS fails to start with NullPointerException in catalina.out logs
[
https://issues.apache.org/jira/browse/RANGER-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242690#comment-17242690
]
Mahesh Hanumant Bandal commented on RANGER-3092:
[~rujia1019] - Ranger-KMS uses same EmbeddedServer as of Ranger-Admin and
"log4j.configuration" command-line argument was missing in Ranger-KMS startup
script.
> KMS fails to start with NullPointerException in catalina.out logs
> -
>
> Key: RANGER-3092
> URL: https://issues.apache.org/jira/browse/RANGER-3092
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 3.0.0
>Reporter: Mahesh Hanumant Bandal
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Fix For: 3.0.0
>
>
> Ranger KMS setup.sh executes successfully, however when we try to start KMS
> service with following command :
> {code:java}
> ./ranger-kms start
> {code}
> It shows "Apache Ranger KMS Service failed to start" and only catalina.out
> log file is created in ews/logs folder.
> Following error at ews/logs/catalina.out file :
> {code:java}
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.ranger.server.tomcat.EmbeddedServer.(EmbeddedServer.java:94)
> at
> org.apache.ranger.server.tomcat.EmbeddedServer.main(EmbeddedServer.java:85){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (RANGER-3090) KMS setup.sh fails due to blank property value of ranger.ks.db.ssl.certificateFile
[
https://issues.apache.org/jira/browse/RANGER-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahesh Hanumant Bandal resolved RANGER-3090.
Resolution: Fixed
commit id master branch :
[7d821605e7f3ac2916970a8a75c67ec649135813|https://github.com/apache/ranger/commit/7d821605e7f3ac2916970a8a75c67ec649135813]
commit id ranger-2.2 branch :
[94e347647b87096cb3ce8e7ed51f9cd27b795388|https://github.com/apache/ranger/commit/94e347647b87096cb3ce8e7ed51f9cd27b795388]
> KMS setup.sh fails due to blank property value of
> ranger.ks.db.ssl.certificateFile
> --
>
> Key: RANGER-3090
> URL: https://issues.apache.org/jira/browse/RANGER-3090
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Mahesh Hanumant Bandal
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
>
> KMS setup.sh fails due to blank property value of db_ssl_certificate_file in
> install.properties file
>
> {code:java}
> [I]
> /root/tarfile/ranger-3.0.0-SNAPSHOT-kms/ews/webapp/WEB-INF/classes/conf/dbks-site.xml
> file found
> Traceback (most recent call last):
> File "update_property.py", line 40, in
> write_properties_to_xml(file_path,parameter_name,parameter_value)
> File "update_property.py", line 22, in write_properties_to_xml
> if(os.path.isfile(xml_path)):
> File "/usr/lib64/python2.7/genericpath.py", line 29, in isfile
> st = os.stat(path)
> TypeError: coercing to Unicode: need string or buffer, NoneType found
> [E] Update property failed for: {'ranger.ks.db.ssl.certificateFile'}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3092) KMS fails to start with NullPointerException in catalina.out logs
[
https://issues.apache.org/jira/browse/RANGER-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240817#comment-17240817
]
Mahesh Hanumant Bandal commented on RANGER-3092:
cc: [~rujia1019] , [~pradeepagrawal8184]
> KMS fails to start with NullPointerException in catalina.out logs
> -
>
> Key: RANGER-3092
> URL: https://issues.apache.org/jira/browse/RANGER-3092
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 3.0.0
>Reporter: Mahesh Hanumant Bandal
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Fix For: 3.0.0
>
>
> Ranger KMS setup.sh executes successfully, however when we try to start KMS
> service with following command :
> {code:java}
> ./ranger-kms start
> {code}
> It shows "Apache Ranger KMS Service failed to start" and only catalina.out
> log file is created in ews/logs folder.
> Following error at ews/logs/catalina.out file :
> {code:java}
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.ranger.server.tomcat.EmbeddedServer.(EmbeddedServer.java:94)
> at
> org.apache.ranger.server.tomcat.EmbeddedServer.main(EmbeddedServer.java:85){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (RANGER-3092) KMS fails to start with NullPointerException in catalina.out logs
Mahesh Hanumant Bandal created RANGER-3092:
--
Summary: KMS fails to start with NullPointerException in
catalina.out logs
Key: RANGER-3092
URL: https://issues.apache.org/jira/browse/RANGER-3092
Project: Ranger
Issue Type: Bug
Components: Ranger
Affects Versions: 3.0.0
Reporter: Mahesh Hanumant Bandal
Assignee: Mahesh Hanumant Bandal
Fix For: 3.0.0
Ranger KMS setup.sh executes successfully, however when we try to start KMS
service with following command :
{code:java}
./ranger-kms start
{code}
It shows "Apache Ranger KMS Service failed to start" and only catalina.out log
file is created in ews/logs folder.
Following error at ews/logs/catalina.out file :
{code:java}
Exception in thread "main" java.lang.NullPointerException
at
org.apache.ranger.server.tomcat.EmbeddedServer.(EmbeddedServer.java:94)
at
org.apache.ranger.server.tomcat.EmbeddedServer.main(EmbeddedServer.java:85){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (RANGER-3090) KMS setup.sh fails due to blank property value of ranger.ks.db.ssl.certificateFile
[
https://issues.apache.org/jira/browse/RANGER-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahesh Hanumant Bandal updated RANGER-3090:
---
Fix Version/s: 2.2.0
3.0.0
> KMS setup.sh fails due to blank property value of
> ranger.ks.db.ssl.certificateFile
> --
>
> Key: RANGER-3090
> URL: https://issues.apache.org/jira/browse/RANGER-3090
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Mahesh Hanumant Bandal
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
>
> KMS setup.sh fails due to blank property value of db_ssl_certificate_file in
> install.properties file
>
> {code:java}
> [I]
> /root/tarfile/ranger-3.0.0-SNAPSHOT-kms/ews/webapp/WEB-INF/classes/conf/dbks-site.xml
> file found
> Traceback (most recent call last):
> File "update_property.py", line 40, in
> write_properties_to_xml(file_path,parameter_name,parameter_value)
> File "update_property.py", line 22, in write_properties_to_xml
> if(os.path.isfile(xml_path)):
> File "/usr/lib64/python2.7/genericpath.py", line 29, in isfile
> st = os.stat(path)
> TypeError: coercing to Unicode: need string or buffer, NoneType found
> [E] Update property failed for: {'ranger.ks.db.ssl.certificateFile'}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (RANGER-3090) KMS setup.sh fails due to blank property value of ranger.ks.db.ssl.certificateFile
[
https://issues.apache.org/jira/browse/RANGER-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahesh Hanumant Bandal updated RANGER-3090:
---
Affects Version/s: 2.2.0
3.0.0
> KMS setup.sh fails due to blank property value of
> ranger.ks.db.ssl.certificateFile
> --
>
> Key: RANGER-3090
> URL: https://issues.apache.org/jira/browse/RANGER-3090
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Mahesh Hanumant Bandal
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
>
> KMS setup.sh fails due to blank property value of db_ssl_certificate_file in
> install.properties file
>
> {code:java}
> [I]
> /root/tarfile/ranger-3.0.0-SNAPSHOT-kms/ews/webapp/WEB-INF/classes/conf/dbks-site.xml
> file found
> Traceback (most recent call last):
> File "update_property.py", line 40, in
> write_properties_to_xml(file_path,parameter_name,parameter_value)
> File "update_property.py", line 22, in write_properties_to_xml
> if(os.path.isfile(xml_path)):
> File "/usr/lib64/python2.7/genericpath.py", line 29, in isfile
> st = os.stat(path)
> TypeError: coercing to Unicode: need string or buffer, NoneType found
> [E] Update property failed for: {'ranger.ks.db.ssl.certificateFile'}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (RANGER-3090) KMS setup.sh fails due to blank property value of ranger.ks.db.ssl.certificateFile
Mahesh Hanumant Bandal created RANGER-3090:
--
Summary: KMS setup.sh fails due to blank property value of
ranger.ks.db.ssl.certificateFile
Key: RANGER-3090
URL: https://issues.apache.org/jira/browse/RANGER-3090
Project: Ranger
Issue Type: Bug
Components: Ranger
Reporter: Mahesh Hanumant Bandal
Assignee: Mahesh Hanumant Bandal
KMS setup.sh fails due to blank property value of db_ssl_certificate_file in
install.properties file
{code:java}
[I]
/root/tarfile/ranger-3.0.0-SNAPSHOT-kms/ews/webapp/WEB-INF/classes/conf/dbks-site.xml
file found
Traceback (most recent call last):
File "update_property.py", line 40, in
write_properties_to_xml(file_path,parameter_name,parameter_value)
File "update_property.py", line 22, in write_properties_to_xml
if(os.path.isfile(xml_path)):
File "/usr/lib64/python2.7/genericpath.py", line 29, in isfile
st = os.stat(path)
TypeError: coercing to Unicode: need string or buffer, NoneType found
[E] Update property failed for: {'ranger.ks.db.ssl.certificateFile'}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (RANGER-3084) Ranger database connection fails when postgres is SSL enabled & postgresql-42.2.14 driver jar is used
[ https://issues.apache.org/jira/browse/RANGER-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal resolved RANGER-3084. Resolution: Fixed commit id on master branch: [https://github.com/apache/ranger/commit/3a21d6a30232ee44d375a2fad5764798afb74f7e|https://github.com/apache/ranger/commit/3a21d6a30232ee44d375a2fad5764798afb74f7e] > Ranger database connection fails when postgres is SSL enabled & > postgresql-42.2.14 driver jar is used > - > > Key: RANGER-3084 > URL: https://issues.apache.org/jira/browse/RANGER-3084 > Project: Ranger > Issue Type: Bug > Components: Ranger >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Major > Fix For: 3.0.0, 2.2.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3084) Ranger database connection fails when postgres is SSL enabled & postgresql-42.2.14 driver jar is used
Mahesh Hanumant Bandal created RANGER-3084: -- Summary: Ranger database connection fails when postgres is SSL enabled & postgresql-42.2.14 driver jar is used Key: RANGER-3084 URL: https://issues.apache.org/jira/browse/RANGER-3084 Project: Ranger Issue Type: Bug Components: Ranger Reporter: Mahesh Hanumant Bandal Assignee: Mahesh Hanumant Bandal -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (RANGER-3067) Schema changes to improve performance of chained plugin feature
[ https://issues.apache.org/jira/browse/RANGER-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal resolved RANGER-3067. Resolution: Fixed commit id : [https://github.com/apache/ranger/commit/34a948c34ab135e75edba1a48f56fdeca2ad3c74|https://github.com/apache/ranger/commit/34a948c34ab135e75edba1a48f56fdeca2ad3c74] > Schema changes to improve performance of chained plugin feature > --- > > Key: RANGER-3067 > URL: https://issues.apache.org/jira/browse/RANGER-3067 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Major > Fix For: 2.2.0 > > > Adding index on table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-3067) Schema changes to improve performance of chained plugin feature
[ https://issues.apache.org/jira/browse/RANGER-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal updated RANGER-3067: --- Description: Adding index on table. (was: Create index on 'resource_signature' column of 'x_rms_service_resource' table.) > Schema changes to improve performance of chained plugin feature > --- > > Key: RANGER-3067 > URL: https://issues.apache.org/jira/browse/RANGER-3067 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Major > Fix For: 2.2.0 > > > Adding index on table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-3067) Schema changes to improve performance of chained plugin feature
[ https://issues.apache.org/jira/browse/RANGER-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal updated RANGER-3067: --- Summary: Schema changes to improve performance of chained plugin feature (was: Create index on resource_signature column of x_rms_service_resource table) > Schema changes to improve performance of chained plugin feature > --- > > Key: RANGER-3067 > URL: https://issues.apache.org/jira/browse/RANGER-3067 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Major > Fix For: 2.2.0 > > > Create index on 'resource_signature' column of 'x_rms_service_resource' table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3067) Create index on resource_signature column of x_rms_service_resource table
Mahesh Hanumant Bandal created RANGER-3067: -- Summary: Create index on resource_signature column of x_rms_service_resource table Key: RANGER-3067 URL: https://issues.apache.org/jira/browse/RANGER-3067 Project: Ranger Issue Type: Improvement Components: Ranger Reporter: Mahesh Hanumant Bandal Assignee: Mahesh Hanumant Bandal Fix For: 2.2.0 Create index on 'resource_signature' column of 'x_rms_service_resource' table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-3046) Policy Labels are not displayed in Ranger audits Admin tab for policy create popup
[ https://issues.apache.org/jira/browse/RANGER-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal updated RANGER-3046: --- Fix Version/s: 2.2.0 > Policy Labels are not displayed in Ranger audits Admin tab for policy create > popup > -- > > Key: RANGER-3046 > URL: https://issues.apache.org/jira/browse/RANGER-3046 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 2.2.0 >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Major > Fix For: 2.2.0 > > Attachments: policyLabels.png > > > When policy is created it saves label name into table 'x_policy_label' and > maps policy_label_id with policy_id into table 'x_policy_label_map'. > However, in 'x_trx_log' table value is empty against attribute name "Policy > Labels" for create action. Therefore, policy create popup in Admin audits tab > does not display Policy Labels. Refer to the following image. > !policyLabels.png|width=538,height=245! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-3046) Policy Labels are not displayed in Ranger audits Admin tab for policy create popup
[ https://issues.apache.org/jira/browse/RANGER-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal updated RANGER-3046: --- Affects Version/s: 2.2.0 > Policy Labels are not displayed in Ranger audits Admin tab for policy create > popup > -- > > Key: RANGER-3046 > URL: https://issues.apache.org/jira/browse/RANGER-3046 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 2.2.0 >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Major > Attachments: policyLabels.png > > > When policy is created it saves label name into table 'x_policy_label' and > maps policy_label_id with policy_id into table 'x_policy_label_map'. > However, in 'x_trx_log' table value is empty against attribute name "Policy > Labels" for create action. Therefore, policy create popup in Admin audits tab > does not display Policy Labels. Refer to the following image. > !policyLabels.png|width=538,height=245! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-3046) Policy Labels are not displayed in Ranger audits Admin tab for policy create popup
[ https://issues.apache.org/jira/browse/RANGER-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal updated RANGER-3046: --- Description: When policy is created it saves label name into table 'x_policy_label' and maps policy_label_id with policy_id into table 'x_policy_label_map'. However, in 'x_trx_log' table value is empty against attribute name "Policy Labels" for create action. Therefore, policy create popup in Admin audits tab does not display Policy Labels. Refer to the following image. !policyLabels.png|width=538,height=245! was: When policy is created it saves label name into table 'x_policy_label' and maps policy_label_id with policy_id into table 'x_policy_label_map'. However, in 'x_trx_log' table value is empty against attribute name "Policy Labels" for create action. Therefore, policy create popup in Admin audits tab does not display Policy Labels. > Policy Labels are not displayed in Ranger audits Admin tab for policy create > popup > -- > > Key: RANGER-3046 > URL: https://issues.apache.org/jira/browse/RANGER-3046 > Project: Ranger > Issue Type: Bug > Components: Ranger >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Major > Attachments: policyLabels.png > > > When policy is created it saves label name into table 'x_policy_label' and > maps policy_label_id with policy_id into table 'x_policy_label_map'. > However, in 'x_trx_log' table value is empty against attribute name "Policy > Labels" for create action. Therefore, policy create popup in Admin audits tab > does not display Policy Labels. Refer to the following image. > !policyLabels.png|width=538,height=245! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-3046) Policy Labels are not displayed in Ranger audits Admin tab for policy create popup
[ https://issues.apache.org/jira/browse/RANGER-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal updated RANGER-3046: --- Attachment: policyLabels.png > Policy Labels are not displayed in Ranger audits Admin tab for policy create > popup > -- > > Key: RANGER-3046 > URL: https://issues.apache.org/jira/browse/RANGER-3046 > Project: Ranger > Issue Type: Bug > Components: Ranger >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Major > Attachments: policyLabels.png > > > When policy is created it saves label name into table 'x_policy_label' and > maps policy_label_id with policy_id into table 'x_policy_label_map'. > However, in 'x_trx_log' table value is empty against attribute name "Policy > Labels" for create action. Therefore, policy create popup in Admin audits tab > does not display Policy Labels. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3046) Policy Labels are not displayed in Ranger audits Admin tab for policy create popup
Mahesh Hanumant Bandal created RANGER-3046: -- Summary: Policy Labels are not displayed in Ranger audits Admin tab for policy create popup Key: RANGER-3046 URL: https://issues.apache.org/jira/browse/RANGER-3046 Project: Ranger Issue Type: Bug Components: Ranger Reporter: Mahesh Hanumant Bandal Assignee: Mahesh Hanumant Bandal When policy is created it saves label name into table 'x_policy_label' and maps policy_label_id with policy_id into table 'x_policy_label_map'. However, in 'x_trx_log' table value is empty against attribute name "Policy Labels" for create action. Therefore, policy create popup in Admin audits tab does not display Policy Labels. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-3027) Improve response time for GET API service/xusers/users
[
https://issues.apache.org/jira/browse/RANGER-3027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17212297#comment-17212297
]
Mahesh Hanumant Bandal commented on RANGER-3027:
Okay. Sorry, I was referring to your review request. Please update your patch
on [Apache Review Request|https://reviews.apache.org/r/72939/] as well. We
usually download patch from there. Thanks, I am able to apply
[^0001-RANGER-3027-Improve-response-time-for-GET-API-servic.patch] this patch.
I will try this and let you know my findings.
> Improve response time for GET API service/xusers/users
> --
>
> Key: RANGER-3027
> URL: https://issues.apache.org/jira/browse/RANGER-3027
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Affects Versions: 2.2.0
>Reporter: Mahesh Hanumant Bandal
>Priority: Minor
> Attachments:
> 0001-RANGER-3027-Improve-response-time-for-GET-API-servic.patch,
> snapshot1.png, snapshot2.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When there are large number of users and group_users data in database; GET
> API service/xusers/users takes longer time to respond if we set request
> parameter pageSize=1000 or more.
> However, with default value pageSize=25 response received instantaneously.
> {code:java}
> select count(*) from x_user;
> 50052
> select count(*) from x_group_users;
> 500056
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3027) Improve response time for GET API service/xusers/users
[
https://issues.apache.org/jira/browse/RANGER-3027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17212274#comment-17212274
]
Mahesh Hanumant Bandal commented on RANGER-3027:
Hi [~RickyMa], I can see conflicts in
security-admin/src/main/java/org/apache/ranger/db/XXGroupUserDao.java file.
Please take latest pull on master branch and update your review request.
> Improve response time for GET API service/xusers/users
> --
>
> Key: RANGER-3027
> URL: https://issues.apache.org/jira/browse/RANGER-3027
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Affects Versions: 2.2.0
>Reporter: Mahesh Hanumant Bandal
>Priority: Minor
> Attachments:
> 0001-RANGER-3027-Improve-response-time-for-GET-API-servic.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When there are large number of users and group_users data in database; GET
> API service/xusers/users takes longer time to respond if we set request
> parameter pageSize=1000 or more.
> However, with default value pageSize=25 response received instantaneously.
> {code:java}
> select count(*) from x_user;
> 50052
> select count(*) from x_group_users;
> 500056
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3027) Improve response time for GET API service/xusers/users
[
https://issues.apache.org/jira/browse/RANGER-3027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17212243#comment-17212243
]
Mahesh Hanumant Bandal commented on RANGER-3027:
Hi [~RickyMa], I tried to apply your patch but its failing. Can you please
update your review request with latest ranger code.
> Improve response time for GET API service/xusers/users
> --
>
> Key: RANGER-3027
> URL: https://issues.apache.org/jira/browse/RANGER-3027
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Affects Versions: 2.2.0
>Reporter: Mahesh Hanumant Bandal
>Priority: Minor
> Attachments:
> 0001-RANGER-3027-Improve-response-time-for-GET-API-servic.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When there are large number of users and group_users data in database; GET
> API service/xusers/users takes longer time to respond if we set request
> parameter pageSize=1000 or more.
> However, with default value pageSize=25 response received instantaneously.
> {code:java}
> select count(*) from x_user;
> 50052
> select count(*) from x_group_users;
> 500056
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (RANGER-3027) Improve response time for GET API service/xusers/users
[
https://issues.apache.org/jira/browse/RANGER-3027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahesh Hanumant Bandal updated RANGER-3027:
---
Priority: Minor (was: Major)
> Improve response time for GET API service/xusers/users
> --
>
> Key: RANGER-3027
> URL: https://issues.apache.org/jira/browse/RANGER-3027
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Affects Versions: 2.2.0
>Reporter: Mahesh Hanumant Bandal
>Priority: Minor
>
> When there are large number of users and group_users data in database; GET
> API service/xusers/users takes longer time to respond if we set request
> parameter pageSize=1000 or more.
> However, with default value pageSize=25 response received instantaneously.
> {code:java}
> select count(*) from x_user;
> 50052
> select count(*) from x_group_users;
> 500056
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3027) Improve response time for GET API service/xusers/users
[
https://issues.apache.org/jira/browse/RANGER-3027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17209588#comment-17209588
]
Mahesh Hanumant Bandal commented on RANGER-3027:
Existing patch of RANGER-2789 caused breakage in functionality of +user search
in permission tab+. Thus reverted it through Jira RANGER-3014.
> Improve response time for GET API service/xusers/users
> --
>
> Key: RANGER-3027
> URL: https://issues.apache.org/jira/browse/RANGER-3027
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Affects Versions: 2.2.0
>Reporter: Mahesh Hanumant Bandal
>Priority: Major
>
> When there are large number of users and group_users data in database; GET
> API service/xusers/users takes longer time to respond if we set request
> parameter pageSize=1000 or more.
> However, with default value pageSize=25 response received instantaneously.
> {code:java}
> select count(*) from x_user;
> 50052
> select count(*) from x_group_users;
> 500056
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3014) fix for RANGER-2789 breaks current functionality
[
https://issues.apache.org/jira/browse/RANGER-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17209583#comment-17209583
]
Mahesh Hanumant Bandal commented on RANGER-3014:
Hi [~RickyMa],
We have reverted
[RANGER-2789|https://issues.apache.org/jira/browse/RANGER-2789] because it was
breaking existing functionality of user search. We can track this issue
separately. Please refer to this new Jira
[RANGER-3027|https://issues.apache.org/jira/browse/RANGER-3027].
> fix for RANGER-2789 breaks current functionality
>
>
> Key: RANGER-3014
> URL: https://issues.apache.org/jira/browse/RANGER-3014
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: Georgi Ivanov
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Attachments:
> 0001-RANGER-3014-fix-for-RANGER-2789-breaks-current-funct.patch, After
> Reverting RANGER-2789.png, Reverted RANGER-2789- userlookup in permission
> tab.png, add-a-query-condition.png, cpu_utilization_for_user_lookup.png,
> ranger-crashed while user-lookup in permission tab.png, with RANGER-2789 and
> RANGER-3014 .png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Since we upgraded to Ranger 2.1.0 in our dev env, we've noticed that user
> list page in Ranger Admin UI is not showing (or it is very very slow - in the
> order of tens of minutes).
> Looking at the commit history we found that the reason was commit
> *f45054d1b9* which was meant as a performance improvement for RANGER-2789.
> Our ranger usersync fetches users and groups from AD. Our tree is huge, here
> are some stats:
> {code:java}
> select count(*) from x_user;
> 43368
> select count(*) from x_portal_user;
> 43366
> select count(*) from x_group;
> 17865
> select count(*) from x_group_users;
> 366180 {code}
>
> Looking at the commit *f45054d1b9* what it meant to solve is perform a user
> lookup and fetching user info such as attributes and group membership in
> bulk, instead of doing it in a loop, one by one. In order to do that it
> provided couple of methods and also an override for searchXUsers in
> service/XUserService.java (before we used the parent method in
> service/XUserServiceBase.java).
>
> The new searchXUsers method (which gets invoked when we call
> /service/xusers/users REST API, calls populateViewBeans (another new method).
> It calls the parent method populateViewBeans in XUserServiceBase.java which
> build a hashmap or users and calls an override of populateViewBeans with
> input hashmap
> {code:java}
> + public List populateViewBeans(List resources) {
> + List viewBeans = new ArrayList<>();
> + if (CollectionUtils.isNotEmpty(resources)) {
> + Map resourceViewBeanMap = new
> HashMap<>(resources.size());
> + Map viewBeanResourceMap = new
> HashMap<>(resources.size());
> + for (XXUser resource : resources) {
> + VXUser viewBean = createViewObject();
> +
> viewBean.setCredStoreId(resource.getCredStoreId());
> +
> viewBean.setDescription(resource.getDescription());
> + viewBean.setName(resource.getName());
> + viewBean.setStatus(resource.getStatus());
> + resourceViewBeanMap.put(resource, viewBean);
> + viewBeanResourceMap.put(viewBean, resource);
> + viewBeans.add(viewBean);
> + }
> + populateViewBeans(resourceViewBeanMap);
> + mapEntityToViewBeans(viewBeanResourceMap);
> + }
> + return viewBeans;
> + }
> +
> + protected void populateViewBeans(Map
> resourceViewBeanMap) {
> + mapBaseAttributesToViewBeans(resourceViewBeanMap);
> + } {code}
>
> This in turns calls mapBaseAttributesToViewBeans, which calls
> daoManager.getXXPortalUser().findAllXPortalUser() and it pulls all users (no
> matter that we limit the users with a REST call to 25 by default)
> That's one thing that hampers performance. However the biggest issue is this:
> {code:java}
> + @Override
> + public List populateViewBeans(List xUsers) {
> + List vObjList = super.populateViewBeans(xUsers);
> + if (CollectionUtils.isNotEmpty(vObjList) &&
> CollectionUtils.isNotEmpty(xUsers) && xUsers.size() == vObjList.size()) {
> + Map xUserIdVObjMap = new
> HashMap<>(xUsers.size());
> + for (int i = 0; i < xUsers.size(); ++i) {
> + VXUser vObj = vObjList.get(i);
> +
[jira] [Created] (RANGER-3027) Improve response time for GET API service/xusers/users
Mahesh Hanumant Bandal created RANGER-3027:
--
Summary: Improve response time for GET API service/xusers/users
Key: RANGER-3027
URL: https://issues.apache.org/jira/browse/RANGER-3027
Project: Ranger
Issue Type: Improvement
Components: Ranger
Affects Versions: 2.2.0
Reporter: Mahesh Hanumant Bandal
When there are large number of users and group_users data in database; GET API
service/xusers/users takes longer time to respond if we set request parameter
pageSize=1000 or more.
However, with default value pageSize=25 response received instantaneously.
{code:java}
select count(*) from x_user;
50052
select count(*) from x_group_users;
500056
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (RANGER-3024) Improve response time and refactor code for GET API /service/xusers/lookup/users
Mahesh Hanumant Bandal created RANGER-3024:
--
Summary: Improve response time and refactor code for GET API
/service/xusers/lookup/users
Key: RANGER-3024
URL: https://issues.apache.org/jira/browse/RANGER-3024
Project: Ranger
Issue Type: Improvement
Components: Ranger
Affects Versions: 2.2.0
Reporter: Mahesh Hanumant Bandal
Assignee: Mahesh Hanumant Bandal
GET API /service/xusers/lookup/users returns List of String object which
contains username. But it also traverse through groupUser list while populating
view object VXUser. When there are large number of users and group_users data
in database; it takes longer time to respond if we set request parameter
pageSize=1000 or more. This causes performance issue and needs to be improved.
{code:java}
select count(*) from x_user;
50052
select count(*) from x_group_users;
500056
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (RANGER-3023) Permission tab takes longer time to load with large number of users and group_users data
Mahesh Hanumant Bandal created RANGER-3023:
--
Summary: Permission tab takes longer time to load with large
number of users and group_users data
Key: RANGER-3023
URL: https://issues.apache.org/jira/browse/RANGER-3023
Project: Ranger
Issue Type: Improvement
Components: Ranger
Affects Versions: 2.2.0
Reporter: Mahesh Hanumant Bandal
Assignee: Mahesh Hanumant Bandal
GET API /service/xusers/permission takes longer time to load with following
number of users and group mappings in db.
{code:java}
select count(*) from x_user;
50052
select count(*) from x_group_users;
500056
{code}
Need to have performance optimization on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (RANGER-3014) fix for RANGER-2789 breaks current functionality
[
https://issues.apache.org/jira/browse/RANGER-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17208740#comment-17208740
]
Mahesh Hanumant Bandal edited comment on RANGER-3014 at 10/6/20, 1:23 PM:
--
Hi [~RickyMa],
As I have mentioned earlier "*User lookup in permission tab breaks current
functionality*". I have tested with RANGER-2789 patch and also with your
[latest patch|https://reviews.apache.org/r/72926/] . Please refer to following
image
!ranger-crashed while user-lookup in permission tab.png|width=448,height=198!
When ranger tries to lookup for users in permission tab, and if we keep
typing/deleting few letters in "Select and Add User" textbox; it sends requests
for user search on keyPressEvent from UI side. After sometime all request
starts failing and ranger stops responding. Also, I checked cpu utilization and
it reaches over 370%. Please refer to this image
!cpu_utilization_for_user_lookup.png|width=210,height=176!
Apart from this, on UI while loading User Tab with default pageSize=25. It
takes 2.92 seconds. Please refer image below
!with RANGER-2789 and RANGER-3014 .png|width=330,height=147!
*Following are analysis after reverting RANGER-2789* :
If we revert RANGER-2789 it took *319.15 milliseconds* with default
pageSize=25, which is comparatively very less. Please refer following image.
!After Reverting RANGER-2789.png|width=321,height=143!
Also, user lookup in permission tab also works smoothly without failing. Please
refer following image
!Reverted RANGER-2789- userlookup in permission tab.png|width=321,height=144!
However, *pageSize=1000 took 16 seconds* to respond but this is related to
RANGER-2789 and this performance issue can be tracked separately. For now, we
need revert this patch as it breaks current functionality as I explained.
Thanks,
Mahesh Bandal
was (Author: maheshbandal):
Hi [~RickyMa],
As I have mentioned earlier "*User lookup in permission tab breaks current
functionality*". I have tested with RANGER-2789 patch and also with your
[latest patch|https://reviews.apache.org/r/72926/] . Please refer to following
image !ranger-crashed while user-lookup in permission
tab.png|width=448,height=198! .
When ranger tries to lookup for users in permission tab, and if we keep
typing/deleting few letters in "Select and Add User" textbox; it sends requests
for user search on keyPressEvent from UI side. After sometime all request
starts failing and ranger stops responding. Also, I checked cpu utilization and
it reaches over 370%. Please refer to this image
!cpu_utilization_for_user_lookup.png|width=210,height=176!
Apart from this, on UI while loading User Tab with default pageSize=25. It
takes 2.92 seconds. Please refer image below
!with RANGER-2789 and RANGER-3014 .png|width=330,height=147!
*Following are analysis after reverting RANGER-2789* :
If we revert RANGER-2789 it took *319.15 milliseconds* with default
pageSize=25, which is comparatively very less. Please refer following image.
!After Reverting RANGER-2789.png|width=321,height=143!
Also, user lookup in permission tab also works smoothly without failing. Please
refer following image
!Reverted RANGER-2789- userlookup in permission tab.png|width=321,height=144!
However, *pageSize=1000 took 16 seconds* to respond but this is related to
RANGER-2789 and this performance issue can be tracked separately. For now, we
need revert this patch as it breaks current functionality as I explained.
Thanks,
- Mahesh Bandal
> fix for RANGER-2789 breaks current functionality
>
>
> Key: RANGER-3014
> URL: https://issues.apache.org/jira/browse/RANGER-3014
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: Georgi Ivanov
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Attachments:
> 0001-RANGER-3014-fix-for-RANGER-2789-breaks-current-funct.patch, After
> Reverting RANGER-2789.png, Reverted RANGER-2789- userlookup in permission
> tab.png, cpu_utilization_for_user_lookup.png, ranger-crashed while
> user-lookup in permission tab.png, with RANGER-2789 and RANGER-3014 .png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Since we upgraded to Ranger 2.1.0 in our dev env, we've noticed that user
> list page in Ranger Admin UI is not showing (or it is very very slow - in the
> order of tens of minutes).
> Looking at the commit history we found that the reason was commit
> *f45054d1b9* which was meant as a performance improvement for RANGER-2789.
> Our ranger usersync fetches users and groups from AD. Our tree is huge, here
> are some stats:
> {code:java}
> select count(*) from x_user;
> 43368
> select count(*) from x_portal_user;
> 43366
> select count(*) f
[jira] [Commented] (RANGER-3014) fix for RANGER-2789 breaks current functionality
[
https://issues.apache.org/jira/browse/RANGER-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17208740#comment-17208740
]
Mahesh Hanumant Bandal commented on RANGER-3014:
Hi [~RickyMa],
As I have mentioned earlier "*User lookup in permission tab breaks current
functionality*". I have tested with RANGER-2789 patch and also with your
[latest patch|https://reviews.apache.org/r/72926/] . Please refer to following
image !ranger-crashed while user-lookup in permission
tab.png|width=448,height=198! .
When ranger tries to lookup for users in permission tab, and if we keep
typing/deleting few letters in "Select and Add User" textbox; it sends requests
for user search on keyPressEvent from UI side. After sometime all request
starts failing and ranger stops responding. Also, I checked cpu utilization and
it reaches over 370%. Please refer to this image
!cpu_utilization_for_user_lookup.png|width=210,height=176!
Apart from this, on UI while loading User Tab with default pageSize=25. It
takes 2.92 seconds. Please refer image below
!with RANGER-2789 and RANGER-3014 .png|width=330,height=147!
*Following are analysis after reverting RANGER-2789* :
If we revert RANGER-2789 it took *319.15 milliseconds* with default
pageSize=25, which is comparatively very less. Please refer following image.
!After Reverting RANGER-2789.png|width=321,height=143!
Also, user lookup in permission tab also works smoothly without failing. Please
refer following image
!Reverted RANGER-2789- userlookup in permission tab.png|width=321,height=144!
However, *pageSize=1000 took 16 seconds* to respond but this is related to
RANGER-2789 and this performance issue can be tracked separately. For now, we
need revert this patch as it breaks current functionality as I explained.
Thanks,
- Mahesh Bandal
> fix for RANGER-2789 breaks current functionality
>
>
> Key: RANGER-3014
> URL: https://issues.apache.org/jira/browse/RANGER-3014
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: Georgi Ivanov
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Attachments:
> 0001-RANGER-3014-fix-for-RANGER-2789-breaks-current-funct.patch, After
> Reverting RANGER-2789.png, Reverted RANGER-2789- userlookup in permission
> tab.png, cpu_utilization_for_user_lookup.png, ranger-crashed while
> user-lookup in permission tab.png, with RANGER-2789 and RANGER-3014 .png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Since we upgraded to Ranger 2.1.0 in our dev env, we've noticed that user
> list page in Ranger Admin UI is not showing (or it is very very slow - in the
> order of tens of minutes).
> Looking at the commit history we found that the reason was commit
> *f45054d1b9* which was meant as a performance improvement for RANGER-2789.
> Our ranger usersync fetches users and groups from AD. Our tree is huge, here
> are some stats:
> {code:java}
> select count(*) from x_user;
> 43368
> select count(*) from x_portal_user;
> 43366
> select count(*) from x_group;
> 17865
> select count(*) from x_group_users;
> 366180 {code}
>
> Looking at the commit *f45054d1b9* what it meant to solve is perform a user
> lookup and fetching user info such as attributes and group membership in
> bulk, instead of doing it in a loop, one by one. In order to do that it
> provided couple of methods and also an override for searchXUsers in
> service/XUserService.java (before we used the parent method in
> service/XUserServiceBase.java).
>
> The new searchXUsers method (which gets invoked when we call
> /service/xusers/users REST API, calls populateViewBeans (another new method).
> It calls the parent method populateViewBeans in XUserServiceBase.java which
> build a hashmap or users and calls an override of populateViewBeans with
> input hashmap
> {code:java}
> + public List populateViewBeans(List resources) {
> + List viewBeans = new ArrayList<>();
> + if (CollectionUtils.isNotEmpty(resources)) {
> + Map resourceViewBeanMap = new
> HashMap<>(resources.size());
> + Map viewBeanResourceMap = new
> HashMap<>(resources.size());
> + for (XXUser resource : resources) {
> + VXUser viewBean = createViewObject();
> +
> viewBean.setCredStoreId(resource.getCredStoreId());
> +
> viewBean.setDescription(resource.getDescription());
> + viewBean.setName(resource.getName());
> + viewBean.setStatus(resource.getStatus());
> + resourceViewBeanMap.put(resource, viewBean);
> +
[jira] [Updated] (RANGER-3014) fix for RANGER-2789 breaks current functionality
[
https://issues.apache.org/jira/browse/RANGER-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahesh Hanumant Bandal updated RANGER-3014:
---
Attachment: After Reverting RANGER-2789.png
> fix for RANGER-2789 breaks current functionality
>
>
> Key: RANGER-3014
> URL: https://issues.apache.org/jira/browse/RANGER-3014
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: Georgi Ivanov
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Attachments:
> 0001-RANGER-3014-fix-for-RANGER-2789-breaks-current-funct.patch, After
> Reverting RANGER-2789.png, cpu_utilization_for_user_lookup.png,
> ranger-crashed while user-lookup in permission tab.png, with RANGER-2789 and
> RANGER-3014 .png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Since we upgraded to Ranger 2.1.0 in our dev env, we've noticed that user
> list page in Ranger Admin UI is not showing (or it is very very slow - in the
> order of tens of minutes).
> Looking at the commit history we found that the reason was commit
> *f45054d1b9* which was meant as a performance improvement for RANGER-2789.
> Our ranger usersync fetches users and groups from AD. Our tree is huge, here
> are some stats:
> {code:java}
> select count(*) from x_user;
> 43368
> select count(*) from x_portal_user;
> 43366
> select count(*) from x_group;
> 17865
> select count(*) from x_group_users;
> 366180 {code}
>
> Looking at the commit *f45054d1b9* what it meant to solve is perform a user
> lookup and fetching user info such as attributes and group membership in
> bulk, instead of doing it in a loop, one by one. In order to do that it
> provided couple of methods and also an override for searchXUsers in
> service/XUserService.java (before we used the parent method in
> service/XUserServiceBase.java).
>
> The new searchXUsers method (which gets invoked when we call
> /service/xusers/users REST API, calls populateViewBeans (another new method).
> It calls the parent method populateViewBeans in XUserServiceBase.java which
> build a hashmap or users and calls an override of populateViewBeans with
> input hashmap
> {code:java}
> + public List populateViewBeans(List resources) {
> + List viewBeans = new ArrayList<>();
> + if (CollectionUtils.isNotEmpty(resources)) {
> + Map resourceViewBeanMap = new
> HashMap<>(resources.size());
> + Map viewBeanResourceMap = new
> HashMap<>(resources.size());
> + for (XXUser resource : resources) {
> + VXUser viewBean = createViewObject();
> +
> viewBean.setCredStoreId(resource.getCredStoreId());
> +
> viewBean.setDescription(resource.getDescription());
> + viewBean.setName(resource.getName());
> + viewBean.setStatus(resource.getStatus());
> + resourceViewBeanMap.put(resource, viewBean);
> + viewBeanResourceMap.put(viewBean, resource);
> + viewBeans.add(viewBean);
> + }
> + populateViewBeans(resourceViewBeanMap);
> + mapEntityToViewBeans(viewBeanResourceMap);
> + }
> + return viewBeans;
> + }
> +
> + protected void populateViewBeans(Map
> resourceViewBeanMap) {
> + mapBaseAttributesToViewBeans(resourceViewBeanMap);
> + } {code}
>
> This in turns calls mapBaseAttributesToViewBeans, which calls
> daoManager.getXXPortalUser().findAllXPortalUser() and it pulls all users (no
> matter that we limit the users with a REST call to 25 by default)
> That's one thing that hampers performance. However the biggest issue is this:
> {code:java}
> + @Override
> + public List populateViewBeans(List xUsers) {
> + List vObjList = super.populateViewBeans(xUsers);
> + if (CollectionUtils.isNotEmpty(vObjList) &&
> CollectionUtils.isNotEmpty(xUsers) && xUsers.size() == vObjList.size()) {
> + Map xUserIdVObjMap = new
> HashMap<>(xUsers.size());
> + for (int i = 0; i < xUsers.size(); ++i) {
> + VXUser vObj = vObjList.get(i);
> + XXUser xUser = xUsers.get(i);
> + vObj.setIsVisible(xUser.getIsVisible());
> + xUserIdVObjMap.put(xUser.getId(), vObj);
> + }
> + populateGroupList(xUserIdVObjMap);
> + }
> + return vObjList;
> + } {code}
> We call populateGroupList
[jira] [Updated] (RANGER-3014) fix for RANGER-2789 breaks current functionality
[
https://issues.apache.org/jira/browse/RANGER-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahesh Hanumant Bandal updated RANGER-3014:
---
Attachment: Reverted RANGER-2789- userlookup in permission tab.png
> fix for RANGER-2789 breaks current functionality
>
>
> Key: RANGER-3014
> URL: https://issues.apache.org/jira/browse/RANGER-3014
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: Georgi Ivanov
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Attachments:
> 0001-RANGER-3014-fix-for-RANGER-2789-breaks-current-funct.patch, After
> Reverting RANGER-2789.png, Reverted RANGER-2789- userlookup in permission
> tab.png, cpu_utilization_for_user_lookup.png, ranger-crashed while
> user-lookup in permission tab.png, with RANGER-2789 and RANGER-3014 .png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Since we upgraded to Ranger 2.1.0 in our dev env, we've noticed that user
> list page in Ranger Admin UI is not showing (or it is very very slow - in the
> order of tens of minutes).
> Looking at the commit history we found that the reason was commit
> *f45054d1b9* which was meant as a performance improvement for RANGER-2789.
> Our ranger usersync fetches users and groups from AD. Our tree is huge, here
> are some stats:
> {code:java}
> select count(*) from x_user;
> 43368
> select count(*) from x_portal_user;
> 43366
> select count(*) from x_group;
> 17865
> select count(*) from x_group_users;
> 366180 {code}
>
> Looking at the commit *f45054d1b9* what it meant to solve is perform a user
> lookup and fetching user info such as attributes and group membership in
> bulk, instead of doing it in a loop, one by one. In order to do that it
> provided couple of methods and also an override for searchXUsers in
> service/XUserService.java (before we used the parent method in
> service/XUserServiceBase.java).
>
> The new searchXUsers method (which gets invoked when we call
> /service/xusers/users REST API, calls populateViewBeans (another new method).
> It calls the parent method populateViewBeans in XUserServiceBase.java which
> build a hashmap or users and calls an override of populateViewBeans with
> input hashmap
> {code:java}
> + public List populateViewBeans(List resources) {
> + List viewBeans = new ArrayList<>();
> + if (CollectionUtils.isNotEmpty(resources)) {
> + Map resourceViewBeanMap = new
> HashMap<>(resources.size());
> + Map viewBeanResourceMap = new
> HashMap<>(resources.size());
> + for (XXUser resource : resources) {
> + VXUser viewBean = createViewObject();
> +
> viewBean.setCredStoreId(resource.getCredStoreId());
> +
> viewBean.setDescription(resource.getDescription());
> + viewBean.setName(resource.getName());
> + viewBean.setStatus(resource.getStatus());
> + resourceViewBeanMap.put(resource, viewBean);
> + viewBeanResourceMap.put(viewBean, resource);
> + viewBeans.add(viewBean);
> + }
> + populateViewBeans(resourceViewBeanMap);
> + mapEntityToViewBeans(viewBeanResourceMap);
> + }
> + return viewBeans;
> + }
> +
> + protected void populateViewBeans(Map
> resourceViewBeanMap) {
> + mapBaseAttributesToViewBeans(resourceViewBeanMap);
> + } {code}
>
> This in turns calls mapBaseAttributesToViewBeans, which calls
> daoManager.getXXPortalUser().findAllXPortalUser() and it pulls all users (no
> matter that we limit the users with a REST call to 25 by default)
> That's one thing that hampers performance. However the biggest issue is this:
> {code:java}
> + @Override
> + public List populateViewBeans(List xUsers) {
> + List vObjList = super.populateViewBeans(xUsers);
> + if (CollectionUtils.isNotEmpty(vObjList) &&
> CollectionUtils.isNotEmpty(xUsers) && xUsers.size() == vObjList.size()) {
> + Map xUserIdVObjMap = new
> HashMap<>(xUsers.size());
> + for (int i = 0; i < xUsers.size(); ++i) {
> + VXUser vObj = vObjList.get(i);
> + XXUser xUser = xUsers.get(i);
> + vObj.setIsVisible(xUser.getIsVisible());
> + xUserIdVObjMap.put(xUser.getId(), vObj);
> + }
> + populateGroupList(xUserIdVObjMap);
> + }
>
[jira] [Resolved] (RANGER-3014) fix for RANGER-2789 breaks current functionality
[
https://issues.apache.org/jira/browse/RANGER-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahesh Hanumant Bandal resolved RANGER-3014.
Resolution: Fixed
commit id on master branch :
[https://github.com/apache/ranger/commit/166dfb6c478faba275766e5e2566031231a7decb]
commit id on ranger-2.2 branch :
[https://github.com/apache/ranger/commit/64d87c1a7eef6b30227478b340980889e07e2816]
patch is available at [Apache Review Board|https://reviews.apache.org/r/72925/]
> fix for RANGER-2789 breaks current functionality
>
>
> Key: RANGER-3014
> URL: https://issues.apache.org/jira/browse/RANGER-3014
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: Georgi Ivanov
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Attachments:
> 0001-RANGER-3014-fix-for-RANGER-2789-breaks-current-funct.patch, After
> Reverting RANGER-2789.png, Reverted RANGER-2789- userlookup in permission
> tab.png, cpu_utilization_for_user_lookup.png, ranger-crashed while
> user-lookup in permission tab.png, with RANGER-2789 and RANGER-3014 .png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Since we upgraded to Ranger 2.1.0 in our dev env, we've noticed that user
> list page in Ranger Admin UI is not showing (or it is very very slow - in the
> order of tens of minutes).
> Looking at the commit history we found that the reason was commit
> *f45054d1b9* which was meant as a performance improvement for RANGER-2789.
> Our ranger usersync fetches users and groups from AD. Our tree is huge, here
> are some stats:
> {code:java}
> select count(*) from x_user;
> 43368
> select count(*) from x_portal_user;
> 43366
> select count(*) from x_group;
> 17865
> select count(*) from x_group_users;
> 366180 {code}
>
> Looking at the commit *f45054d1b9* what it meant to solve is perform a user
> lookup and fetching user info such as attributes and group membership in
> bulk, instead of doing it in a loop, one by one. In order to do that it
> provided couple of methods and also an override for searchXUsers in
> service/XUserService.java (before we used the parent method in
> service/XUserServiceBase.java).
>
> The new searchXUsers method (which gets invoked when we call
> /service/xusers/users REST API, calls populateViewBeans (another new method).
> It calls the parent method populateViewBeans in XUserServiceBase.java which
> build a hashmap or users and calls an override of populateViewBeans with
> input hashmap
> {code:java}
> + public List populateViewBeans(List resources) {
> + List viewBeans = new ArrayList<>();
> + if (CollectionUtils.isNotEmpty(resources)) {
> + Map resourceViewBeanMap = new
> HashMap<>(resources.size());
> + Map viewBeanResourceMap = new
> HashMap<>(resources.size());
> + for (XXUser resource : resources) {
> + VXUser viewBean = createViewObject();
> +
> viewBean.setCredStoreId(resource.getCredStoreId());
> +
> viewBean.setDescription(resource.getDescription());
> + viewBean.setName(resource.getName());
> + viewBean.setStatus(resource.getStatus());
> + resourceViewBeanMap.put(resource, viewBean);
> + viewBeanResourceMap.put(viewBean, resource);
> + viewBeans.add(viewBean);
> + }
> + populateViewBeans(resourceViewBeanMap);
> + mapEntityToViewBeans(viewBeanResourceMap);
> + }
> + return viewBeans;
> + }
> +
> + protected void populateViewBeans(Map
> resourceViewBeanMap) {
> + mapBaseAttributesToViewBeans(resourceViewBeanMap);
> + } {code}
>
> This in turns calls mapBaseAttributesToViewBeans, which calls
> daoManager.getXXPortalUser().findAllXPortalUser() and it pulls all users (no
> matter that we limit the users with a REST call to 25 by default)
> That's one thing that hampers performance. However the biggest issue is this:
> {code:java}
> + @Override
> + public List populateViewBeans(List xUsers) {
> + List vObjList = super.populateViewBeans(xUsers);
> + if (CollectionUtils.isNotEmpty(vObjList) &&
> CollectionUtils.isNotEmpty(xUsers) && xUsers.size() == vObjList.size()) {
> + Map xUserIdVObjMap = new
> HashMap<>(xUsers.size());
> + for (int i = 0; i < xUsers.size(); ++i) {
> + VXUser vObj = vObjList.get(i);
> + XXUser xUser = xUse
[jira] [Updated] (RANGER-3014) fix for RANGER-2789 breaks current functionality
[
https://issues.apache.org/jira/browse/RANGER-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahesh Hanumant Bandal updated RANGER-3014:
---
Attachment: with RANGER-2789 and RANGER-3014 .png
> fix for RANGER-2789 breaks current functionality
>
>
> Key: RANGER-3014
> URL: https://issues.apache.org/jira/browse/RANGER-3014
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: Georgi Ivanov
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Attachments:
> 0001-RANGER-3014-fix-for-RANGER-2789-breaks-current-funct.patch,
> cpu_utilization_for_user_lookup.png, ranger-crashed while user-lookup in
> permission tab.png, with RANGER-2789 and RANGER-3014 .png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Since we upgraded to Ranger 2.1.0 in our dev env, we've noticed that user
> list page in Ranger Admin UI is not showing (or it is very very slow - in the
> order of tens of minutes).
> Looking at the commit history we found that the reason was commit
> *f45054d1b9* which was meant as a performance improvement for RANGER-2789.
> Our ranger usersync fetches users and groups from AD. Our tree is huge, here
> are some stats:
> {code:java}
> select count(*) from x_user;
> 43368
> select count(*) from x_portal_user;
> 43366
> select count(*) from x_group;
> 17865
> select count(*) from x_group_users;
> 366180 {code}
>
> Looking at the commit *f45054d1b9* what it meant to solve is perform a user
> lookup and fetching user info such as attributes and group membership in
> bulk, instead of doing it in a loop, one by one. In order to do that it
> provided couple of methods and also an override for searchXUsers in
> service/XUserService.java (before we used the parent method in
> service/XUserServiceBase.java).
>
> The new searchXUsers method (which gets invoked when we call
> /service/xusers/users REST API, calls populateViewBeans (another new method).
> It calls the parent method populateViewBeans in XUserServiceBase.java which
> build a hashmap or users and calls an override of populateViewBeans with
> input hashmap
> {code:java}
> + public List populateViewBeans(List resources) {
> + List viewBeans = new ArrayList<>();
> + if (CollectionUtils.isNotEmpty(resources)) {
> + Map resourceViewBeanMap = new
> HashMap<>(resources.size());
> + Map viewBeanResourceMap = new
> HashMap<>(resources.size());
> + for (XXUser resource : resources) {
> + VXUser viewBean = createViewObject();
> +
> viewBean.setCredStoreId(resource.getCredStoreId());
> +
> viewBean.setDescription(resource.getDescription());
> + viewBean.setName(resource.getName());
> + viewBean.setStatus(resource.getStatus());
> + resourceViewBeanMap.put(resource, viewBean);
> + viewBeanResourceMap.put(viewBean, resource);
> + viewBeans.add(viewBean);
> + }
> + populateViewBeans(resourceViewBeanMap);
> + mapEntityToViewBeans(viewBeanResourceMap);
> + }
> + return viewBeans;
> + }
> +
> + protected void populateViewBeans(Map
> resourceViewBeanMap) {
> + mapBaseAttributesToViewBeans(resourceViewBeanMap);
> + } {code}
>
> This in turns calls mapBaseAttributesToViewBeans, which calls
> daoManager.getXXPortalUser().findAllXPortalUser() and it pulls all users (no
> matter that we limit the users with a REST call to 25 by default)
> That's one thing that hampers performance. However the biggest issue is this:
> {code:java}
> + @Override
> + public List populateViewBeans(List xUsers) {
> + List vObjList = super.populateViewBeans(xUsers);
> + if (CollectionUtils.isNotEmpty(vObjList) &&
> CollectionUtils.isNotEmpty(xUsers) && xUsers.size() == vObjList.size()) {
> + Map xUserIdVObjMap = new
> HashMap<>(xUsers.size());
> + for (int i = 0; i < xUsers.size(); ++i) {
> + VXUser vObj = vObjList.get(i);
> + XXUser xUser = xUsers.get(i);
> + vObj.setIsVisible(xUser.getIsVisible());
> + xUserIdVObjMap.put(xUser.getId(), vObj);
> + }
> + populateGroupList(xUserIdVObjMap);
> + }
> + return vObjList;
> + } {code}
> We call populateGroupList on the list of users (by defa
[jira] [Updated] (RANGER-3014) fix for RANGER-2789 breaks current functionality
[
https://issues.apache.org/jira/browse/RANGER-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahesh Hanumant Bandal updated RANGER-3014:
---
Attachment: cpu_utilization_for_user_lookup.png
> fix for RANGER-2789 breaks current functionality
>
>
> Key: RANGER-3014
> URL: https://issues.apache.org/jira/browse/RANGER-3014
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: Georgi Ivanov
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Attachments:
> 0001-RANGER-3014-fix-for-RANGER-2789-breaks-current-funct.patch,
> cpu_utilization_for_user_lookup.png, ranger-crashed while user-lookup in
> permission tab.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Since we upgraded to Ranger 2.1.0 in our dev env, we've noticed that user
> list page in Ranger Admin UI is not showing (or it is very very slow - in the
> order of tens of minutes).
> Looking at the commit history we found that the reason was commit
> *f45054d1b9* which was meant as a performance improvement for RANGER-2789.
> Our ranger usersync fetches users and groups from AD. Our tree is huge, here
> are some stats:
> {code:java}
> select count(*) from x_user;
> 43368
> select count(*) from x_portal_user;
> 43366
> select count(*) from x_group;
> 17865
> select count(*) from x_group_users;
> 366180 {code}
>
> Looking at the commit *f45054d1b9* what it meant to solve is perform a user
> lookup and fetching user info such as attributes and group membership in
> bulk, instead of doing it in a loop, one by one. In order to do that it
> provided couple of methods and also an override for searchXUsers in
> service/XUserService.java (before we used the parent method in
> service/XUserServiceBase.java).
>
> The new searchXUsers method (which gets invoked when we call
> /service/xusers/users REST API, calls populateViewBeans (another new method).
> It calls the parent method populateViewBeans in XUserServiceBase.java which
> build a hashmap or users and calls an override of populateViewBeans with
> input hashmap
> {code:java}
> + public List populateViewBeans(List resources) {
> + List viewBeans = new ArrayList<>();
> + if (CollectionUtils.isNotEmpty(resources)) {
> + Map resourceViewBeanMap = new
> HashMap<>(resources.size());
> + Map viewBeanResourceMap = new
> HashMap<>(resources.size());
> + for (XXUser resource : resources) {
> + VXUser viewBean = createViewObject();
> +
> viewBean.setCredStoreId(resource.getCredStoreId());
> +
> viewBean.setDescription(resource.getDescription());
> + viewBean.setName(resource.getName());
> + viewBean.setStatus(resource.getStatus());
> + resourceViewBeanMap.put(resource, viewBean);
> + viewBeanResourceMap.put(viewBean, resource);
> + viewBeans.add(viewBean);
> + }
> + populateViewBeans(resourceViewBeanMap);
> + mapEntityToViewBeans(viewBeanResourceMap);
> + }
> + return viewBeans;
> + }
> +
> + protected void populateViewBeans(Map
> resourceViewBeanMap) {
> + mapBaseAttributesToViewBeans(resourceViewBeanMap);
> + } {code}
>
> This in turns calls mapBaseAttributesToViewBeans, which calls
> daoManager.getXXPortalUser().findAllXPortalUser() and it pulls all users (no
> matter that we limit the users with a REST call to 25 by default)
> That's one thing that hampers performance. However the biggest issue is this:
> {code:java}
> + @Override
> + public List populateViewBeans(List xUsers) {
> + List vObjList = super.populateViewBeans(xUsers);
> + if (CollectionUtils.isNotEmpty(vObjList) &&
> CollectionUtils.isNotEmpty(xUsers) && xUsers.size() == vObjList.size()) {
> + Map xUserIdVObjMap = new
> HashMap<>(xUsers.size());
> + for (int i = 0; i < xUsers.size(); ++i) {
> + VXUser vObj = vObjList.get(i);
> + XXUser xUser = xUsers.get(i);
> + vObj.setIsVisible(xUser.getIsVisible());
> + xUserIdVObjMap.put(xUser.getId(), vObj);
> + }
> + populateGroupList(xUserIdVObjMap);
> + }
> + return vObjList;
> + } {code}
> We call populateGroupList on the list of users (by default 25) but we call a
> new method that
[jira] [Updated] (RANGER-3014) fix for RANGER-2789 breaks current functionality
[
https://issues.apache.org/jira/browse/RANGER-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahesh Hanumant Bandal updated RANGER-3014:
---
Attachment: ranger-crashed while user-lookup in permission tab.png
> fix for RANGER-2789 breaks current functionality
>
>
> Key: RANGER-3014
> URL: https://issues.apache.org/jira/browse/RANGER-3014
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: Georgi Ivanov
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Attachments:
> 0001-RANGER-3014-fix-for-RANGER-2789-breaks-current-funct.patch,
> ranger-crashed while user-lookup in permission tab.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Since we upgraded to Ranger 2.1.0 in our dev env, we've noticed that user
> list page in Ranger Admin UI is not showing (or it is very very slow - in the
> order of tens of minutes).
> Looking at the commit history we found that the reason was commit
> *f45054d1b9* which was meant as a performance improvement for RANGER-2789.
> Our ranger usersync fetches users and groups from AD. Our tree is huge, here
> are some stats:
> {code:java}
> select count(*) from x_user;
> 43368
> select count(*) from x_portal_user;
> 43366
> select count(*) from x_group;
> 17865
> select count(*) from x_group_users;
> 366180 {code}
>
> Looking at the commit *f45054d1b9* what it meant to solve is perform a user
> lookup and fetching user info such as attributes and group membership in
> bulk, instead of doing it in a loop, one by one. In order to do that it
> provided couple of methods and also an override for searchXUsers in
> service/XUserService.java (before we used the parent method in
> service/XUserServiceBase.java).
>
> The new searchXUsers method (which gets invoked when we call
> /service/xusers/users REST API, calls populateViewBeans (another new method).
> It calls the parent method populateViewBeans in XUserServiceBase.java which
> build a hashmap or users and calls an override of populateViewBeans with
> input hashmap
> {code:java}
> + public List populateViewBeans(List resources) {
> + List viewBeans = new ArrayList<>();
> + if (CollectionUtils.isNotEmpty(resources)) {
> + Map resourceViewBeanMap = new
> HashMap<>(resources.size());
> + Map viewBeanResourceMap = new
> HashMap<>(resources.size());
> + for (XXUser resource : resources) {
> + VXUser viewBean = createViewObject();
> +
> viewBean.setCredStoreId(resource.getCredStoreId());
> +
> viewBean.setDescription(resource.getDescription());
> + viewBean.setName(resource.getName());
> + viewBean.setStatus(resource.getStatus());
> + resourceViewBeanMap.put(resource, viewBean);
> + viewBeanResourceMap.put(viewBean, resource);
> + viewBeans.add(viewBean);
> + }
> + populateViewBeans(resourceViewBeanMap);
> + mapEntityToViewBeans(viewBeanResourceMap);
> + }
> + return viewBeans;
> + }
> +
> + protected void populateViewBeans(Map
> resourceViewBeanMap) {
> + mapBaseAttributesToViewBeans(resourceViewBeanMap);
> + } {code}
>
> This in turns calls mapBaseAttributesToViewBeans, which calls
> daoManager.getXXPortalUser().findAllXPortalUser() and it pulls all users (no
> matter that we limit the users with a REST call to 25 by default)
> That's one thing that hampers performance. However the biggest issue is this:
> {code:java}
> + @Override
> + public List populateViewBeans(List xUsers) {
> + List vObjList = super.populateViewBeans(xUsers);
> + if (CollectionUtils.isNotEmpty(vObjList) &&
> CollectionUtils.isNotEmpty(xUsers) && xUsers.size() == vObjList.size()) {
> + Map xUserIdVObjMap = new
> HashMap<>(xUsers.size());
> + for (int i = 0; i < xUsers.size(); ++i) {
> + VXUser vObj = vObjList.get(i);
> + XXUser xUser = xUsers.get(i);
> + vObj.setIsVisible(xUser.getIsVisible());
> + xUserIdVObjMap.put(xUser.getId(), vObj);
> + }
> + populateGroupList(xUserIdVObjMap);
> + }
> + return vObjList;
> + } {code}
> We call populateGroupList on the list of users (by default 25) but we call a
> new method that accepts a map as an i
[jira] [Updated] (RANGER-3022) Upgrade Spring framework to version 4.3.29.RELEASE
[ https://issues.apache.org/jira/browse/RANGER-3022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal updated RANGER-3022: --- Fix Version/s: 2.1.1 > Upgrade Spring framework to version 4.3.29.RELEASE > -- > > Key: RANGER-3022 > URL: https://issues.apache.org/jira/browse/RANGER-3022 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.1 >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Minor > Fix For: 2.1.1 > > > We need to upgrade springframework versions part of security improvement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-3022) Upgrade Spring framework to version 4.3.29.RELEASE
[ https://issues.apache.org/jira/browse/RANGER-3022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal updated RANGER-3022: --- Affects Version/s: 2.1.1 > Upgrade Spring framework to version 4.3.29.RELEASE > -- > > Key: RANGER-3022 > URL: https://issues.apache.org/jira/browse/RANGER-3022 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.1 >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Minor > > We need to upgrade springframework versions part of security improvement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3022) Upgrade Spring framework to version 4.3.29.RELEASE
Mahesh Hanumant Bandal created RANGER-3022: -- Summary: Upgrade Spring framework to version 4.3.29.RELEASE Key: RANGER-3022 URL: https://issues.apache.org/jira/browse/RANGER-3022 Project: Ranger Issue Type: Improvement Components: Ranger Reporter: Mahesh Hanumant Bandal Assignee: Mahesh Hanumant Bandal We need to upgrade springframework versions part of security improvement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-3014) fix for RANGER-2789 breaks current functionality
[
https://issues.apache.org/jira/browse/RANGER-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17208128#comment-17208128
]
Mahesh Hanumant Bandal commented on RANGER-3014:
Hi [~RickyMa],
I have reviewed your latest patch. After testing it, I found that issue still
exists.
> fix for RANGER-2789 breaks current functionality
>
>
> Key: RANGER-3014
> URL: https://issues.apache.org/jira/browse/RANGER-3014
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: Georgi Ivanov
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
> Attachments:
> 0001-RANGER-3014-fix-for-RANGER-2789-breaks-current-funct.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Since we upgraded to Ranger 2.1.0 in our dev env, we've noticed that user
> list page in Ranger Admin UI is not showing (or it is very very slow - in the
> order of tens of minutes).
> Looking at the commit history we found that the reason was commit
> *f45054d1b9* which was meant as a performance improvement for RANGER-2789.
> Our ranger usersync fetches users and groups from AD. Our tree is huge, here
> are some stats:
> {code:java}
> select count(*) from x_user;
> 43368
> select count(*) from x_portal_user;
> 43366
> select count(*) from x_group;
> 17865
> select count(*) from x_group_users;
> 366180 {code}
>
> Looking at the commit *f45054d1b9* what it meant to solve is perform a user
> lookup and fetching user info such as attributes and group membership in
> bulk, instead of doing it in a loop, one by one. In order to do that it
> provided couple of methods and also an override for searchXUsers in
> service/XUserService.java (before we used the parent method in
> service/XUserServiceBase.java).
>
> The new searchXUsers method (which gets invoked when we call
> /service/xusers/users REST API, calls populateViewBeans (another new method).
> It calls the parent method populateViewBeans in XUserServiceBase.java which
> build a hashmap or users and calls an override of populateViewBeans with
> input hashmap
> {code:java}
> + public List populateViewBeans(List resources) {
> + List viewBeans = new ArrayList<>();
> + if (CollectionUtils.isNotEmpty(resources)) {
> + Map resourceViewBeanMap = new
> HashMap<>(resources.size());
> + Map viewBeanResourceMap = new
> HashMap<>(resources.size());
> + for (XXUser resource : resources) {
> + VXUser viewBean = createViewObject();
> +
> viewBean.setCredStoreId(resource.getCredStoreId());
> +
> viewBean.setDescription(resource.getDescription());
> + viewBean.setName(resource.getName());
> + viewBean.setStatus(resource.getStatus());
> + resourceViewBeanMap.put(resource, viewBean);
> + viewBeanResourceMap.put(viewBean, resource);
> + viewBeans.add(viewBean);
> + }
> + populateViewBeans(resourceViewBeanMap);
> + mapEntityToViewBeans(viewBeanResourceMap);
> + }
> + return viewBeans;
> + }
> +
> + protected void populateViewBeans(Map
> resourceViewBeanMap) {
> + mapBaseAttributesToViewBeans(resourceViewBeanMap);
> + } {code}
>
> This in turns calls mapBaseAttributesToViewBeans, which calls
> daoManager.getXXPortalUser().findAllXPortalUser() and it pulls all users (no
> matter that we limit the users with a REST call to 25 by default)
> That's one thing that hampers performance. However the biggest issue is this:
> {code:java}
> + @Override
> + public List populateViewBeans(List xUsers) {
> + List vObjList = super.populateViewBeans(xUsers);
> + if (CollectionUtils.isNotEmpty(vObjList) &&
> CollectionUtils.isNotEmpty(xUsers) && xUsers.size() == vObjList.size()) {
> + Map xUserIdVObjMap = new
> HashMap<>(xUsers.size());
> + for (int i = 0; i < xUsers.size(); ++i) {
> + VXUser vObj = vObjList.get(i);
> + XXUser xUser = xUsers.get(i);
> + vObj.setIsVisible(xUser.getIsVisible());
> + xUserIdVObjMap.put(xUser.getId(), vObj);
> + }
> + populateGroupList(xUserIdVObjMap);
> + }
> + return vObjList;
> + } {code}
> We call populateGroupList on the list of users (by default 25) but we call a
> new me
[jira] [Commented] (RANGER-3014) fix for RANGER-2789 breaks current functionality
[
https://issues.apache.org/jira/browse/RANGER-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17206530#comment-17206530
]
Mahesh Hanumant Bandal commented on RANGER-3014:
Hi [~mollonado],
Thank you for your detailed analysis on RANGER-2789. I have also done testing
on this patch with following user and groups data
{code:java}
select count(*) from x_user;
50052
select count(*) from x_group_users;
500056
{code}
Following are issues caused with above users and group-users mapping data:
* It took significantly longer time to load users tab on RANGER UI with patch
of RANGER-2789, but Ranger responded within a minute with default pageSize=25.
* User lookup in permission tab breaks current functionality. In this case
ranger service crashes after couple of minutes when in edit permission tab of
User/Group module, multiple requests will be sent on keyPressEvent for user
search. With this patch ranger is pulling all users from the DB regardless of
pageLimit. It traverse over all users and also traversing the whole group_users
mapping which hampers performance.
* Permission tab takes longer time to load UI.
However, without patch of RANGER-2789 ranger works smoothly. Users tab on
RANGER UI loads in milliseconds with default pageSize=25. I also tried with
pageSize=1000 and it took 22 seconds to load UI. User lookup in permission tab
also responded quickly.
As RANGER-2789 breaks current functionality, I will create a request to revert
this patch. Please refer to this [Apache Review
Request|https://reviews.apache.org/r/72925/].
CC : [~RickyMa], [~pradeep]
> fix for RANGER-2789 breaks current functionality
>
>
> Key: RANGER-3014
> URL: https://issues.apache.org/jira/browse/RANGER-3014
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: Georgi Ivanov
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
>
> Since we upgraded to Ranger 2.1.0 in our dev env, we've noticed that user
> list page in Ranger Admin UI is not showing (or it is very very slow - in the
> order of tens of minutes).
> Looking at the commit history we found that the reason was commit
> *f45054d1b9* which was meant as a performance improvement for RANGER-2789.
> Our ranger usersync fetches users and groups from AD. Our tree is huge, here
> are some stats:
> {code:java}
> select count(*) from x_user;
> 43368
> select count(*) from x_portal_user;
> 43366
> select count(*) from x_group;
> 17865
> select count(*) from x_group_users;
> 366180 {code}
>
> Looking at the commit *f45054d1b9* what it meant to solve is perform a user
> lookup and fetching user info such as attributes and group membership in
> bulk, instead of doing it in a loop, one by one. In order to do that it
> provided couple of methods and also an override for searchXUsers in
> service/XUserService.java (before we used the parent method in
> service/XUserServiceBase.java).
>
> The new searchXUsers method (which gets invoked when we call
> /service/xusers/users REST API, calls populateViewBeans (another new method).
> It calls the parent method populateViewBeans in XUserServiceBase.java which
> build a hashmap or users and calls an override of populateViewBeans with
> input hashmap
> {code:java}
> + public List populateViewBeans(List resources) {
> + List viewBeans = new ArrayList<>();
> + if (CollectionUtils.isNotEmpty(resources)) {
> + Map resourceViewBeanMap = new
> HashMap<>(resources.size());
> + Map viewBeanResourceMap = new
> HashMap<>(resources.size());
> + for (XXUser resource : resources) {
> + VXUser viewBean = createViewObject();
> +
> viewBean.setCredStoreId(resource.getCredStoreId());
> +
> viewBean.setDescription(resource.getDescription());
> + viewBean.setName(resource.getName());
> + viewBean.setStatus(resource.getStatus());
> + resourceViewBeanMap.put(resource, viewBean);
> + viewBeanResourceMap.put(viewBean, resource);
> + viewBeans.add(viewBean);
> + }
> + populateViewBeans(resourceViewBeanMap);
> + mapEntityToViewBeans(viewBeanResourceMap);
> + }
> + return viewBeans;
> + }
> +
> + protected void populateViewBeans(Map
> resourceViewBeanMap) {
> + mapBaseAttributesToViewBeans(resourceViewBeanMap);
> + } {code}
>
> This in turns calls mapBaseAttributesToViewBeans, which calls
> daoManager.getXXPortalUser().f
[jira] [Assigned] (RANGER-3014) fix for RANGER-2789 breaks current functionality
[
https://issues.apache.org/jira/browse/RANGER-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahesh Hanumant Bandal reassigned RANGER-3014:
--
Assignee: Mahesh Hanumant Bandal
> fix for RANGER-2789 breaks current functionality
>
>
> Key: RANGER-3014
> URL: https://issues.apache.org/jira/browse/RANGER-3014
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: Georgi Ivanov
>Assignee: Mahesh Hanumant Bandal
>Priority: Major
>
> Since we upgraded to Ranger 2.1.0 in our dev env, we've noticed that user
> list page in Ranger Admin UI is not showing (or it is very very slow - in the
> order of tens of minutes).
> Looking at the commit history we found that the reason was commit
> *f45054d1b9* which was meant as a performance improvement for RANGER-2789.
> Our ranger usersync fetches users and groups from AD. Our tree is huge, here
> are some stats:
> {code:java}
> select count(*) from x_user;
> 43368
> select count(*) from x_portal_user;
> 43366
> select count(*) from x_group;
> 17865
> select count(*) from x_group_users;
> 366180 {code}
>
> Looking at the commit *f45054d1b9* what it meant to solve is perform a user
> lookup and fetching user info such as attributes and group membership in
> bulk, instead of doing it in a loop, one by one. In order to do that it
> provided couple of methods and also an override for searchXUsers in
> service/XUserService.java (before we used the parent method in
> service/XUserServiceBase.java).
>
> The new searchXUsers method (which gets invoked when we call
> /service/xusers/users REST API, calls populateViewBeans (another new method).
> It calls the parent method populateViewBeans in XUserServiceBase.java which
> build a hashmap or users and calls an override of populateViewBeans with
> input hashmap
> {code:java}
> + public List populateViewBeans(List resources) {
> + List viewBeans = new ArrayList<>();
> + if (CollectionUtils.isNotEmpty(resources)) {
> + Map resourceViewBeanMap = new
> HashMap<>(resources.size());
> + Map viewBeanResourceMap = new
> HashMap<>(resources.size());
> + for (XXUser resource : resources) {
> + VXUser viewBean = createViewObject();
> +
> viewBean.setCredStoreId(resource.getCredStoreId());
> +
> viewBean.setDescription(resource.getDescription());
> + viewBean.setName(resource.getName());
> + viewBean.setStatus(resource.getStatus());
> + resourceViewBeanMap.put(resource, viewBean);
> + viewBeanResourceMap.put(viewBean, resource);
> + viewBeans.add(viewBean);
> + }
> + populateViewBeans(resourceViewBeanMap);
> + mapEntityToViewBeans(viewBeanResourceMap);
> + }
> + return viewBeans;
> + }
> +
> + protected void populateViewBeans(Map
> resourceViewBeanMap) {
> + mapBaseAttributesToViewBeans(resourceViewBeanMap);
> + } {code}
>
> This in turns calls mapBaseAttributesToViewBeans, which calls
> daoManager.getXXPortalUser().findAllXPortalUser() and it pulls all users (no
> matter that we limit the users with a REST call to 25 by default)
> That's one thing that hampers performance. However the biggest issue is this:
> {code:java}
> + @Override
> + public List populateViewBeans(List xUsers) {
> + List vObjList = super.populateViewBeans(xUsers);
> + if (CollectionUtils.isNotEmpty(vObjList) &&
> CollectionUtils.isNotEmpty(xUsers) && xUsers.size() == vObjList.size()) {
> + Map xUserIdVObjMap = new
> HashMap<>(xUsers.size());
> + for (int i = 0; i < xUsers.size(); ++i) {
> + VXUser vObj = vObjList.get(i);
> + XXUser xUser = xUsers.get(i);
> + vObj.setIsVisible(xUser.getIsVisible());
> + xUserIdVObjMap.put(xUser.getId(), vObj);
> + }
> + populateGroupList(xUserIdVObjMap);
> + }
> + return vObjList;
> + } {code}
> We call populateGroupList on the list of users (by default 25) but we call a
> new method that accepts a map as an input. Inside that method we call
> {code:java}
> List allXXGroupUsers = daoManager.getXXGroupUser().getAll();
> {code}
> Which in our case will pull all 366180 group to user membership mappings from
> x_group_users table.
> Next
[jira] [Updated] (RANGER-2966) Upgrade POI to 4.1.2
[ https://issues.apache.org/jira/browse/RANGER-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal updated RANGER-2966: --- Attachment: RANGER-2966-V1.patch > Upgrade POI to 4.1.2 > > > Key: RANGER-2966 > URL: https://issues.apache.org/jira/browse/RANGER-2966 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Minor > Fix For: 2.1.0 > > Attachments: RANGER-2966-V1.patch > > > Upgrade POI version to follow best practices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-2966) Upgrade POI to 4.1.2
Mahesh Hanumant Bandal created RANGER-2966: -- Summary: Upgrade POI to 4.1.2 Key: RANGER-2966 URL: https://issues.apache.org/jira/browse/RANGER-2966 Project: Ranger Issue Type: Improvement Components: Ranger Affects Versions: 2.1.0 Reporter: Mahesh Hanumant Bandal Assignee: Mahesh Hanumant Bandal Fix For: 2.1.0 Upgrade POI version to follow best practices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (RANGER-2949) Upgrade Tomcat to 7.0.105
[ https://issues.apache.org/jira/browse/RANGER-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal reassigned RANGER-2949: -- Assignee: Mahesh Hanumant Bandal Patch is available at [Apache Review Board|https://reviews.apache.org/r/72767/] > Upgrade Tomcat to 7.0.105 > - > > Key: RANGER-2949 > URL: https://issues.apache.org/jira/browse/RANGER-2949 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Minor > Fix For: 2.1.0 > > > Upgrade Tomcat version to follow best practices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-2949) Upgrade Tomcat to 7.0.105
Mahesh Hanumant Bandal created RANGER-2949: -- Summary: Upgrade Tomcat to 7.0.105 Key: RANGER-2949 URL: https://issues.apache.org/jira/browse/RANGER-2949 Project: Ranger Issue Type: Improvement Components: Ranger Affects Versions: 2.1.0 Reporter: Mahesh Hanumant Bandal Fix For: 2.1.0 Upgrade Tomcat version to follow best practices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (RANGER-2923) Changing data type of sync_source_info column to accommodate more characters
[ https://issues.apache.org/jira/browse/RANGER-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal resolved RANGER-2923. Fix Version/s: 2.1.0 Resolution: Fixed commit id : [https://github.com/apache/ranger/commit/5c5c297a9c66f1a41a8dddf1c0148690b1d1438e] > Changing data type of sync_source_info column to accommodate more characters > > > Key: RANGER-2923 > URL: https://issues.apache.org/jira/browse/RANGER-2923 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Major > Fix For: 2.1.0 > > > Insert values for the column "sync_source_info" under table > "x_ugsync_audit_info" can go beyond the maximum length of column. (currently > limit is VARCHAR 4000). > Now changing data type to different db flavours as follows : > mysql : VARCHAR => MEDIUMTEXT > oracle : VARCHAR => CLOB > postgres : VARCHAR => TEXT > sqlanywhere : VARCHAR => TEXT > sqlserver : VARCHAR => nvarchar(max) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (RANGER-2935) Upgrade joda-time to 2.10.6
[ https://issues.apache.org/jira/browse/RANGER-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahesh Hanumant Bandal resolved RANGER-2935. Resolution: Fixed commit id : [https://github.com/apache/ranger/commit/b6b53a9c1851825e19d70e92a1d5e63e86163f31] > Upgrade joda-time to 2.10.6 > --- > > Key: RANGER-2935 > URL: https://issues.apache.org/jira/browse/RANGER-2935 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.0.0 >Reporter: Mahesh Hanumant Bandal >Assignee: Mahesh Hanumant Bandal >Priority: Major > Fix For: 2.1.0 > > > Upgrade joda-time version to follow best practices. -- This message was sent by Atlassian Jira (v8.3.4#803005)
