[ https://issues.apache.org/jira/browse/RANGER-3899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648661#comment-17648661 ]
Ramachandran commented on RANGER-3899: -------------------------------------- [~mad...@apache.org] [~abhayk] [~vel] [~spolavarapu] [~pradeep] Hi All, could you Please review it.? > Policy creation takes more time when there are more users,groups,roles > ---------------------------------------------------------------------- > > Key: RANGER-3899 > URL: https://issues.apache.org/jira/browse/RANGER-3899 > Project: Ranger > Issue Type: Improvement > Components: Ranger > Affects Versions: 3.0.0 > Reporter: Ramachandran > Assignee: Ramachandran > Priority: Major > Fix For: 3.0.0 > > Attachments: > 0001-RANGER-3899-Policy-creation-takes-more-time-when-the.patch > > > *{color:#0000ff}Policy Creation Steps in Apache > Ranger:{color}*{color:#0000ff} {color} > > {code:java} > 1. Get the service by Name: > RangerService service = getServiceByName(policy.getService());→ 1DB Read call > 2. Get XXServiceDef by Name > XXServiceDef xServiceDef = > daoMgr.getXXServiceDef().findByName(service.getType()); → 1DB Read call > 3. Get the existing XXPolicy by Name > XXPolicy existing = > daoMgr.getXXPolicy().findByNameAndServiceIdAndZoneId(policy.getName(), > service.getId(), zoneId); → 1DB Read call > 4. Create a policy > policy = policyService.create(policy, true); → 1DB Write call > 5. Get policy by Id > XXPolicy xCreatedPolicy = daoMgr.getXXPolicy().getById(policy.getId()); → 1DB > Read call > 6. createObjectDataHistory for the newly created policy > dataHistService.createObjectDataHistory(createdPolicy, > RangerDataHistService.ACTION_CREATE); → 1DB Write call > 7. createTrxLog for the newly created policy > bizUtil.createTrxLog(trxLogList); → 1DB Write call > 8. Create Policy Label for the newly created policy > createOrMapLabels(xCreatedPolicy, uniquePolicyLabels); → 1DB Write call > 9. Create Policy Mapping For Ref Table > policyRefUpdater.createNewPolMappingForRefTable(policy, xCreatedPolicy, > xServiceDef); > 1. Getting > roleNames,groupNames,userNames,accessTypes,dataMaskTypes,conditionTypes from > each policy items of the policy > 2. Getting resourceNames from the policy > 3.For each resource > Getting the XXResourceDef by resource and policy.getId() > XXResourceDef xResDef = > daoMgr.getXXResourceDef().findByNameAndPolicyId(resource, policy.getId()); → > 1DB Read call for each resource > populate XXPolicyRefResource > 4. Create the batch of XXPolicyRefResource in DB > daoMgr.getXXPolicyRefResource().batchCreate(xPolResources); → 1DB Batch > Write call > 5.For each roleNames > Getting XXRole by roleName > XXRole xRole = daoMgr.getXXRole().findByRoleName(role); → 1DB Read call > for each role > populate XXPolicyRefRole > 6. Create the batch of XXPolicyRefRole in DB > daoMgr.getXXPolicyRefResource().batchCreate(xPolRoles); → 1DB Batch Write > call > 7.For each groupNames > Getting the XXGroup by groupName > XXGroup xGroup = > daoMgr.getXXGroup().findByGroupName(context.group.getName());→ 1DB Read call > for each group > populate XXPolicyRefGroup > Insert into DB > daoMgr.getXXPolicyRefGroup().create(xPolGroup);→ 1DB write call for each group > 8.For each userNames > Getting XXUser by userName > XXUser xUser = daoMgr.getXXUser().findByUserName(user); → 1DB Read call for > each user > populate XXPolicyRefUser > daoMgr.getXXPolicyRefUser().create(xPolUser); → 1DB write call for each user > 9.For each accessTypes > Getting the XXAccessTypeDef by accessType and xPolicy.getService() > XXAccessTypeDef xAccTypeDef = > daoMgr.getXXAccessTypeDef().findByNameAndServiceId(accessType, > xPolicy.getService()); → 1DB Read call for each accesType > populate XXAccessTypeDef > 10. Create the batch of xPolAccesses in DB > daoMgr.getXXPolicyRefAccessType().batchCreate(xPolAccesses); → 1DB Batch > Write call > 11.For each conditionTypes > Getting the XXPolicyConditionDef by condition and xServiceDef.getId() > XXPolicyConditionDef xPolCondDef = > daoMgr.getXXPolicyConditionDef().findByServiceDefIdAndName(xServiceDef.getId(), > condition); → 1DB Read call for each conditionType > populate XXPolicyConditionDef > 12. Create the batch of xPolConds in DB > daoMgr.getXXPolicyRefCondition().batchCreate(xPolConds); → 1DB Batch Write > call > {code} > > *{color:#0000ff}A total number of DB calls involved for the below Policy > creation in Apache Ranger(Before Fix):{color}* > > The Policy contains 500 users,5 access Types (permissions),12 resources > (1DB,1Table,10 columns) > > ||DB Write count||DB Read count||DB Batch Write count||Time is taken to > create the policy|| > |504|526|3|{color:#ff0000}4~8 seconds{color}| > *{color:#0000ff}A total number of DB calls involved for the below Policy > creation in Apache Ranger(After Fix):{color}* > > The policy contains 500 users,5 access Types (permissions),12 resources > (1DB,1Table,10 columns) > {color:#ff0000}Proposal :{color} > {color:#ff0000}After changing to batch create of XXPolicyRefUser instead of > calling 1 DB write XXPolicyRefUser for every user of the policy:{color} > > ||DB Write count||DB Read count||DB Batch Write count||Time is taken to > create the policy|| > |6|526|5|{color:#ff0000}2.6~4 seconds{color}| > A similar will be applied for roles and groups as well. > > After fixing (RANGER-3899, RANGER-2732,) > When Policy contains 500 users,5 access Types (permissions),12 resources > (1DB,1Table,10 columns) > After the Batch read the calls will be reduced into > ||DB Write count||DB Read count||DB Batch Write count||Time is taken to > create the policy|| > |6|27|5|<2.6 seconds (needs to be validated)| > DB read calls will be reduced to {color:#de350b}27{color} from > {color:#403294}526{color} > {color:#403294}Fixing this part will reduce the DB read calls to a very > minimum which will improve performance drastically.{color} > > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)