----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69072/ -----------------------------------------------------------
(Updated Oct. 18, 2018, 10:01 p.m.) Review request for ranger, Madhan Neethiraj, Ramesh Mani, and Velmurugan Periasamy. Changes ------- Addressed review comments on a related patch for ranger-0.7 (https://reviews.apache.org/r/68881/) Bugs: RANGER-2203 and RANGER-2219 https://issues.apache.org/jira/browse/RANGER-2203 https://issues.apache.org/jira/browse/RANGER-2219 Repository: ranger Description ------- Currently, ranger policies are fully normalized and stored in a multiple Relational database tables. There is a performance overhead incurred when retrieving a ranger policy, as multiple database accesses are required to fully reconstruct it. This is significant when there are large ranger policies (that is, the number of resources addressed by the policy is large), and/or when there is a large number of ranger policies in an installation. This Jira tracks alternate design of database schema, where a policy is stored in a de-normalized way, in its entirely, in one database table (preferably as a JSON string). Currently, tag-definitions, tags and service-resources are stored in database using a normalized form. When constructing resource->tag mappings, this schema design may lead to a large number of database accesses, thereby causing a major performance bottleneck when the number of resource->tag associations is large. Denormalized schema will reduce the number of database accesses, and improve overall performance significantly. Diffs (updated) ----- agents-common/pom.xml 09a32fa8b agents-common/src/main/java/org/apache/ranger/authorization/utils/JsonUtils.java bc4a8b523 agents-common/src/main/java/org/apache/ranger/plugin/model/RangerServiceDef.java adafb9985 agents-common/src/main/java/org/apache/ranger/plugin/store/TagStore.java 5918b1292 embeddedwebserver/scripts/ranger-admin-services.sh b8ca6c7d5 security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 32cf6db3b security-admin/db/mysql/patches/035-update-schema-for-x-policy.sql PRE-CREATION security-admin/db/mysql/patches/036-denormalize-tag-tables.sql PRE-CREATION security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql 2e577f3bd security-admin/db/oracle/patches/035-update-schema-for-x-policy.sql PRE-CREATION security-admin/db/oracle/patches/036-denormalize-tag-tables.sql PRE-CREATION security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql bad32efbc security-admin/db/postgres/patches/035-update-schema-for-x-policy.sql PRE-CREATION security-admin/db/postgres/patches/036-denormalize-tag-tables.sql PRE-CREATION security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql 9482992e7 security-admin/db/sqlanywhere/patches/035-update-schema-for-x-policy.sql PRE-CREATION security-admin/db/sqlanywhere/patches/036-denormalize-tag-tables.sql PRE-CREATION security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 85f328556 security-admin/db/sqlserver/patches/035-update-schema-for-x-policy.sql PRE-CREATION security-admin/db/sqlserver/patches/036-denormalize-tag-tables.sql PRE-CREATION security-admin/scripts/db_setup.py 02701c726 security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java c26f0a576 security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java 7875bc2a8 security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 6c699cad9 security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java d29df930c security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java 0d5689a03 security-admin/src/main/java/org/apache/ranger/db/RangerDaoManagerBase.java 5dffc0ef2 security-admin/src/main/java/org/apache/ranger/db/XXAccessTypeDefDao.java e04280bc8 security-admin/src/main/java/org/apache/ranger/db/XXDataMaskTypeDefDao.java 3dd4376e4 security-admin/src/main/java/org/apache/ranger/db/XXGroupDao.java 9a87b4cfd security-admin/src/main/java/org/apache/ranger/db/XXPolicyConditionDefDao.java a12140aba security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemAccessDao.java 9be38decd security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemConditionDao.java 9b11545a0 security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemDataMaskInfoDao.java 67c7e9961 security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemGroupPermDao.java a6fd8c65b security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemRowFilterInfoDao.java 01a36a50d security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemUserPermDao.java 667265485 security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceDao.java a7157deb4 security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceMapDao.java 55d8c50e4 security-admin/src/main/java/org/apache/ranger/db/XXResourceDefDao.java b2e311fef security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceDao.java ee0e40043 security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementDao.java c9a1c2132 security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementValueDao.java 364af6d0f security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDao.java 40c3a887b security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDefDao.java 129f3c13c security-admin/src/main/java/org/apache/ranger/db/XXTagDao.java e14f836e7 security-admin/src/main/java/org/apache/ranger/db/XXTagDefDao.java c0dd88371 security-admin/src/main/java/org/apache/ranger/db/XXTagResourceMapDao.java f9e041a3f security-admin/src/main/java/org/apache/ranger/db/XXUserDao.java 58bf4d891 security-admin/src/main/java/org/apache/ranger/entity/XXDBBase.java 8405eb3e4 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyBase.java 584a103ff security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/entity/XXServiceResource.java 961627a3c security-admin/src/main/java/org/apache/ranger/entity/XXTag.java d26a0b079 security-admin/src/main/java/org/apache/ranger/entity/XXTagDef.java 818908ba8 security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingTagsJson_J10020.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 97b8d7192 security-admin/src/main/java/org/apache/ranger/service/RangerAuditFields.java e6a519a17 security-admin/src/main/java/org/apache/ranger/service/RangerPolicyServiceBase.java 8d42165f7 security-admin/src/main/java/org/apache/ranger/service/RangerServiceDefServiceBase.java 5cbe47ad3 security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceService.java d7256802a security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceServiceBase.java 6af682a81 security-admin/src/main/java/org/apache/ranger/service/RangerTagDefService.java 82eb252e6 security-admin/src/main/java/org/apache/ranger/service/RangerTagDefServiceBase.java 408358c48 security-admin/src/main/java/org/apache/ranger/service/RangerTagService.java 28b9115fa security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java c96878d21 security-admin/src/main/resources/META-INF/jpa_named_queries.xml cdf6ba655 security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java c9db90a0e security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java 15344ea30 security-admin/src/test/java/org/apache/ranger/service/TestRangerTagDefService.java 8f19ffd78 security-admin/src/test/java/org/apache/ranger/service/TestRangerTagService.java 1175989e2 Diff: https://reviews.apache.org/r/69072/diff/2/ Changes: https://reviews.apache.org/r/69072/diff/1-2/ Testing ------- Tested with local VM for migration, policy CRUD, policy browsing and policy enforcement in Hive plugin. Thanks, Abhay Kulkarni
