----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69072/#review209758 -----------------------------------------------------------
Ship it! Ship It! - Madhan Neethiraj On Oct. 18, 2018, 7:27 p.m., Abhay Kulkarni wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/69072/ > ----------------------------------------------------------- > > (Updated Oct. 18, 2018, 7:27 p.m.) > > > Review request for ranger, Madhan Neethiraj, Ramesh Mani, and Velmurugan > Periasamy. > > > Bugs: RANGER-2203 and RANGER-2219 > https://issues.apache.org/jira/browse/RANGER-2203 > https://issues.apache.org/jira/browse/RANGER-2219 > > > Repository: ranger > > > Description > ------- > > Currently, ranger policies are fully normalized and stored in a multiple > Relational database tables. There is a performance overhead incurred when > retrieving a ranger policy, as multiple database accesses are required to > fully reconstruct it. This is significant when there are large ranger > policies (that is, the number of resources addressed by the policy is large), > and/or when there is a large number of ranger policies in an installation. > > This Jira tracks alternate design of database schema, where a policy is > stored in a de-normalized way, in its entirely, in one database table > (preferably as a JSON string). > > Currently, tag-definitions, tags and service-resources are stored in database > using a normalized form. When constructing resource->tag mappings, this > schema design may lead to a large number of database accesses, thereby > causing a major performance bottleneck when the number of resource->tag > associations is large. > > Denormalized schema will reduce the number of database accesses, and improve > overall performance significantly. > > > Diffs > ----- > > agents-common/pom.xml 09a32fa8b > > agents-common/src/main/java/org/apache/ranger/authorization/utils/JsonUtils.java > bc4a8b523 > > agents-common/src/main/java/org/apache/ranger/plugin/model/RangerServiceDef.java > adafb9985 > agents-common/src/main/java/org/apache/ranger/plugin/store/TagStore.java > 5918b1292 > embeddedwebserver/scripts/ranger-admin-services.sh b8ca6c7d5 > security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql > 32cf6db3b > security-admin/db/mysql/patches/035-update-schema-for-x-policy.sql > PRE-CREATION > security-admin/db/mysql/patches/036-denormalize-tag-tables.sql PRE-CREATION > security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql > 2e577f3bd > security-admin/db/oracle/patches/035-update-schema-for-x-policy.sql > PRE-CREATION > security-admin/db/oracle/patches/036-denormalize-tag-tables.sql > PRE-CREATION > security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql > bad32efbc > security-admin/db/postgres/patches/035-update-schema-for-x-policy.sql > PRE-CREATION > security-admin/db/postgres/patches/036-denormalize-tag-tables.sql > PRE-CREATION > > security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql > 9482992e7 > security-admin/db/sqlanywhere/patches/035-update-schema-for-x-policy.sql > PRE-CREATION > security-admin/db/sqlanywhere/patches/036-denormalize-tag-tables.sql > PRE-CREATION > security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql > 85f328556 > security-admin/db/sqlserver/patches/035-update-schema-for-x-policy.sql > PRE-CREATION > security-admin/db/sqlserver/patches/036-denormalize-tag-tables.sql > PRE-CREATION > security-admin/scripts/db_setup.py 02701c726 > security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java > PRE-CREATION > > security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java > c26f0a576 > > security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java > 7875bc2a8 > security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java > 6c699cad9 > security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java > d29df930c > > security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java > 0d5689a03 > security-admin/src/main/java/org/apache/ranger/db/RangerDaoManagerBase.java > 5dffc0ef2 > security-admin/src/main/java/org/apache/ranger/db/XXAccessTypeDefDao.java > e04280bc8 > security-admin/src/main/java/org/apache/ranger/db/XXDataMaskTypeDefDao.java > 3dd4376e4 > security-admin/src/main/java/org/apache/ranger/db/XXGroupDao.java 9a87b4cfd > > security-admin/src/main/java/org/apache/ranger/db/XXPolicyConditionDefDao.java > a12140aba > > security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemAccessDao.java > 9be38decd > > security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemConditionDao.java > 9b11545a0 > > security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemDataMaskInfoDao.java > 67c7e9961 > > security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemGroupPermDao.java > a6fd8c65b > > security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemRowFilterInfoDao.java > 01a36a50d > > security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemUserPermDao.java > 667265485 > > security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java > PRE-CREATION > > security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java > PRE-CREATION > > security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java > PRE-CREATION > security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java > PRE-CREATION > > security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java > PRE-CREATION > security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java > PRE-CREATION > security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceDao.java > a7157deb4 > > security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceMapDao.java > 55d8c50e4 > security-admin/src/main/java/org/apache/ranger/db/XXResourceDefDao.java > b2e311fef > security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceDao.java > ee0e40043 > > security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementDao.java > c9a1c2132 > > security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementValueDao.java > 364af6d0f > security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDao.java > 40c3a887b > security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDefDao.java > 129f3c13c > security-admin/src/main/java/org/apache/ranger/db/XXTagDao.java e14f836e7 > security-admin/src/main/java/org/apache/ranger/db/XXTagDefDao.java > c0dd88371 > security-admin/src/main/java/org/apache/ranger/db/XXTagResourceMapDao.java > f9e041a3f > security-admin/src/main/java/org/apache/ranger/db/XXUserDao.java 58bf4d891 > security-admin/src/main/java/org/apache/ranger/entity/XXDBBase.java > 8405eb3e4 > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyBase.java > 584a103ff > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java > PRE-CREATION > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java > PRE-CREATION > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java > PRE-CREATION > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java > PRE-CREATION > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java > PRE-CREATION > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java > PRE-CREATION > > security-admin/src/main/java/org/apache/ranger/entity/XXServiceResource.java > 961627a3c > security-admin/src/main/java/org/apache/ranger/entity/XXTag.java d26a0b079 > security-admin/src/main/java/org/apache/ranger/entity/XXTagDef.java > 818908ba8 > > security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java > PRE-CREATION > > security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingTagsJson_J10020.java > PRE-CREATION > security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java > 97b8d7192 > > security-admin/src/main/java/org/apache/ranger/service/RangerAuditFields.java > e6a519a17 > > security-admin/src/main/java/org/apache/ranger/service/RangerPolicyServiceBase.java > 8d42165f7 > > security-admin/src/main/java/org/apache/ranger/service/RangerServiceDefServiceBase.java > 5cbe47ad3 > > security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceService.java > d7256802a > > security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceServiceBase.java > 6af682a81 > > security-admin/src/main/java/org/apache/ranger/service/RangerTagDefService.java > 82eb252e6 > > security-admin/src/main/java/org/apache/ranger/service/RangerTagDefServiceBase.java > 408358c48 > > security-admin/src/main/java/org/apache/ranger/service/RangerTagService.java > 28b9115fa > > security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java > c96878d21 > security-admin/src/main/resources/META-INF/jpa_named_queries.xml cdf6ba655 > security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java > c9db90a0e > security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java > 15344ea30 > > security-admin/src/test/java/org/apache/ranger/service/TestRangerTagDefService.java > 8f19ffd78 > > security-admin/src/test/java/org/apache/ranger/service/TestRangerTagService.java > 1175989e2 > > > Diff: https://reviews.apache.org/r/69072/diff/1/ > > > Testing > ------- > > Tested with local VM for migration, policy CRUD, policy browsing and policy > enforcement in Hive plugin. > > > Thanks, > > Abhay Kulkarni > >
