-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69072/
-----------------------------------------------------------
Review request for ranger, Madhan Neethiraj, Ramesh Mani, and Velmurugan
Periasamy.
Summary (updated)
-----------------
RANGER-2203 : Review and update database schema for ranger policies to minimize
database queries/updates, RANGER-2219: De-normalize schema for storing tags and
related objects
Bugs: RANGER-2203 and RANGER-2219
https://issues.apache.org/jira/browse/RANGER-2203
https://issues.apache.org/jira/browse/RANGER-2219
Repository: ranger
Description (updated)
-------
Currently, ranger policies are fully normalized and stored in a multiple
Relational database tables. There is a performance overhead incurred when
retrieving a ranger policy, as multiple database accesses are required to fully
reconstruct it. This is significant when there are large ranger policies (that
is, the number of resources addressed by the policy is large), and/or when
there is a large number of ranger policies in an installation.
This Jira tracks alternate design of database schema, where a policy is stored
in a de-normalized way, in its entirely, in one database table (preferably as a
JSON string).
Currently, tag-definitions, tags and service-resources are stored in database
using a normalized form. When constructing resource->tag mappings, this schema
design may lead to a large number of database accesses, thereby causing a major
performance bottleneck when the number of resource->tag associations is large.
Denormalized schema will reduce the number of database accesses, and improve
overall performance significantly.
Diffs (updated)
-----
agents-common/pom.xml 09a32fa8b
agents-common/src/main/java/org/apache/ranger/authorization/utils/JsonUtils.java
bc4a8b523
agents-common/src/main/java/org/apache/ranger/plugin/model/RangerServiceDef.java
adafb9985
agents-common/src/main/java/org/apache/ranger/plugin/store/TagStore.java
5918b1292
embeddedwebserver/scripts/ranger-admin-services.sh b8ca6c7d5
security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 32cf6db3b
security-admin/db/mysql/patches/035-update-schema-for-x-policy.sql
PRE-CREATION
security-admin/db/mysql/patches/036-denormalize-tag-tables.sql PRE-CREATION
security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql
2e577f3bd
security-admin/db/oracle/patches/035-update-schema-for-x-policy.sql
PRE-CREATION
security-admin/db/oracle/patches/036-denormalize-tag-tables.sql PRE-CREATION
security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
bad32efbc
security-admin/db/postgres/patches/035-update-schema-for-x-policy.sql
PRE-CREATION
security-admin/db/postgres/patches/036-denormalize-tag-tables.sql
PRE-CREATION
security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql
9482992e7
security-admin/db/sqlanywhere/patches/035-update-schema-for-x-policy.sql
PRE-CREATION
security-admin/db/sqlanywhere/patches/036-denormalize-tag-tables.sql
PRE-CREATION
security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql
85f328556
security-admin/db/sqlserver/patches/035-update-schema-for-x-policy.sql
PRE-CREATION
security-admin/db/sqlserver/patches/036-denormalize-tag-tables.sql
PRE-CREATION
security-admin/scripts/db_setup.py 02701c726
security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java
PRE-CREATION
security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java
c26f0a576
security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java
7875bc2a8
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
6c699cad9
security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java d29df930c
security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java
0d5689a03
security-admin/src/main/java/org/apache/ranger/db/RangerDaoManagerBase.java
5dffc0ef2
security-admin/src/main/java/org/apache/ranger/db/XXAccessTypeDefDao.java
e04280bc8
security-admin/src/main/java/org/apache/ranger/db/XXDataMaskTypeDefDao.java
3dd4376e4
security-admin/src/main/java/org/apache/ranger/db/XXGroupDao.java 9a87b4cfd
security-admin/src/main/java/org/apache/ranger/db/XXPolicyConditionDefDao.java
a12140aba
security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemAccessDao.java
9be38decd
security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemConditionDao.java
9b11545a0
security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemDataMaskInfoDao.java
67c7e9961
security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemGroupPermDao.java
a6fd8c65b
security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemRowFilterInfoDao.java
01a36a50d
security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemUserPermDao.java
667265485
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java
PRE-CREATION
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java
PRE-CREATION
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java
PRE-CREATION
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java
PRE-CREATION
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java
PRE-CREATION
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java
PRE-CREATION
security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceDao.java
a7157deb4
security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceMapDao.java
55d8c50e4
security-admin/src/main/java/org/apache/ranger/db/XXResourceDefDao.java
b2e311fef
security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceDao.java
ee0e40043
security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementDao.java
c9a1c2132
security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementValueDao.java
364af6d0f
security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDao.java
40c3a887b
security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDefDao.java
129f3c13c
security-admin/src/main/java/org/apache/ranger/db/XXTagDao.java e14f836e7
security-admin/src/main/java/org/apache/ranger/db/XXTagDefDao.java c0dd88371
security-admin/src/main/java/org/apache/ranger/db/XXTagResourceMapDao.java
f9e041a3f
security-admin/src/main/java/org/apache/ranger/db/XXUserDao.java 58bf4d891
security-admin/src/main/java/org/apache/ranger/entity/XXDBBase.java 8405eb3e4
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyBase.java
584a103ff
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java
PRE-CREATION
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java
PRE-CREATION
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java
PRE-CREATION
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java
PRE-CREATION
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java
PRE-CREATION
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java
PRE-CREATION
security-admin/src/main/java/org/apache/ranger/entity/XXServiceResource.java
961627a3c
security-admin/src/main/java/org/apache/ranger/entity/XXTag.java d26a0b079
security-admin/src/main/java/org/apache/ranger/entity/XXTagDef.java 818908ba8
security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java
PRE-CREATION
security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingTagsJson_J10020.java
PRE-CREATION
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java
97b8d7192
security-admin/src/main/java/org/apache/ranger/service/RangerAuditFields.java
e6a519a17
security-admin/src/main/java/org/apache/ranger/service/RangerPolicyServiceBase.java
8d42165f7
security-admin/src/main/java/org/apache/ranger/service/RangerServiceDefServiceBase.java
5cbe47ad3
security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceService.java
d7256802a
security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceServiceBase.java
6af682a81
security-admin/src/main/java/org/apache/ranger/service/RangerTagDefService.java
82eb252e6
security-admin/src/main/java/org/apache/ranger/service/RangerTagDefServiceBase.java
408358c48
security-admin/src/main/java/org/apache/ranger/service/RangerTagService.java
28b9115fa
security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java
c96878d21
security-admin/src/main/resources/META-INF/jpa_named_queries.xml cdf6ba655
security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java
c9db90a0e
security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java
15344ea30
security-admin/src/test/java/org/apache/ranger/service/TestRangerTagDefService.java
8f19ffd78
security-admin/src/test/java/org/apache/ranger/service/TestRangerTagService.java
1175989e2
Diff: https://reviews.apache.org/r/69072/diff/1/
Testing (updated)
-------
Tested with local VM for migration, policy CRUD, policy browsing and policy
enforcement in Hive plugin.
Thanks,
Abhay Kulkarni
