----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68881/ -----------------------------------------------------------
(Updated Oct. 5, 2018, 2:47 a.m.) Review request for ranger, Madhan Neethiraj, Mehul Parikh, Nikhil P, Pradeep Agrawal, and Velmurugan Periasamy. Changes ------- Addressed some review comments. Bugs: RANGER-2186, RANGER-2195 and RANGER-2219 https://issues.apache.org/jira/browse/RANGER-2186 https://issues.apache.org/jira/browse/RANGER-2195 https://issues.apache.org/jira/browse/RANGER-2219 Repository: ranger Description ------- Notes: This is a composite patch for updating database schema for Ranger policy and tag related objects. Notes: This patch is more complete than the one posted earlier (https://reviews.apache.org/r/68876/). Database scripts to drop foreign-keys are missing for some database types. They need to be added. This patch also subsumes back-ports of RANGER-2186 and RANGER-2195 from master to ranger-0.7 branch. Porting those JIRA separately involved fixing conflicts in many common files. Description: Currently, ranger policies are fully normalized and stored in a multiple Relational database tables. There is a performance overhead incurred when retrieving a ranger policy, as multiple database accesses are required to fully reconstruct it. This is significant when there are large ranger policies (that is, the number of resources addressed by the policy is large), and/or when there is a large number of ranger policies in an installation. This Jira tracks alternate design of database schema, where a policy is stored in a de-normalized way, in its entirely, in one database table (preferably as a JSON string). Diffs (updated) ----- agents-common/src/main/java/org/apache/ranger/authorization/utils/JsonUtils.java PRE-CREATION agents-common/src/main/java/org/apache/ranger/plugin/model/RangerTag.java 743b02825 agents-common/src/main/java/org/apache/ranger/plugin/store/AbstractServiceStore.java a75ca59e8 agents-common/src/main/java/org/apache/ranger/plugin/store/TagStore.java 5918b1292 embeddedwebserver/scripts/ranger-admin-services.sh 19dab2a9d security-admin/db/mysql/patches/035-update-schema-for-x-policy.sql PRE-CREATION security-admin/db/mysql/patches/040-denormalize-tag-tables.sql PRE-CREATION security-admin/db/oracle/patches/035-update-schema-for-x-policy.sql PRE-CREATION security-admin/db/oracle/patches/040-denormalize-tag-tables.sql PRE-CREATION security-admin/db/postgres/patches/035-update-schema-for-x-policy.sql PRE-CREATION security-admin/db/postgres/patches/040-denormalize-tag-tables.sql PRE-CREATION security-admin/db/sqlanywhere/patches/035-update-schema-for-x-policy.sql PRE-CREATION security-admin/db/sqlanywhere/patches/040-denormalize-tag-tables.sql PRE-CREATION security-admin/db/sqlserver/patches/035-update-schema-for-x-policy.sql PRE-CREATION security-admin/db/sqlserver/patches/040-denormalize-tag-tables.sql PRE-CREATION security-admin/scripts/db_setup.py d4f37ed6f security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 354ab1cba security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java 53683ec04 security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ceee8cef2 security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java 3234be6a2 security-admin/src/main/java/org/apache/ranger/common/db/RangerTransactionSynchronizationAdapter.java 2a62fb408 security-admin/src/main/java/org/apache/ranger/db/RangerDaoManager.java e3b878b3e security-admin/src/main/java/org/apache/ranger/db/RangerDaoManagerBase.java d718441ee security-admin/src/main/java/org/apache/ranger/db/XXAccessTypeDefDao.java 8f3a50687 security-admin/src/main/java/org/apache/ranger/db/XXDataMaskTypeDefDao.java f6e1aff0b security-admin/src/main/java/org/apache/ranger/db/XXGroupDao.java 19e2e11f3 security-admin/src/main/java/org/apache/ranger/db/XXPolicyConditionDefDao.java 21afcac89 security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemAccessDao.java de37e10cd security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemConditionDao.java 11596ef4d security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemDataMaskInfoDao.java a8418c626 security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemGroupPermDao.java 8c05699b2 security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemRowFilterInfoDao.java 4618e7dc7 security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemUserPermDao.java 40a0da13b security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceDao.java 4b04b9664 security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceMapDao.java 70657370a security-admin/src/main/java/org/apache/ranger/db/XXResourceDefDao.java aee56e833 security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceDao.java f87c0ae59 security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementDao.java 72fe2140b security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementValueDao.java e726d047c security-admin/src/main/java/org/apache/ranger/db/XXServiceVersionInfoDao.java 0098bff23 security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDao.java e361b33bd security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDefDao.java 145399f1c security-admin/src/main/java/org/apache/ranger/db/XXTagDao.java 77428dbf9 security-admin/src/main/java/org/apache/ranger/db/XXTagDefDao.java dfd2fcb5d security-admin/src/main/java/org/apache/ranger/db/XXTagResourceMapDao.java bbcd54618 security-admin/src/main/java/org/apache/ranger/db/XXUserDao.java 7af1bf96a security-admin/src/main/java/org/apache/ranger/entity/XXDBBase.java 8405eb3e4 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyBase.java 69d28bb54 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/entity/XXServiceResource.java 961627a3c security-admin/src/main/java/org/apache/ranger/entity/XXTag.java 9155385ec security-admin/src/main/java/org/apache/ranger/entity/XXTagDef.java 818908ba8 security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10017.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingTagsJson_J10020.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 6685e1a58 security-admin/src/main/java/org/apache/ranger/service/RangerAuditFields.java 7223f109c security-admin/src/main/java/org/apache/ranger/service/RangerPolicyServiceBase.java 0195c72a7 security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceService.java d7256802a security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceServiceBase.java 6af682a81 security-admin/src/main/java/org/apache/ranger/service/RangerTagDefService.java 82eb252e6 security-admin/src/main/java/org/apache/ranger/service/RangerTagService.java 28b9115fa security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4a7055dee security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 44523fb22 security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java 68908b205 security-admin/src/test/java/org/apache/ranger/service/TestRangerServiceDefService.java d65b0d6ee security-admin/src/test/java/org/apache/ranger/service/TestRangerTagDefService.java 2ca0fd2fb Diff: https://reviews.apache.org/r/68881/diff/8/ Changes: https://reviews.apache.org/r/68881/diff/7-8/ Testing ------- Thanks, Abhay Kulkarni