> On Aug. 5, 2019, 6:01 p.m., Abhay Kulkarni wrote: > > security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java > > Line 4178 (original), 4189 (patched) > > <https://reviews.apache.org/r/71237/diff/1/?file=2159811#file2159811line4191> > > > > totalDeletedPolicies is not incremented in the loop. It will cause > > bulkModeOnlyFlushAndClear() to be called for each deleted policy. Please > > review.
Thanks for the review. Still testing this issue with large number of policy resources(like 500 table+columns in each policy). Found performance issue there. Seems we need to incorporate your suggestion mentioned in the Approach-1 if not then little more optimization in Approach-1 patch. - Pradeep ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71237/#review217080 ----------------------------------------------------------- On Aug. 6, 2019, 3:29 a.m., Pradeep Agrawal wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71237/ > ----------------------------------------------------------- > > (Updated Aug. 6, 2019, 3:29 a.m.) > > > Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay > Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh > Mani, Sailaja Polavarapu, and Velmurugan Periasamy. > > > Bugs: RANGER-2499 > https://issues.apache.org/jira/browse/RANGER-2499 > > > Repository: ranger > > > Description > ------- > > **This RR contains alternate solution of RANGER-2499. First solution RR is > https://reviews.apache.org/r/71236/** > > **Problem Statement:** Ranger takes an extremely long time to import bulk > policies. > **Current performance(Without patch):** > Time taken to import 500 policies : 27+ Minutes > Time taken to import 1000 policies : 81+ Minutes > > **Proposed Solution:** > in the bulk policy creation/deletion operation proposed patch shall call > EntityManager.flush() and EntityManager.clear() after processing a batch of > 10 records. > > > ** Note: ** > Most of the changes in this patch comes from revert work of RANGER-2493 as I > am trying to generalize the solution for bulk create and update policies also. > Important changes are: > 1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class. > 2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore > wherever bulk create or update is made. > 3) batch create/delete optimization and batch is configurable through > ranger-admin-default-site.xml file > 4) few null checks in the existing code. > > > **References: ** > https://en.wikibooks.org/wiki/Java_Persistence/Persisting > https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/ > > > Diffs > ----- > > > agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java > d4879766f > security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java > e242d9092 > security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java > d49ea9886 > > security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java > 4815f5bdb > security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java > ef22354de > security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java > 4f339087d > > security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java > 0788fb5bc > security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java > 348d072e6 > > security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java > de5d34307 > > security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java > 019816e49 > security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml > eb7947a47 > security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java > b6f13f47e > > security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java > 655e17f40 > > > Diff: https://reviews.apache.org/r/71237/diff/2/ > > > Testing > ------- > > Time taken to import 500 policies(500 create) : 45 Seconds > Time taken to import 500 policies(500 delete+500 create) : 60 Seconds > > Time taken to import 1000 policies(1000 create) : 76 Seconds > Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds > > > Time taken to delete 500 policies : 30 Seconds > > > Thanks, > > Pradeep Agrawal > >
