[ https://issues.apache.org/jira/browse/SENTRY-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671815#comment-16671815 ]
kalyan kumar kalvagadda commented on SENTRY-2249: ------------------------------------------------- Batching in datanucleus works as below. It batches the commands and sends them to datastore at once. {noformat} BATCH [INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<1>,<1>,<'/u/h/w/db9/tb6/30'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<2>,<1>,<'/u/h/w/db9/tb6/9'>); INSERT INT O AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<3>,<1>,<'/u/h/w/db9/tb6/8'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<4>,<1>,<'/u/h/w/db9/tb6/7'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<5>,<1>,<'/u/h/w/db9/tb6/6'>); INSERT INTO AUTHZ_PATH (PATH_ID,AU THZ_OBJ_ID,PATH_NAME) VALUES (<6>,<1>,<'/u/h/w/db9/tb6/5'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<7>,<1>,<'/u/h/w/db9/tb6/4'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<8>,<1>,<'/u/h/w/db9/tb6/3'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VA LUES (<9>,<1>,<'/u/h/w/db9/tb6/26'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<10>,<1>,<'/u/h/w/db9/tb6/2'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<11>,<1>,<'/u/h/w/db9/tb6/27'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<12>,<1>,<'/u/ h/w/db9/tb6/1'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<13>,<1>,<'/u/h/w/db9/tb6/24'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<14>,<1>,<'/u/h/w/db9/tb6/25'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<15>,<1>,<'/u/h/w/db9/tb6/22'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<16>,<1>,<'/u/h/w/db9/tb6/23'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<17>,<1>,<'/u/h/w/db9/tb6/20'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<18>,<1>,<'/u/h/w/db9/tb6/21'>); INSERT INTO AUTHZ_ PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<19>,<1>,<'/u/h/w/db9/tb6/28'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<20>,<1>,<'/u/h/w/db9/tb6/29'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<21>,<1>,<'/u/h/w/db9/tb6/15'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTH Z_OBJ_ID,PATH_NAME) VALUES (<22>,<1>,<'/u/h/w/db9/tb6/16'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<23>,<1>,<'/u/h/w/db9/tb6/13'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<24>,<1>,<'/u/h/w/db9/tb6/14'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME ) VALUES (<25>,<1>,<'/u/h/w/db9/tb6/11'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<26>,<1>,<'/u/h/w/db9/tb6/12'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<27>,<1>,<'/u/h/w/db9/tb6/10'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<28>,<1> ,<'/u/h/w/db9/tb6/19'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<29>,<1>,<'/u/h/w/db9/tb6/17'>); INSERT INTO AUTHZ_PATH (PATH_ID,AUTHZ_OBJ_ID,PATH_NAME) VALUES (<30>,<1>,<'/u/h/w/db9/tb6/18'>)] {noformat} > Enable batch insert of HMS paths in Full Snapshot. > --------------------------------------------------- > > Key: SENTRY-2249 > URL: https://issues.apache.org/jira/browse/SENTRY-2249 > Project: Sentry > Issue Type: Improvement > Components: Sentry > Affects Versions: 2.1.0 > Reporter: kalyan kumar kalvagadda > Assignee: kalyan kumar kalvagadda > Priority: Major > Attachments: SENTRY-2249.001.patch, SENTRY-2249.002.patch, > SENTRY-2249.003.patch, SENTRY-2249.004.patch, SENTRY-2249.005.patch > > > Currently each entry in full snapshot of HMS is persisted one entry at a > time. Instead it could be optimized by persisting the path entries in > batches. DB operations are expensive, reducing the number of database > operations and around trip time will help. This would decrease the time to > persist the snapshot in to database significantly. > Size of the batch could be configurable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)