[ https://issues.apache.org/jira/browse/SENTRY-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16037384#comment-16037384 ]
Na Li commented on SENTRY-1771: ------------------------------- "2017-06-05 11:04:45,790 ERROR org.apache.sentry.provider.db.service.persistent.SentryStore: Certain perm delta is missing in SENTRY_PERM_CHANEG table! Current size of elements = 7 and expected size = 9, from changeID: 1029331. The table may get corrupted. perm changeID list: 1029331,1029332,1029333,1029334,1029335,1029336,1029339" With more debugging info, we can see the main cause of getting full snapshot is because the latest ID is one more than what's in the delta list. Maybe the latest ID is allocated, but its entry is not committed into DB. > HDFS client concurrently requests full permission update multiple times > ----------------------------------------------------------------------- > > Key: SENTRY-1771 > URL: https://issues.apache.org/jira/browse/SENTRY-1771 > Project: Sentry > Issue Type: Sub-task > Components: Sentry > Affects Versions: sentry-ha-redesign > Reporter: Na Li > Assignee: Na Li > Labels: hfs > Fix For: sentry-ha-redesign > > > When running performance test, we found there are OOM, and the memory dump > shows that there are 6 requests from HDFS to get full update for permission > when OOM happens. > Server sends back full update if the sequence number from client is 1 at > DBUpdateForwarder.getAllUpdatesFrom() > Example of call stack at the Sentry server is > {code} > 166,999K (17.4%) (org.datanucleus.ExecutionContextThreadedImpl) > <-- Java Local (org.datanucleus.ExecutionContextThreadedImpl) > [@c01196f0,@c07dbb10] > 99,019K (10.3%) (j.u.HashSet, j.u.HashMap) > <-- > org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <-- > Java Local (org.datanucleus.ExecutionContextThreadedImpl) > [@c01196f0,@c07dbb10] > 85,196K (8.9%) (org.datanucleus.identity.OIDImpl) > <-- {j.u.HashSet} <-- > org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <-- > Java Local (org.datanucleus.ExecutionContextThreadedImpl) > [@c01196f0,@c07dbb10] > 64,606K (6.7%) > <-- org.datanucleus.identity.OIDImpl.toString <-- {j.u.HashSet} <-- > org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <-- > Java Local (org.datanucleus.ExecutionContextThreadedImpl) > [@c01196f0,@c07dbb10] > GC root stack trace: > > org.datanucleus.store.rdbms.scostore.JoinSetStore.iterator(JoinSetStore.java:914) > org.datanucleus.store.types.backed.Set.loadFromStore(Set.java:323) > org.datanucleus.store.types.backed.Set.initialise(Set.java:272) > org.datanucleus.store.types.SCOUtils.createSCOWrapper(SCOUtils.java:256) > org.datanucleus.store.types.SCOUtils.newSCOInstance(SCOUtils.java:142) > > org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.replaceFieldWithWrapper(AbstractContainerMapping.java:399) > > org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.postFetch(AbstractContainerMapping.java:417) > > org.datanucleus.store.rdbms.request.FetchRequest.execute(FetchRequest.java:420) > > org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:324) > > org.datanucleus.state.AbstractStateManager.loadFieldsFromDatastore(AbstractStateManager.java:1122) > > org.datanucleus.state.JDOStateManager.loadUnloadedFieldsInFetchPlan(JDOStateManager.java:3000) > > org.datanucleus.state.AbstractStateManager.loadFieldsInFetchPlan(AbstractStateManager.java:1064) > > org.datanucleus.ExecutionContextImpl.performDetachAllOnTxnEndPreparation(ExecutionContextImpl.java:4574) > > org.datanucleus.ExecutionContextImpl.preCommit(ExecutionContextImpl.java:4259) > > org.datanucleus.ExecutionContextImpl.transactionPreCommit(ExecutionContextImpl.java:654) > > org.datanucleus.TransactionImpl.internalPreCommit(TransactionImpl.java:379) > org.datanucleus.TransactionImpl.commit(TransactionImpl.java:268) > org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98) > > org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:113) > > org.apache.sentry.provider.db.service.persistent.SentryStore.retrieveFullPermssionsImage(SentryStore.java:2225) > > org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:56) > > org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:40) > > org.apache.sentry.hdfs.DBUpdateForwarder.getAllUpdatesFrom(DBUpdateForwarder.java:85) > > org.apache.sentry.hdfs.SentryPlugin.getAllPermsUpdatesFrom(SentryPlugin.java:178) > > org.apache.sentry.hdfs.SentryHDFSServiceProcessor.get_all_authz_updates_from(SentryHDFSServiceProcessor.java:48) > > org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:403) > > org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:388) > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > > org.apache.sentry.hdfs.SentryHDFSServiceProcessorFactory$ProcessorWrapper.process(SentryHDFSServiceProcessorFactory.java:47) > > org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123) > > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)