[
https://issues.apache.org/jira/browse/SENTRY-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hong updated SENTRY-2567:
-------------------------
Priority: Blocker (was: Major)
> The operation of CreateFullSnapshot is very slowly
> --------------------------------------------------
>
> Key: SENTRY-2567
> URL: https://issues.apache.org/jira/browse/SENTRY-2567
> Project: Sentry
> Issue Type: Bug
> Components: sentrystore
> Affects Versions: 2.0.1
> Reporter: hong
> Priority: Blocker
>
> The program has been running for more than ten hours, and the progress is
> only 50%。
> {code:}
> 2021-01-20 07:51:37,068 INFO
> org.apache.sentry.provider.db.service.persistent.SentryStore: Persisting HMS
> Paths on Snapshot: authz_objs_persisted=2363(48.22%)
> authz_paths_persisted=4120420(46.96%) authz_objs_total=4900
> authz_paths_total=8773714
> 2021-01-20 07:53:07,257 INFO
> org.apache.sentry.provider.db.service.persistent.SentryStore: Persisting HMS
> Paths on Snapshot: authz_objs_persisted=2371(48.39%)
> authz_paths_persisted=4124234(47.01%) authz_objs_total=4900
> authz_paths_total=8773714
> 2021-01-20 08:16:41,192 INFO
> org.apache.sentry.provider.db.service.persistent.SentryStore: Persisting HMS
> Paths on Snapshot: authz_objs_persisted=2376(48.49%)
> authz_paths_persisted=4184844(47.70%) authz_objs_total=4900
> authz_paths_total=8773714
> 2021-01-20 08:23:05,609 INFO
> org.apache.sentry.provider.db.service.persistent.SentryStore: Persisting HMS
> Paths on Snapshot: authz_objs_persisted=2377(48.51%)
> authz_paths_persisted=4200077(47.87%) authz_objs_total=4900
> authz_paths_total=8773714
> 2021-01-20 08:39:21,506 INFO
> org.apache.sentry.provider.db.service.persistent.SentryStore: Persisting HMS
> Paths on Snapshot: authz_objs_persisted=2384(48.65%)
> authz_paths_persisted=4238551(48.31%) authz_objs_total=4900
> authz_paths_total=8773714
> 2021-01-20 08:40:45,525 INFO
> org.apache.sentry.provider.db.service.persistent.SentryStore: Persisting HMS
> Paths on Snapshot: authz_objs_persisted=2386(48.69%)
> authz_paths_persisted=4241994(48.35%) authz_objs_total=4900
> authz_paths_total=8773714
> 2021-01-20 08:42:48,569 INFO
> org.apache.sentry.provider.db.service.persistent.SentryStore: Persisting HMS
> Paths on Snapshot: authz_objs_persisted=2392(48.82%)
> authz_paths_persisted=4247050(48.41%) authz_objs_total=4900
> authz_paths_total=8773714
> 2021-01-20 08:44:00,875 INFO
> org.apache.sentry.provider.db.service.persistent.SentryStore: Persisting HMS
> Paths on Snapshot: authz_objs_persisted=2397(48.92%)
> authz_paths_persisted=4249730(48.44%) authz_objs_total=4900
> authz_paths_total=8773714
> 2021-01-20 08:52:11,582 INFO
> org.apache.sentry.provider.db.service.persistent.SentryStore: Persisting HMS
> Paths on Snapshot: authz_objs_persisted=2404(49.06%)
> authz_paths_persisted=4267894(48.64%) authz_objs_total=4900
> authz_paths_total=8773714
> 2021-01-20 08:53:14,906 INFO
> org.apache.sentry.provider.db.service.persistent.SentryStore: Persisting HMS
> Paths on Snapshot: authz_objs_persisted=2414(49.27%)
> authz_paths_persisted=4270396(48.67%) authz_objs_total=4900
> authz_paths_total=8773714
> 2021-01-20 08:55:02,671 INFO
> org.apache.sentry.provider.db.service.persistent.SentryStore: Persisting HMS
> Paths on Snapshot: authz_objs_persisted=2416(49.31%)
> authz_paths_persisted=4274600(48.72%) authz_objs_total=4900
> authz_paths_total=8773714
> 2021-01-20 09:03:00,757 INFO
> org.apache.sentry.provider.db.service.persistent.SentryStore: Persisting HMS
> Paths on Snapshot: authz_objs_persisted=2418(49.35%)
> authz_paths_persisted=4292653(48.93%) authz_objs_total=4900
> authz_paths_total=8773714
> 2021-01-20 09:10:30,447 INFO
> org.apache.sentry.provider.db.service.persistent.SentryStore: Persisting HMS
> Paths on Snapshot: authz_objs_persisted=2427(49.53%)
> authz_paths_persisted=4311199(49.14%) authz_objs_total=4900
> authz_paths_total=8773714
> 2021-01-20 09:18:00,685 INFO
> org.apache.sentry.provider.db.service.persistent.SentryStore: Persisting HMS
> Paths on Snapshot: authz_objs_persisted=2453(50.06%)
> authz_paths_persisted=4329165(49.34%) authz_objs_total=4900
> authz_paths_total=8773714
> 2021-01-20 09:23:55,050 INFO
> org.apache.sentry.provider.db.service.persistent.SentryStore: Persisting HMS
> Paths on Snapshot: authz_objs_persisted=2467(50.35%)
> authz_paths_persisted=4343061(49.50%) authz_objs_total=4900
> authz_paths_total=8773714
> 2021-01-20 09:31:23,497 INFO
> org.apache.sentry.provider.db.service.persistent.SentryStore: Persisting HMS
> Paths on Snapshot: authz_objs_persisted=2473(50.47%)
> authz_paths_persisted=4360569(49.70%) authz_objs_total=4900
> authz_paths_total=8773714
> 2021-01-20 09:32:29,101 INFO
> org.apache.sentry.provider.db.service.persistent.SentryStore: Persisting HMS
> Paths on Snapshot: authz_objs_persisted=2476(50.53%)
> authz_paths_persisted=4363157(49.73%) authz_objs_total=4900
> authz_paths_total=8773714
> 2021-01-20 09:43:28,183 INFO
> org.apache.sentry.provider.db.service.persistent.SentryStore: Persisting HMS
> Paths on Snapshot: authz_objs_persisted=2477(50.55%)
> authz_paths_persisted=4389564(50.03%) authz_objs_total=4900
> authz_paths_total=8773714
> {code}
> The thread is busy at this code:
> {code:java}
> "hms-follower" #47 prio=5 os_prio=0 tid=0x00007f00b09bb000 nid=0x1d901
> runnable [0x00007f0086253000]
> java.lang.Thread.State: RUNNABLE
> at java.util.ArrayList.indexOf(ArrayList.java:317)
> at java.util.ArrayList.contains(ArrayList.java:300)
> at
> org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:2077)
> at
> org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1913)
> at
> org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:727)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at
> org.apache.sentry.provider.db.service.model.MAuthzPathsMapping.makePersistent(MAuthzPathsMapping.java:187)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore$41.execute(SentryStore.java:2742)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:123)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager$1.call(TransactionManager.java:192)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager$ExponentialBackoff.execute(TransactionManager.java:233)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionWithRetry(TransactionManager.java:188)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.persistFullPathsImage(SentryStore.java:2711)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
