[
https://issues.apache.org/jira/browse/HIVE-18348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16308513#comment-16308513
]
Eugene Koifman commented on HIVE-18348:
---------------------------------------
TxnHandler needs to use 2 pools otherwise it can deadlock.
> Hive creates 4 different connection pools to metastore of different size
> ------------------------------------------------------------------------
>
> Key: HIVE-18348
> URL: https://issues.apache.org/jira/browse/HIVE-18348
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 3.0.0
> Reporter: Prasanth Jayachandran
>
> Enabling debug logging with HikariCP, I can see that Hive creates 4
> connection pools. {code:title=first connection pool creation stack trace}
> "main@1" prio=5 tid=0x1 nid=NA runnable java.lang.Thread.State: RUNNABLE at
> com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:73) at
> org.datanucleus.store.rdbms.connectionpool.HikariCPConnectionPoolFactory.createConnectionPool(HikariCPConnectionPoolFactory.java:176)
> at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:213)
> at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:117)
> - locked <0x102b> (a org.datanucleus.store.rdbms.ConnectionFactoryImpl) at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:82)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
> at
> org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:330)
> at
> org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:203)
> at
> org.datanucleus.store.AbstractStoreManager.<init>(AbstractStoreManager.java:162)
> at
> org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:285)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
> at
> org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
> at
> org.datanucleus.NucleusContextHelper.createStoreManagerForProperties(NucleusContextHelper.java:133)
> at
> org.datanucleus.PersistenceNucleusContextImpl.initialise(PersistenceNucleusContextImpl.java:422)
> - locked <0x1035> (a org.datanucleus.PersistenceNucleusContextImpl) at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:817)
> - locked <0x1036> (a org.datanucleus.api.jdo.JDOPersistenceManagerFactory)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:334)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:213)
> - locked <0xeb8> (a java.lang.Class) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> javax.jdo.JDOHelper$16.run(JDOHelper.java:1965) at
> java.security.AccessController.doPrivileged(AccessController.java:-1) at
> javax.jdo.JDOHelper.invoke(JDOHelper.java:1960) at
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808) at
> javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701) at
> org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:619) -
> locked <0x957> (a java.lang.Class) at
> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:662)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:452)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:389)
> at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:344)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76) at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:59)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:668)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:647)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:641)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:695)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:491)
> - locked <0xe59> (a java.lang.Class) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:7518)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:161)
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:79)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:84)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:95)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:4100)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4152) - locked
> <0xe92> (a org.apache.hadoop.hive.ql.metadata.Hive) at
> org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4132) at
> org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4394) at
> org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:267) at
> org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:250)
> at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:407) at
> org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:351) at
> org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:331) at
> org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:307) at
> org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:942)
> at
> org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:906)
> at
> org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1856)
> at
> org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:131)
> at org.apache.hive.service.cli.CLIService.init(CLIService.java:115) - locked
> <0xe94> (a org.apache.hive.service.cli.CLIService) at
> org.apache.hive.service.CompositeService.init(CompositeService.java:59) -
> locked <0xe95> (a org.apache.hive.service.server.HiveServer2) at
> org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:167) at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:790)
> at
> org.apache.hive.service.server.HiveServer2.access$900(HiveServer2.java:110)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1034)
> at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:903) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.util.RunJar.run(RunJar.java:234) at
> org.apache.hadoop.util.RunJar.main(RunJar.java:148) {code} {code:title=second
> connection pool creation stack trace} "main@1" prio=5 tid=0x1 nid=NA runnable
> java.lang.Thread.State: RUNNABLE at
> com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:73) at
> org.datanucleus.store.rdbms.connectionpool.HikariCPConnectionPoolFactory.createConnectionPool(HikariCPConnectionPoolFactory.java:176)
> at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:213)
> at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:134)
> - locked <0x105e> (a org.datanucleus.store.rdbms.ConnectionFactoryImpl) at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl.createManagedConnection(ConnectionFactoryImpl.java:249)
> at
> org.datanucleus.store.connection.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:302)
> at
> org.datanucleus.store.connection.AbstractConnectionFactory.getConnection(AbstractConnectionFactory.java:84)
> at
> org.datanucleus.store.AbstractStoreManager.getConnection(AbstractStoreManager.java:378)
> at
> org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:296)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
> at
> org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
> at
> org.datanucleus.NucleusContextHelper.createStoreManagerForProperties(NucleusContextHelper.java:133)
> at
> org.datanucleus.PersistenceNucleusContextImpl.initialise(PersistenceNucleusContextImpl.java:422)
> - locked <0x1035> (a org.datanucleus.PersistenceNucleusContextImpl) at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:817)
> - locked <0x1036> (a org.datanucleus.api.jdo.JDOPersistenceManagerFactory)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:334)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:213)
> - locked <0xeb8> (a java.lang.Class) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> javax.jdo.JDOHelper$16.run(JDOHelper.java:1965) at
> java.security.AccessController.doPrivileged(AccessController.java:-1) at
> javax.jdo.JDOHelper.invoke(JDOHelper.java:1960) at
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808) at
> javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701) at
> org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:619) -
> locked <0x957> (a java.lang.Class) at
> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:662)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:452)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:389)
> at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:344)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76) at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:59)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:668)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:647)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:641)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:695)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:491)
> - locked <0xe59> (a java.lang.Class) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:7518)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:161)
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:79)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:84)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:95)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:148)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:4100)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4152) - locked
> <0xe92> (a org.apache.hadoop.hive.ql.metadata.Hive) at
> org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:4132) at
> org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4394) at
> org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:267) at
> org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:250)
> at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:407) at
> org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:351) at
> org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:331) at
> org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:307) at
> org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:942)
> at
> org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:906)
> at
> org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1856)
> at
> org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:131)
> at org.apache.hive.service.cli.CLIService.init(CLIService.java:115) - locked
> <0xe94> (a org.apache.hive.service.cli.CLIService) at
> org.apache.hive.service.CompositeService.init(CompositeService.java:59) -
> locked <0xe95> (a org.apache.hive.service.server.HiveServer2) at
> org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:167) at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:790)
> at
> org.apache.hive.service.server.HiveServer2.access$900(HiveServer2.java:110)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1034)
> at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:903) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.util.RunJar.run(RunJar.java:234) at
> org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {code}
> 2 other instances of connection pool are explicitly created (connPool and
> connPoolMutex) in TxnHandler
> https://github.com/apache/hive/blob/035eca39fc0bd7aa0d9c1809a26e000ac52978d0/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L256-L264
>
> If datanucleus.connectionPool.maxPoolSize is set to 10, then hive will create
> 47 (connPoolMutex creates 7 more than maxPoolSize) connections per HS2 and
> HMS instance. This blows up the number of connections on DB side. Ideally, we
> want a single shared connection pool for ObjectStore and TxnHandler so that
> it is easily tuneable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
