[jira] [Commented] (HIVE-18348) Hive creates 4 different connection pools to metastore of different size
[
https://issues.apache.org/jira/browse/HIVE-18348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308532#comment-16308532
]
Eugene Koifman commented on HIVE-18348:
---
It does require 2 pools.
connPoolMutex is used to manage locking (as in multiple metastores trying to
perform the same op) via RDBMS.
This has to be done in a separate pool otherwise if you have many concurrent
ops you may end up with the 1st pool being maxed out but and so the all
operation get blocked because they need another connection to acquire the
mutex. HIVE-16321 has more details.
> Hive creates 4 different connection pools to metastore of different size
>
>
> Key: HIVE-18348
> URL: https://issues.apache.org/jira/browse/HIVE-18348
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>
> Enabling debug logging with HikariCP, I can see that Hive creates 4
> connection pools. {code:title=first connection pool creation stack trace}
> "main@1" prio=5 tid=0x1 nid=NA runnable java.lang.Thread.State: RUNNABLE at
> com.zaxxer.hikari.HikariDataSource.(HikariDataSource.java:73) at
> org.datanucleus.store.rdbms.connectionpool.HikariCPConnectionPoolFactory.createConnectionPool(HikariCPConnectionPoolFactory.java:176)
> at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:213)
> at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:117)
> - locked <0x102b> (a org.datanucleus.store.rdbms.ConnectionFactoryImpl) at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl.(ConnectionFactoryImpl.java:82)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
> at
> org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:330)
> at
> org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:203)
> at
> org.datanucleus.store.AbstractStoreManager.(AbstractStoreManager.java:162)
> at
> org.datanucleus.store.rdbms.RDBMSStoreManager.(RDBMSStoreManager.java:285)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
> at
> org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
> at
> org.datanucleus.NucleusContextHelper.createStoreManagerForProperties(NucleusContextHelper.java:133)
> at
> org.datanucleus.PersistenceNucleusContextImpl.initialise(PersistenceNucleusContextImpl.java:422)
> - locked <0x1035> (a org.datanucleus.PersistenceNucleusContextImpl) at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:817)
> - locked <0x1036> (a org.datanucleus.api.jdo.JDOPersistenceManagerFactory)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:334)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:213)
> - locked <0xeb8> (a java.lang.Class) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> javax.jdo.JDOHelper$16.run(JDOHelper.java:1965) at
> java.security.AccessController.doPrivileged(AccessController.java:-1) at
> javax.jdo.JDOHelper.invoke(JDOHelper.java:1960) at
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808) at
> javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701) at
> org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:619) -
> locked <0x957> (a
[jira] [Commented] (HIVE-18348) Hive creates 4 different connection pools to metastore of different size
[
https://issues.apache.org/jira/browse/HIVE-18348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308525#comment-16308525
]
Prasanth Jayachandran commented on HIVE-18348:
--
[~ekoifman] Does it require 2 separate pools or guaranteed connection for
cleaner and other threads? How is it different from having 2 separate pools vs
single big pool of 2x size?
> Hive creates 4 different connection pools to metastore of different size
>
>
> Key: HIVE-18348
> URL: https://issues.apache.org/jira/browse/HIVE-18348
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>
> Enabling debug logging with HikariCP, I can see that Hive creates 4
> connection pools. {code:title=first connection pool creation stack trace}
> "main@1" prio=5 tid=0x1 nid=NA runnable java.lang.Thread.State: RUNNABLE at
> com.zaxxer.hikari.HikariDataSource.(HikariDataSource.java:73) at
> org.datanucleus.store.rdbms.connectionpool.HikariCPConnectionPoolFactory.createConnectionPool(HikariCPConnectionPoolFactory.java:176)
> at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:213)
> at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:117)
> - locked <0x102b> (a org.datanucleus.store.rdbms.ConnectionFactoryImpl) at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl.(ConnectionFactoryImpl.java:82)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
> at
> org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:330)
> at
> org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:203)
> at
> org.datanucleus.store.AbstractStoreManager.(AbstractStoreManager.java:162)
> at
> org.datanucleus.store.rdbms.RDBMSStoreManager.(RDBMSStoreManager.java:285)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
> at
> org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
> at
> org.datanucleus.NucleusContextHelper.createStoreManagerForProperties(NucleusContextHelper.java:133)
> at
> org.datanucleus.PersistenceNucleusContextImpl.initialise(PersistenceNucleusContextImpl.java:422)
> - locked <0x1035> (a org.datanucleus.PersistenceNucleusContextImpl) at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:817)
> - locked <0x1036> (a org.datanucleus.api.jdo.JDOPersistenceManagerFactory)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:334)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:213)
> - locked <0xeb8> (a java.lang.Class) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> javax.jdo.JDOHelper$16.run(JDOHelper.java:1965) at
> java.security.AccessController.doPrivileged(AccessController.java:-1) at
> javax.jdo.JDOHelper.invoke(JDOHelper.java:1960) at
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808) at
> javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701) at
> org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:619) -
> locked <0x957> (a java.lang.Class) at
> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:662)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:452)
> at
[jira] [Commented] (HIVE-18348) Hive creates 4 different connection pools to metastore of different size
[
https://issues.apache.org/jira/browse/HIVE-18348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308513#comment-16308513
]
Eugene Koifman commented on HIVE-18348:
---
TxnHandler needs to use 2 pools otherwise it can deadlock.
> Hive creates 4 different connection pools to metastore of different size
>
>
> Key: HIVE-18348
> URL: https://issues.apache.org/jira/browse/HIVE-18348
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>
> Enabling debug logging with HikariCP, I can see that Hive creates 4
> connection pools. {code:title=first connection pool creation stack trace}
> "main@1" prio=5 tid=0x1 nid=NA runnable java.lang.Thread.State: RUNNABLE at
> com.zaxxer.hikari.HikariDataSource.(HikariDataSource.java:73) at
> org.datanucleus.store.rdbms.connectionpool.HikariCPConnectionPoolFactory.createConnectionPool(HikariCPConnectionPoolFactory.java:176)
> at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:213)
> at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:117)
> - locked <0x102b> (a org.datanucleus.store.rdbms.ConnectionFactoryImpl) at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl.(ConnectionFactoryImpl.java:82)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
> at
> org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:330)
> at
> org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:203)
> at
> org.datanucleus.store.AbstractStoreManager.(AbstractStoreManager.java:162)
> at
> org.datanucleus.store.rdbms.RDBMSStoreManager.(RDBMSStoreManager.java:285)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeConstructorAccessorImpl.java:-1)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
> at
> org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
> at
> org.datanucleus.NucleusContextHelper.createStoreManagerForProperties(NucleusContextHelper.java:133)
> at
> org.datanucleus.PersistenceNucleusContextImpl.initialise(PersistenceNucleusContextImpl.java:422)
> - locked <0x1035> (a org.datanucleus.PersistenceNucleusContextImpl) at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:817)
> - locked <0x1036> (a org.datanucleus.api.jdo.JDOPersistenceManagerFactory)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:334)
> at
> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:213)
> - locked <0xeb8> (a java.lang.Class) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> javax.jdo.JDOHelper$16.run(JDOHelper.java:1965) at
> java.security.AccessController.doPrivileged(AccessController.java:-1) at
> javax.jdo.JDOHelper.invoke(JDOHelper.java:1960) at
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
> at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808) at
> javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701) at
> org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:619) -
> locked <0x957> (a java.lang.Class) at
> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:662)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:452)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:389)
> at
