[
https://issues.apache.org/jira/browse/HIVE-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904093#action_12904093
]
Bennie Schut commented on HIVE-1539:
------------------------------------
Ok, I just had another few instances of hanging threads. Last time I used a
SynchronizedFactory to eliminate any threading issues in the metastore client
but that doesn't solve it like Paul correctly saw.
"pool-1-thread-224" prio=10 tid=0x00007f717d62c800 nid=0x7b4d runnable
[0x000000004557c000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.get(HashMap.java:303)
at
org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186)
at
org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:185)
at
org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:415)
at
org.datanucleus.store.mapped.MappedTypeManager.isSupportedMappedType(MappedTypeManager.java:84)
at
org.datanucleus.store.rdbms.query.legacy.AbstractIteratorStatement.<init>(AbstractIteratorStatement.java:83)
at
org.datanucleus.store.rdbms.query.legacy.UnionIteratorStatement.<init>(UnionIteratorStatement.java:147)
at
org.datanucleus.store.rdbms.query.legacy.ClassTableExtent.newQueryStatement(ClassTableExtent.java:204)
at
org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile(QueryCompiler.java:323)
at
org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(JDOQLQueryCompiler.java:225)
at
org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(JDOQLQuery.java:175)
at org.datanucleus.store.query.Query.executeQuery(Query.java:1628)
at
org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245)
at org.datanucleus.store.query.Query.executeWithArray(Query.java:1499)
at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:243)
at
org.apache.hadoop.hive.metastore.ObjectStore.getMDatabase(ObjectStore.java:323)
at
org.apache.hadoop.hive.metastore.ObjectStore.convertToMTable(ObjectStore.java:640)
at
org.apache.hadoop.hive.metastore.ObjectStore.alterTable(ObjectStore.java:946)
at
org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:177)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$23.run(HiveMetaStore.java:1329)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$23.run(HiveMetaStore.java:1326)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:229)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table(HiveMetaStore.java:1326)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:143)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.common.SynchronizedFactory$Handler.invoke(SynchronizedFactory.java:46)
- locked <0x00000007ffc6fd00> (a
org.apache.hadoop.hive.metastore.HiveMetaStoreClient)
at $Proxy8.alter_table(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:267)
at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:875)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:174)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:108)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:609)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:478)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:356)
at
org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:114)
at
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:378)
at
org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:366)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
> Concurrent metastore threading problem
> ---------------------------------------
>
> Key: HIVE-1539
> URL: https://issues.apache.org/jira/browse/HIVE-1539
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.7.0
> Reporter: Bennie Schut
> Assignee: Bennie Schut
> Attachments: thread_dump_hanging.txt
>
>
> When running hive as a service and running a high number of queries
> concurrently I end up with multiple threads running at 100% cpu without any
> progress.
> Looking at these threads I notice this thread(484e):
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:598)
> But on a different thread(63a2):
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoReplaceField(MStorageDescriptor.java)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
