[jira] Commented: (HIVE-1539) Concurrent metastore threading problem
[ https://issues.apache.org/jira/browse/HIVE-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908472#action_12908472 ] Bennie Schut commented on HIVE-1539: Patch got comitted on datanucleus 2.2.0.m2 We could consider moving from 2.0 to 2.2 The temporary fix of loading your own classloader also wroked nicely. > Concurrent metastore threading problem > --- > > Key: HIVE-1539 > URL: https://issues.apache.org/jira/browse/HIVE-1539 > Project: Hadoop Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.7.0 >Reporter: Bennie Schut >Assignee: Bennie Schut > Attachments: ClassLoaderResolver.patch, thread_dump_hanging.txt > > > When running hive as a service and running a high number of queries > concurrently I end up with multiple threads running at 100% cpu without any > progress. > Looking at these threads I notice this thread(484e): > at > org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:598) > But on a different thread(63a2): > at > org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoReplaceField(MStorageDescriptor.java) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1539) Concurrent metastore threading problem
[ https://issues.apache.org/jira/browse/HIVE-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905455#action_12905455 ] Bennie Schut commented on HIVE-1539: JDOClassLoaderResolver doesn't seem thread safe. That's a bit of a surprise. I filed a bug with datanucleus: http://www.datanucleus.org/servlet/jira/browse/NUCCORE-559 I just made my own threadsafe version of the JDOClassLoaderResolver and am loading it to see if that fixes it. Will probably take a few days to be sure it got fixed. > Concurrent metastore threading problem > --- > > Key: HIVE-1539 > URL: https://issues.apache.org/jira/browse/HIVE-1539 > Project: Hadoop Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.7.0 >Reporter: Bennie Schut >Assignee: Bennie Schut > Attachments: thread_dump_hanging.txt > > > When running hive as a service and running a high number of queries > concurrently I end up with multiple threads running at 100% cpu without any > progress. > Looking at these threads I notice this thread(484e): > at > org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:598) > But on a different thread(63a2): > at > org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoReplaceField(MStorageDescriptor.java) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1539) Concurrent metastore threading problem
[ https://issues.apache.org/jira/browse/HIVE-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904093#action_12904093 ] Bennie Schut commented on HIVE-1539: Ok, I just had another few instances of hanging threads. Last time I used a SynchronizedFactory to eliminate any threading issues in the metastore client but that doesn't solve it like Paul correctly saw. "pool-1-thread-224" prio=10 tid=0x7f717d62c800 nid=0x7b4d runnable [0x4557c000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.get(HashMap.java:303) at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186) at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:185) at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:415) at org.datanucleus.store.mapped.MappedTypeManager.isSupportedMappedType(MappedTypeManager.java:84) at org.datanucleus.store.rdbms.query.legacy.AbstractIteratorStatement.(AbstractIteratorStatement.java:83) at org.datanucleus.store.rdbms.query.legacy.UnionIteratorStatement.(UnionIteratorStatement.java:147) at org.datanucleus.store.rdbms.query.legacy.ClassTableExtent.newQueryStatement(ClassTableExtent.java:204) at org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile(QueryCompiler.java:323) at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(JDOQLQueryCompiler.java:225) at org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(JDOQLQuery.java:175) at org.datanucleus.store.query.Query.executeQuery(Query.java:1628) at org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245) at org.datanucleus.store.query.Query.executeWithArray(Query.java:1499) at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:243) at org.apache.hadoop.hive.metastore.ObjectStore.getMDatabase(ObjectStore.java:323) at org.apache.hadoop.hive.metastore.ObjectStore.convertToMTable(ObjectStore.java:640) at org.apache.hadoop.hive.metastore.ObjectStore.alterTable(ObjectStore.java:946) at org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:177) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$23.run(HiveMetaStore.java:1329) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$23.run(HiveMetaStore.java:1326) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:229) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table(HiveMetaStore.java:1326) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:143) at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.common.SynchronizedFactory$Handler.invoke(SynchronizedFactory.java:46) - locked <0x0007ffc6fd00> (a org.apache.hadoop.hive.metastore.HiveMetaStoreClient) at $Proxy8.alter_table(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:267) at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:875) at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:174) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:108) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:609) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:478) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:356) at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:114) at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:378) at org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:366) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) > Concurrent metastore threading problem > --- > > Key: HIVE-1539 > URL: https://issues.apache.org/jira/browse/HIVE-1539 > Project: Hadoop Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.7.0 >Reporter: Bennie Schut >Assignee: Bennie Sch
[jira] Commented: (HIVE-1539) Concurrent metastore threading problem
[ https://issues.apache.org/jira/browse/HIVE-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901356#action_12901356 ] Bennie Schut commented on HIVE-1539: thanks Paul, I didn't notice that yet so in theory it shouldn't be the cause. However after synchronizing the metastore client I never saw this problem again where I used to see this problem every day. Strange issue. I'll see if I can create something reproducible but that might be tricky for this type of problem. > Concurrent metastore threading problem > --- > > Key: HIVE-1539 > URL: https://issues.apache.org/jira/browse/HIVE-1539 > Project: Hadoop Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.7.0 >Reporter: Bennie Schut >Assignee: Bennie Schut > Attachments: thread_dump_hanging.txt > > > When running hive as a service and running a high number of queries > concurrently I end up with multiple threads running at 100% cpu without any > progress. > Looking at these threads I notice this thread(484e): > at > org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:598) > But on a different thread(63a2): > at > org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoReplaceField(MStorageDescriptor.java) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1539) Concurrent metastore threading problem
[ https://issues.apache.org/jira/browse/HIVE-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899157#action_12899157 ] Paul Yang commented on HIVE-1539: - >From looking at the Hive class, we are creating a ThreadLocal instance of Hive >when we call get(). Since the HiveMetaStoreClient is a member of Hive, >shouldn't the client be accessed by only 1 thread at a time? > Concurrent metastore threading problem > --- > > Key: HIVE-1539 > URL: https://issues.apache.org/jira/browse/HIVE-1539 > Project: Hadoop Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.7.0 >Reporter: Bennie Schut >Assignee: Bennie Schut > Attachments: thread_dump_hanging.txt > > > When running hive as a service and running a high number of queries > concurrently I end up with multiple threads running at 100% cpu without any > progress. > Looking at these threads I notice this thread(484e): > at > org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:598) > But on a different thread(63a2): > at > org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoReplaceField(MStorageDescriptor.java) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1539) Concurrent metastore threading problem
[ https://issues.apache.org/jira/browse/HIVE-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898810#action_12898810 ] Bennie Schut commented on HIVE-1539: That would be HIVE-1482 not 28 :( > Concurrent metastore threading problem > --- > > Key: HIVE-1539 > URL: https://issues.apache.org/jira/browse/HIVE-1539 > Project: Hadoop Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.7.0 >Reporter: Bennie Schut >Assignee: Bennie Schut > Attachments: thread_dump_hanging.txt > > > When running hive as a service and running a high number of queries > concurrently I end up with multiple threads running at 100% cpu without any > progress. > Looking at these threads I notice this thread(484e): > at > org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:598) > But on a different thread(63a2): > at > org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoReplaceField(MStorageDescriptor.java) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1539) Concurrent metastore threading problem
[
https://issues.apache.org/jira/browse/HIVE-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898808#action_12898808
]
Bennie Schut commented on HIVE-1539:
This is starting to look more and more like a similar problem we have on the
jdbc driver (HIVE-1428). the HiveMetatStoreClient is doing calls on the thrift
client but the thrift client isn't threadsafe (probably by design). So when the
HiveMetaStoreClient is used it should be synchronized or a new
HiveMetaStoreClient object should be created for each thread.
>From the "Hive" class:
private IMetaStoreClient getMSC() throws MetaException {
if (metaStoreClient == null) {
metaStoreClient = createMetaStoreClient();
}
return metaStoreClient;
}
I could make the sync factory we use on HIVE-1428 a common thing and reuse it
here. Any objections?
> Concurrent metastore threading problem
> ---
>
> Key: HIVE-1539
> URL: https://issues.apache.org/jira/browse/HIVE-1539
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.7.0
>Reporter: Bennie Schut
>Assignee: Bennie Schut
> Attachments: thread_dump_hanging.txt
>
>
> When running hive as a service and running a high number of queries
> concurrently I end up with multiple threads running at 100% cpu without any
> progress.
> Looking at these threads I notice this thread(484e):
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:598)
> But on a different thread(63a2):
> at
> org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoReplaceField(MStorageDescriptor.java)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
