[jira] [Commented] (HIVE-11616) DelegationTokenSecretManager reuses the same objectstore, which has concurrency issues

2015-11-12 Thread Sergey Shelukhin (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-11616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15002734#comment-15002734
 ] 

Sergey Shelukhin commented on HIVE-11616:
-

Committed to branch-1

> DelegationTokenSecretManager reuses the same objectstore, which has 
> concurrency issues
> --
>
> Key: HIVE-11616
> URL: https://issues.apache.org/jira/browse/HIVE-11616
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.2.1
>Reporter: wangwenli
>Assignee: Sergey Shelukhin
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-11616.01.patch, HIVE-11616.02.patch, 
> HIVE-11616.patch
>
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
> sometime in metastore log, will get below exception,  after analysis, we 
> found that :
> when hivemetastore start, the DelegationTokenSecretManager will maintain the 
> same objectstore, see here
> {code}
> saslServer.startDelegationTokenSecretManager(conf, *baseHandler.getMS()*, 
> ServerMode.METASTORE);
> {code}
> this lead to the cocurrent issue.
> {code}
> 2015-08-18 20:59:10,520 | ERROR | pool-6-thread-200 | Error occurred during 
> processing of message. | 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:296)
> org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: 
> org.datanucleus.transaction.NucleusTransactionException: Invalid state. 
> Transaction has already started
>   at 
> org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:154)
>   at 
> org.apache.hadoop.hive.thrift.DBTokenStore.getToken(DBTokenStore.java:88)
>   at 
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:112)
>   at 
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:56)
>   at 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.getPassword(HadoopThriftAuthBridge.java:565)
>   at 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.handle(HadoopThriftAuthBridge.java:596)
>   at 
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
>   at 
> com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
>   at 
> org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
>   at 
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
>   at 
> org.apache.thrift.transport.HiveTSaslServerTransport.open(HiveTSaslServerTransport.java:133)
>   at 
> org.apache.thrift.transport.HiveTSaslServerTransport$Factory.getTransport(HiveTSaslServerTransport.java:261)
>   at 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
>   at 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:360)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1652)
>   at 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.datanucleus.transaction.NucleusTransactionException: Invalid 
> state. Transaction has already started
>   at 
> org.datanucleus.transaction.TransactionManager.begin(TransactionManager.java:47)
>   at org.datanucleus.TransactionImpl.begin(TransactionImpl.java:131)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.internalBegin(JDOTransaction.java:88)
>   at org.datanucleus.api.jdo.JDOTransaction.begin(JDOTransaction.java:80)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.openTransaction(ObjectStore.java:420)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.getToken(ObjectStore.java:6455)
>   at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
>

[jira] [Commented] (HIVE-11616) DelegationTokenSecretManager reuses the same objectstore, which has concurrency issues

2015-11-11 Thread wangwenli (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-11616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001560#comment-15001560
 ] 

wangwenli commented on HIVE-11616:
--

[~sershe], yes it should be

> DelegationTokenSecretManager reuses the same objectstore, which has 
> concurrency issues
> --
>
> Key: HIVE-11616
> URL: https://issues.apache.org/jira/browse/HIVE-11616
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.2.1
>Reporter: wangwenli
>Assignee: Sergey Shelukhin
> Fix For: 2.0.0
>
> Attachments: HIVE-11616.01.patch, HIVE-11616.02.patch, 
> HIVE-11616.patch
>
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
> sometime in metastore log, will get below exception,  after analysis, we 
> found that :
> when hivemetastore start, the DelegationTokenSecretManager will maintain the 
> same objectstore, see here
> {code}
> saslServer.startDelegationTokenSecretManager(conf, *baseHandler.getMS()*, 
> ServerMode.METASTORE);
> {code}
> this lead to the cocurrent issue.
> {code}
> 2015-08-18 20:59:10,520 | ERROR | pool-6-thread-200 | Error occurred during 
> processing of message. | 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:296)
> org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: 
> org.datanucleus.transaction.NucleusTransactionException: Invalid state. 
> Transaction has already started
>   at 
> org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:154)
>   at 
> org.apache.hadoop.hive.thrift.DBTokenStore.getToken(DBTokenStore.java:88)
>   at 
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:112)
>   at 
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:56)
>   at 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.getPassword(HadoopThriftAuthBridge.java:565)
>   at 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.handle(HadoopThriftAuthBridge.java:596)
>   at 
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
>   at 
> com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
>   at 
> org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
>   at 
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
>   at 
> org.apache.thrift.transport.HiveTSaslServerTransport.open(HiveTSaslServerTransport.java:133)
>   at 
> org.apache.thrift.transport.HiveTSaslServerTransport$Factory.getTransport(HiveTSaslServerTransport.java:261)
>   at 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
>   at 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:360)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1652)
>   at 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.datanucleus.transaction.NucleusTransactionException: Invalid 
> state. Transaction has already started
>   at 
> org.datanucleus.transaction.TransactionManager.begin(TransactionManager.java:47)
>   at org.datanucleus.TransactionImpl.begin(TransactionImpl.java:131)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.internalBegin(JDOTransaction.java:88)
>   at org.datanucleus.api.jdo.JDOTransaction.begin(JDOTransaction.java:80)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.openTransaction(ObjectStore.java:420)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.getToken(ObjectStore.java:6455)
>   at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
>

[jira] [Commented] (HIVE-11616) DelegationTokenSecretManager reuses the same objectstore, which has concurrency issues

2015-10-29 Thread Chaoyu Tang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-11616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981016#comment-14981016
 ] 

Chaoyu Tang commented on HIVE-11616:


+1

> DelegationTokenSecretManager reuses the same objectstore, which has 
> concurrency issues
> --
>
> Key: HIVE-11616
> URL: https://issues.apache.org/jira/browse/HIVE-11616
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.2.1
>Reporter: wangwenli
>Assignee: Sergey Shelukhin
> Attachments: HIVE-11616.01.patch, HIVE-11616.02.patch, 
> HIVE-11616.patch
>
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
> sometime in metastore log, will get below exception,  after analysis, we 
> found that :
> when hivemetastore start, the DelegationTokenSecretManager will maintain the 
> same objectstore, see here
> {code}
> saslServer.startDelegationTokenSecretManager(conf, *baseHandler.getMS()*, 
> ServerMode.METASTORE);
> {code}
> this lead to the cocurrent issue.
> {code}
> 2015-08-18 20:59:10,520 | ERROR | pool-6-thread-200 | Error occurred during 
> processing of message. | 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:296)
> org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: 
> org.datanucleus.transaction.NucleusTransactionException: Invalid state. 
> Transaction has already started
>   at 
> org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:154)
>   at 
> org.apache.hadoop.hive.thrift.DBTokenStore.getToken(DBTokenStore.java:88)
>   at 
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:112)
>   at 
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:56)
>   at 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.getPassword(HadoopThriftAuthBridge.java:565)
>   at 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.handle(HadoopThriftAuthBridge.java:596)
>   at 
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
>   at 
> com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
>   at 
> org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
>   at 
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
>   at 
> org.apache.thrift.transport.HiveTSaslServerTransport.open(HiveTSaslServerTransport.java:133)
>   at 
> org.apache.thrift.transport.HiveTSaslServerTransport$Factory.getTransport(HiveTSaslServerTransport.java:261)
>   at 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
>   at 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:360)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1652)
>   at 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.datanucleus.transaction.NucleusTransactionException: Invalid 
> state. Transaction has already started
>   at 
> org.datanucleus.transaction.TransactionManager.begin(TransactionManager.java:47)
>   at org.datanucleus.TransactionImpl.begin(TransactionImpl.java:131)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.internalBegin(JDOTransaction.java:88)
>   at org.datanucleus.api.jdo.JDOTransaction.begin(JDOTransaction.java:80)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.openTransaction(ObjectStore.java:420)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.getToken(ObjectStore.java:6455)
>   at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
>