[
https://issues.apache.org/jira/browse/HCATALOG-366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412100#comment-13412100
]
Rohini Palaniswamy commented on HCATALOG-366:
---------------------------------------------
Rajesh,
Can you add -Dsun.security.krb5.debug=true to your client and see if you see
any of these errors in the
logs?
KrbException: Request is a replay (34) - PROCESS_TGS
or
KrbException: Fail to create credential. (63) - No service creds
> Client can't connect to metastore server
> ----------------------------------------
>
> Key: HCATALOG-366
> URL: https://issues.apache.org/jira/browse/HCATALOG-366
> Project: HCatalog
> Issue Type: Bug
> Components: client
> Affects Versions: 0.4, 0.5, 0.4.1
> Reporter: David Capwell
>
> Seems that after a while the client will start to see the following exception:
> 12/04/09 19:20:23 INFO hive.metastore: Trying to connect to metastore with
> URI thrift://hcatserver.exampl.com:9080
> 12/04/09 19:20:23 WARN hive.metastore: Failed to connect to the MetaStore
> Server...
> 12/04/09 19:20:23 INFO hive.metastore: Waiting 1 seconds before next
> connection attempt.
> 12/04/09 19:20:24 WARN hive.metastore: Failed to connect to the MetaStore
> Server...
> 12/04/09 19:20:24 INFO hive.metastore: Waiting 1 seconds before next
> connection attempt.
> 12/04/09 19:20:25 WARN hive.metastore: Failed to connect to the MetaStore
> Server...
> 12/04/09 19:20:25 INFO hive.metastore: Waiting 1 seconds before next
> connection attempt.
> 12/04/09 19:20:26 WARN hive.metastore: Failed to connect to the MetaStore
> Server...
> 12/04/09 19:20:26 INFO hive.metastore: Waiting 1 seconds before next
> connection attempt.
> 12/04/09 19:20:27 WARN hive.metastore: Failed to connect to the MetaStore
> Server...
> 12/04/09 19:20:27 WARN hive.metastore: [Ljava.lang.StackTraceElement;@92668c
> 12/04/09 19:20:27 WARN hive.metastore: Unable to connect to metastore with
> URI thrift://hcatserver.example.com:9080
> java.io.IOException: MetaException(message:Could not connect to meta store
> using any of the URIs provided)
> at
> org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:42)
> at org.apache.hcatalog.test.StoreNumbers.main(StoreNumbers.java:162)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> at
> org.apache.hcatalog.test.HCatTestDriver.main(HCatTestDriver.java:56)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: MetaException(message:Could not connect to meta store using any of
> the URIs provided)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:205)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:157
> at
> org.apache.hcatalog.mapreduce.InitializeInput.createHiveMetaClient(InitializeInput.java:58)
> at
> org.apache.hcatalog.mapreduce.InitializeInput.getSerializedHcatKeyJobInfo(InitializeInput.java:85)
> at
> org.apache.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:73)
> at
> org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:40)
> ... 13 more
> On the server side I see the following exception:
> 2012-04-09 19:20:00,344 ERROR server.TThreadPoolServer
> (TThreadPoolServer.java:run(182)) - Error occurred during processing of
> message.
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException:
> Peer indicated failure: GSS initiate failed
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:218)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:557)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:555)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:337)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1064)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge20S.java:555)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:170)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.thrift.transport.TTransportException: Peer indicated
> failure: GSS initiate failed
> at
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:190)
> at
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:124)
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:40)
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:215)
> ... 10 more
> This happens both on 0.4 and trunk. After a few seconds it seems that
> everything stabilizes again and the metastore can reply again.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
