[
https://issues.apache.org/jira/browse/IMPALA-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sailesh Mukil resolved IMPALA-5596.
-----------------------------------
Resolution: Fixed
This should be fixed by:
https://github.com/apache/incubator-impala/commit/2774d80f6b604a4e6256606c6f50c9d21f715fa4#diff-d68c0b0cb2757d9c0d2e8c9a77452cdd
Reopen if we see it again.
> Data load failed with " Failed to find any Kerberos tgt" on secure cluster
> --------------------------------------------------------------------------
>
> Key: IMPALA-5596
> URL: https://issues.apache.org/jira/browse/IMPALA-5596
> Project: IMPALA
> Issue Type: Sub-task
> Components: Distributed Exec
> Reporter: Mostafa Mokhtar
> Assignee: Sailesh Mukil
>
> Error from node which failed to scan the data
> {code}
> W0627 23:31:48.568421 31308 UserGroupInformation.java:1920]
> PriviledgedActionException as:impala/va1045....@foo.com (auth:KERBEROS)
> cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> W0627 23:31:48.575223 31308 Client.java:709] Couldn't setup connection for
> impala/va1045....@foo.com to vc1512.foo/10.:8020
> Java exception follows:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:594)
> at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:396)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:761)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:757)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:756)
> at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1555)
> at org.apache.hadoop.ipc.Client.call(Client.java:1478)
> at org.apache.hadoop.ipc.Client.call(Client.java:1439)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy19.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:256)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:260)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
> at com.sun.proxy.$Proxy20.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1282)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1269)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1257)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:305)
> at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:271)
> at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:263)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1588)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:335)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:331)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:331)
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> at
> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
> ... 33 more
> W0627 23:31:48.576414 31308 UserGroupInformation.java:1920]
> PriviledgedActionException as:impala/va1045....@foo.com (auth:KERBEROS)
> cause:java.io.IOException: Couldn't setup connection for
> impala/va1045....@foo.com to vc1512.foo/10.:8020
> I0627 23:31:48.603538 31308 status.cc:123] Failed to open HDFS file
> hdfs://vc1512.foo:8020/user/hive/warehouse/tpcds_30000_parquet_new.db/catalog_sales/cs_sold_date_sk=2451404/1544efc7887e3b9b-40c41e99000000c1_1344577430_data.2.parq
> Error(255): Unknown error 255
> Root cause: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> @ 0x90cf99 impala::Status::Status()
> @ 0xbb6434 impala::DiskIoMgr::ScanRange::Open()
> @ 0xbaf00f impala::DiskIoMgr::ReadRange()
> @ 0xbb0234 impala::DiskIoMgr::WorkLoop()
> @ 0xd75742 impala::Thread::SuperviseThread()
> @ 0xd75fe4 boost::detail::thread_data<>::run()
> @ 0x102f32a thread_proxy
> @ 0x3434607aa1 (unknown)
> @ 0x34342e893d (unknown)
> I0627 23:31:48.606230 171088 hdfs-scan-node.cc:522] Error preparing scanner
> for scan range
> hdfs://vc1512.foo:8020/user/hive/warehouse/tpcds_30000_parquet_new.db/catalog_sales/cs_sold_date_sk=2451404/1544efc7887e3b9b-40c41e99000000c1_1344577430_data.2.parq(264400819:102400).
> I0627 23:31:50.615875 171066 query-state.cc:327] Instance completed.
> instance_id=3b4aeb6665ce8dae:7757b04300000003 #in-flight=1 status=GENERAL:
> Failed to open HDFS file
> hdfs://vc1512.foo:8020/user/hive/warehouse/tpcds_30000_parquet_new.db/catalog_sales/cs_sold_date_sk=2451404/1544efc7887e3b9b-40c41e99000000c1_1344577430_data.2.parq
> Error(255): Unknown error 255
> Root cause: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> I0627 23:31:50.616274 171066 query-state.cc:337] Cancel:
> query_id=3b4aeb6665ce8dae:7757b04300000000
> {code}
> Error from coordinator node
> {code}
> W0627 23:32:02.232866 191757 UserGroupInformation.java:1920]
> PriviledgedActionException as:impala/va1026.foo....@foo.com (auth:KERBEROS)
> cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> W0627 23:32:02.237445 191757 Client.java:709] Couldn't setup connection for
> impala/va1026.foo....@foo.com to vc1512.foo.com/10.:8020
> Java exception follows:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:594)
> at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:396)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:761)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:757)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:756)
> at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1555)
> at org.apache.hadoop.ipc.Client.call(Client.java:1478)
> at org.apache.hadoop.ipc.Client.call(Client.java:1439)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy19.delete(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535)
> at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:260)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
> at com.sun.proxy.$Proxy20.delete(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2062)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:684)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:680)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:680)
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> at
> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
> ... 27 more
> W0627 23:32:02.238351 191757 UserGroupInformation.java:1920]
> PriviledgedActionException as:impala/va1026.foo....@foo.com (auth:KERBEROS)
> cause:java.io.IOException: Couldn't setup connection for
> impala/va1026.foo....@foo.com to vc1512.foo.com/10.:8020
> I0627 23:32:02.939185 45977 impala-beeswax-server.cc:233] close():
> query_id=3b4aeb6665ce8dae:7757b04300000000
> I0627 23:32:02.939621 45977 impala-server.cc:952] UnregisterQuery():
> query_id=3b4aeb6665ce8dae:7757b04300000000
> I0627 23:32:02.939944 45977 impala-server.cc:1037] Cancel():
> query_id=3b4aeb6665ce8dae:7757b04300000000
> I0627 23:32:02.971863 45977 query-exec-mgr.cc:138] ReleaseQueryState():
> query_id=3b4aeb6665ce8dae:7757b04300000000 refcnt=1
> I0627 23:32:03.003536 45977 impala-server.cc:1713] Connection from client
> 10.17.187.36:48502 closed, closing 1 associated session(s)
> I0627 23:32:03.011557 45977 status.cc:123] Session closed
> @ 0x90cf99 impala::Status::Status()
> @ 0xc4421a impala::ImpalaServer::CloseSessionInternal()
> @ 0xc44900 impala::ImpalaServer::ConnectionEnd()
> @ 0xb39640
> impala::ThriftServer::ThriftServerEventProcessor::deleteContext()
> @ 0x1d24d73
> apache::thrift::server::TThreadPoolServer::Task::run()
> @ 0x1d0be59
> apache::thrift::concurrency::ThreadManager::Worker::run()
> @ 0xb3f119 impala::ThriftThread::RunRunnable()
> @ 0xb3fb72
> boost::detail::function::void_function_obj_invoker0<>::invoke()
> @ 0xd75742 impala::Thread::SuperviseThread()
> @ 0xd75fe4 boost::detail::thread_data<>::run()
> @ 0x102f32a thread_proxy
> @ 0x37dd207aa1 (unknown)
> @ 0x37dcee893d (unknown)
> W0627 23:32:12.421926 44973 UserGroupInformation.java:1403] Not attempting to
> re-login since the last re-login was attempted less than 60 seconds before.
> Last Login=1498631482824
> W0627 23:32:12.422919 44973 UserGroupInformation.java:1027] No TGT after
> renewal. Aborting renew thread for impala/va1026.foo....@foo.com
> I0627 23:43:13.903360 44895 init.cc:294] Successfully renewed kerberos TGT
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
