[
https://issues.apache.org/jira/browse/ACCUMULO-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser resolved ACCUMULO-3849.
----------------------------------
Resolution: Fixed
> Proxy sets incorrect primary for SASL server transport
> ------------------------------------------------------
>
> Key: ACCUMULO-3849
> URL: https://issues.apache.org/jira/browse/ACCUMULO-3849
> Project: Accumulo
> Issue Type: Bug
> Components: proxy
> Reporter: Josh Elser
> Assignee: Josh Elser
> Priority: Blocker
> Fix For: 1.8.0, 1.7.1
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> A doozie for a Friday afternoon before a long weekend:
> On SuSE11, KerberosProxyIT was failing with the client unable to set up the
> SASL handshake.
> {noformat}
> 2015-05-20 06:27:44,670 [proxy.Proxy] INFO : Proxy server started on
> ip-172-31-5-57.ec2.internal:57147
> 2015-05-20 06:27:45,227 [transport.TSaslServerTransport] DEBUG: transport map
> does not contain key
> 2015-05-20 06:27:45,232 [transport.TSaslServerTransport] DEBUG: Received
> start message with status START
> 2015-05-20 06:27:45,232 [transport.TSaslServerTransport] DEBUG: Received
> mechanism name 'GSSAPI'
> 2015-05-20 06:27:45,248 [transport.TSaslTransport] ERROR: SASL negotiation
> failure
> javax.security.sasl.SaslException: Failure to initialize security context
> [Caused by GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos credentails)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(GssKrb5Server.java:125)
> at
> com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85)
> at javax.security.sasl.Sasl.createSaslServer(Sasl.java:524)
> at
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:140)
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> at
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
> at
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
> at
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos credentails)
> at
> sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:87)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:127)
> at
> sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193)
> at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427)
> at sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:62)
> at
> sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(GssKrb5Server.java:108)
> ... 17 more
> 2015-05-20 06:27:45,254 [transport.TSaslServerTransport] DEBUG: failed to
> open server transport
> org.apache.thrift.transport.TTransportException: Failure to initialize
> security context
> at
> org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:221)
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:297)
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> at
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
> at
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
> at
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> at java.lang.Thread.run(Thread.java:745)
> 2015-05-20 06:27:45,260 [server.TThreadPoolServer] ERROR: Error occurred
> during processing of message.
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException:
> Failure to initialize security context
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
> at
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
> at
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
> at
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.thrift.transport.TTransportException: Failure to
> initialize security context
> at
> org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:221)
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:297)
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> ... 11 more
> {noformat}
> So, the Thrift code is unable to actually use the KRB credentials we _know_
> we logged in with. Strange.
> Looking a bit earlier, we can see that we did log in.
> {noformat}
> 2015-05-20 06:27:44,498 [security.UserGroupInformation] INFO : Login
> successful for user proxy/hostn...@example.com using keytab file
> /grid/0/hadoopqe/artifacts/accumulo/test/target/kerberos/keytabs/proxy.keytab
> 2015-05-20 06:27:44,498 [proxy.Proxy] INFO : Logged in as
> proxy/hostn...@example.com
> {noformat}
> So, for some reason, when we log in on SuSE, we somehow later dont' have the
> right credentials?
> Just after we log in, we start the Thrift server for the proxy
> {noformat}
> 2015-05-20 06:27:44,516 [rpc.TServerUtils] DEBUG: Instantiating SASL Thrift
> server
> 2015-05-20 06:27:44,524 [rpc.TServerUtils] INFO : Creating SASL thread pool
> thrift server on listening on hostname:57147
> 2015-05-20 06:27:44,532 [rpc.TServerUtils] DEBUG: Logged in as
> proxy/hostn...@example.com (auth:KERBEROS), creating TSaslServerTransport
> factory with accumulo/hostname
> {noformat}
> Hold up:
> {noformat}
> proxy/hostn...@example.com != accumulo/hostname
> {noformat}
> Turns out, when we created the ClientConfiguration for the ProxyServer, we
> didn't actually set the kerberosPrimary (the client needs to know the
> 'primary' of the principal of the server in which it's authenticating with).
> Somehow, on _every other OS and environment_ this didn't error out like it
> should have. I have no explanation why.
> Sorry, SuSE. You did it right.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
