> Caused by: AUTHENTICATION : Could not authenticate, GSSException: No valid > credentials provided (Mechanism level: Failed to find any Kerberos tgt)
This shows that the Falcon kerberos context become invalidated. There was a fix in later versions and patch should be available from HWX support for the older versions Thanks Venkat ________________________________________ From: Vishal Gupta <[email protected]> Sent: Wednesday, December 21, 2016 9:19 PM To: [email protected] Subject: Re: Falcon issue after Kerberos implementation Cluster is HDP 2.3.2 On Thu, Dec 22, 2016 at 1:17 PM, Vishal Gupta <[email protected]> wrote: > Hello Team, > We are implementing Kerberos at one the clusters. > > After Kerberos implementation looks like Falcon while using SPNEGO to > authenticate the user, is failing while authenticating and connecting to > Oozie. > > > > Any suggestions please. Details below.. > > > 2016-12-21 07:52:42,597 ERROR - [1529601858@qtp-708533063-243 - > 737162dd-6235-494c-a0b4-c8e132fc5491:ifrsdev:DELETE// > entities/delete/process/edmhdpif-oozie-icbdev-th] ~ Unable to reach > workflow engine for deletion or deletion failed (AbstractEntityManager:265) > > org.apache.falcon.FalconException: IO_ERROR : java.io.IOException: Error > while connecting Oozie server. No of retries = 1. Exception = null > > at org.apache.falcon.workflow.engine.OozieWorkflowEngine. > findBundles(OozieWorkflowEngine.java:303) > > at org.apache.falcon.workflow.engine.OozieWorkflowEngine. > doBundleAction(OozieWorkflowEngine.java:377) > > at org.apache.falcon.workflow.engine.OozieWorkflowEngine. > doBundleAction(OozieWorkflowEngine.java:371) > > at org.apache.falcon.workflow.engine.OozieWorkflowEngine. > delete(OozieWorkflowEngine.java:355) > > at org.apache.falcon.resource. > AbstractEntityManager.delete(AbstractEntityManager.java:252) > > at org.apache.falcon.resource.ConfigSyncService.delete( > ConfigSyncService.java:62) > > at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown > Source) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at org.apache.falcon.resource.channel.IPCChannel.invoke( > IPCChannel.java:49) > > at org.apache.falcon.resource.proxy. > SchedulableEntityManagerProxy$3.doExecute(SchedulableEntityManagerProxy. > java:230) > > at org.apache.falcon.resource.proxy. > SchedulableEntityManagerProxy$EntityProxy.execute( > SchedulableEntityManagerProxy.java:575) > > at org.apache.falcon.resource.proxy. > SchedulableEntityManagerProxy$3.execute(SchedulableEntityManagerProxy. > java:219) > > at org.apache.falcon.resource.proxy. > SchedulableEntityManagerProxy.delete_aroundBody2( > SchedulableEntityManagerProxy.java:232) > > at org.apache.falcon.resource.proxy. > SchedulableEntityManagerProxy$AjcClosure3.run( > SchedulableEntityManagerProxy.java:1) > > at org.aspectj.runtime.reflect.JoinPointImpl.proceed( > JoinPointImpl.java:149) > > at org.apache.falcon.aspect.AbstractFalconAspect. > logAroundMonitored(AbstractFalconAspect.java:51) > > at org.apache.falcon.resource.proxy. > SchedulableEntityManagerProxy.delete(SchedulableEntityManagerProxy. > java:206) > > at sun.reflect.GeneratedMethodAccessor87.invoke(Unknown > Source) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at com.sun.jersey.spi.container. > JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) > > at com.sun.jersey.server.impl.model.method.dispatch. > AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch( > AbstractResourceMethodDispatchProvider.java:185) > > at com.sun.jersey.server.impl.model.method.dispatch. > ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher. > java:75) > > at com.sun.jersey.server.impl.uri.rules.HttpMethodRule. > accept(HttpMethodRule.java:288) > > at com.sun.jersey.server.impl.uri.rules.RightHandPathRule. > accept(RightHandPathRule.java:147) > > at com.sun.jersey.server.impl.uri.rules.ResourceClassRule. > accept(ResourceClassRule.java:108) > > at com.sun.jersey.server.impl.uri.rules.RightHandPathRule. > accept(RightHandPathRule.java:147) > > at com.sun.jersey.server.impl.uri.rules. > RootResourceClassesRule.accept(RootResourceClassesRule.java:84) > > at com.sun.jersey.server.impl.application. > WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) > > at com.sun.jersey.server.impl.application. > WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) > > at com.sun.jersey.server.impl.application. > WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) > > at com.sun.jersey.server.impl.application. > WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) > > at com.sun.jersey.spi.container. > servlet.WebComponent.service(WebComponent.java:416) > > at com.sun.jersey.spi.container.servlet.ServletContainer. > service(ServletContainer.java:537) > > at com.sun.jersey.spi.container.servlet.ServletContainer. > service(ServletContainer.java:699) > > at javax.servlet.http.HttpServlet.service( > HttpServlet.java:820) > > at org.mortbay.jetty.servlet.ServletHolder.handle( > ServletHolder.java:511) > > at org.mortbay.jetty.servlet.ServletHandler$CachedChain. > doFilter(ServletHandler.java:1221) > > at org.apache.falcon.security.FalconAuthorizationFilter. > doFilter(FalconAuthorizationFilter.java:106) > > at org.mortbay.jetty.servlet.ServletHandler$CachedChain. > doFilter(ServletHandler.java:1212) > > at org.apache.falcon.security. > FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter.java:184) > > at org.apache.hadoop.security.authentication.server. > AuthenticationFilter.doFilter(AuthenticationFilter.java:595) > > at org.apache.hadoop.security.authentication.server. > AuthenticationFilter.doFilter(AuthenticationFilter.java:554) > > at org.apache.falcon.security.FalconAuthenticationFilter. > doFilter(FalconAuthenticationFilter.java:193) > > at org.mortbay.jetty.servlet.ServletHandler$CachedChain. > doFilter(ServletHandler.java:1212) > > at org.apache.falcon.security.FalconAuditFilter.doFilter( > FalconAuditFilter.java:64) > > at org.mortbay.jetty.servlet.ServletHandler$CachedChain. > doFilter(ServletHandler.java:1212) > > at org.mortbay.jetty.servlet.ServletHandler.handle( > ServletHandler.java:399) > > at org.mortbay.jetty.security.SecurityHandler.handle( > SecurityHandler.java:216) > > at org.mortbay.jetty.servlet.SessionHandler.handle( > SessionHandler.java:182) > > at org.mortbay.jetty.handler.ContextHandler.handle( > ContextHandler.java:767) > > at org.mortbay.jetty.webapp.WebAppContext.handle( > WebAppContext.java:450) > > at org.mortbay.jetty.handler.HandlerWrapper.handle( > HandlerWrapper.java:152) > > at org.mortbay.jetty.Server.handle(Server.java:326) > > at org.mortbay.jetty.HttpConnection.handleRequest( > HttpConnection.java:542) > > at org.mortbay.jetty.HttpConnection$RequestHandler. > headerComplete(HttpConnection.java:928) > > at org.mortbay.jetty.HttpParser. > parseNext(HttpParser.java:549) > > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser. > java:212) > > at org.mortbay.jetty.HttpConnection.handle( > HttpConnection.java:404) > > at org.mortbay.jetty.bio.SocketConnector$Connection. > run(SocketConnector.java:228) > > at org.mortbay.thread.QueuedThreadPool$PoolThread. > run(QueuedThreadPool.java:582) > > Caused by: IO_ERROR : java.io.IOException: Error while connecting Oozie > server. No of retries = 1. Exception = null > > at org.apache.oozie.client.OozieClient.validateWSVersion( > OozieClient.java:374) > > at org.apache.oozie.client.OozieClient.createURL( > OozieClient.java:459) > > at org.apache.oozie.client.OozieClient.access$000( > OozieClient.java:80) > > at org.apache.oozie.client.OozieClient$ClientCallable. > call(OozieClient.java:555) > > at org.apache.oozie.client.OozieClient.getBundleJobsInfo( > OozieClient.java:2149) > > at org.apache.oozie.client.ProxyOozieClient.access$2901( > ProxyOozieClient.java:48) > > at org.apache.oozie.client.ProxyOozieClient$29.call( > ProxyOozieClient.java:598) > > at org.apache.oozie.client.ProxyOozieClient$29.call( > ProxyOozieClient.java:596) > > at org.apache.oozie.client.OozieClient.doAs(OozieClient. > java:244) > > at org.apache.oozie.client.ProxyOozieClient. > getBundleJobsInfo(ProxyOozieClient.java:596) > > at org.apache.falcon.workflow.engine.OozieWorkflowEngine. > findBundles(OozieWorkflowEngine.java:290) > > ... 61 more > > Caused by: java.io.IOException: Error while connecting Oozie server. No of > retries = 1. Exception = null > > at org.apache.oozie.client.retry. > ConnectionRetriableClient.execute(ConnectionRetriableClient.java:66) > > at org.apache.oozie.client.OozieClient. > createRetryableConnection(OozieClient.java:504) > > at org.apache.oozie.client.OozieClient. > getSupportedProtocolVersions(OozieClient.java:384) > > at org.apache.oozie.client.OozieClient.validateWSVersion( > OozieClient.java:344) > > ... 71 more > > Caused by: java.lang.reflect.UndeclaredThrowableException > > at org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1672) > > at org.apache.oozie.client.ProxyOozieClient. > createConnection(ProxyOozieClient.java:79) > > at org.apache.oozie.client.OozieClient$1.doExecute( > OozieClient.java:507) > > at org.apache.oozie.client.retry. > ConnectionRetriableClient.execute(ConnectionRetriableClient.java:44) > > ... 74 more > > Caused by: AUTHENTICATION : Could not authenticate, GSSException: No valid > credentials provided (Mechanism level: Failed to find any Kerberos tgt) > > at org.apache.oozie.client.AuthOozieClient. > createConnection(AuthOozieClient.java:136) > > at org.apache.oozie.client.ProxyOozieClient.access$001( > ProxyOozieClient.java:48) > > at org.apache.oozie.client.ProxyOozieClient$1.run( > ProxyOozieClient.java:81) > > at org.apache.oozie.client.ProxyOozieClient$1.run( > ProxyOozieClient.java:79) > > at java.security.AccessController.doPrivileged(Native > Method) > > at javax.security.auth.Subject.doAs(Subject.java:422) > > at org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1657) > > ... 77 more > > Caused by: > org.apache.hadoop.security.authentication.client.AuthenticationException: > GSSException: No valid credentials provided (Mechanism level: Failed to > find any Kerberos tgt) > > at org.apache.hadoop.security.authentication.client. > KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332) > > at org.apache.hadoop.security.authentication.client. > KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205) > > at org.apache.hadoop.security.authentication.client. > AuthenticatedURL.openConnection(AuthenticatedURL.java:215) > > at org.apache.oozie.client.AuthOozieClient. > createConnection(AuthOozieClient.java:127) > > ... 83 more > > Caused by: GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt) > > at sun.security.jgss.krb5.Krb5InitCredential.getInstance( > Krb5InitCredential.java:147) > > at sun.security.jgss.krb5.Krb5MechFactory. > getCredentialElement(Krb5MechFactory.java:122) > > at sun.security.jgss.krb5.Krb5MechFactory. > getMechanismContext(Krb5MechFactory.java:187) > > at sun.security.jgss.GSSManagerImpl.getMechanismContext( > GSSManagerImpl.java:224) > > at sun.security.jgss.GSSContextImpl.initSecContext( > GSSContextImpl.java:212) > > at sun.security.jgss.GSSContextImpl.initSecContext( > GSSContextImpl.java:179) > > at org.apache.hadoop.security.authentication.client. > KerberosAuthenticator$1.run(KerberosAuthenticator.java:311) > > at org.apache.hadoop.security.authentication.client. > KerberosAuthenticator$1.run(KerberosAuthenticator.java:287) > > at java.security.AccessController.doPrivileged(Native > Method) > > at javax.security.auth.Subject.doAs(Subject.java:422) > > at org.apache.hadoop.security.authentication.client. > KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:287) > > ... 86 more > > > > >
