Hello Venkat, HDP 2.3.3 release notes says that patch has been applied for FALCON-954 (which is the exact issue as we have). So looks like it is something else.
Regards, Vishal. On Thu, Dec 22, 2016 at 2:11 PM, Vishal Gupta <[email protected]> wrote: > sorry typo, we are on 0.6.*1* > > Regards, > Vishal. > > On Thu, Dec 22, 2016 at 2:09 PM, Vishal Gupta <[email protected]> wrote: > >> Thanks a lot Venkat for quick help. >> >> I will look into it. We will contact Hortonworks for patch or check the >> possibility of upgrade. we are on 0.6.0 version as of now. >> >> Regards >> Vishal. >> >> >> On Thu, Dec 22, 2016 at 1:58 PM, Venkat Ranganathan < >> [email protected]> wrote: >> >>> This is addressed as part of FALCON-2025. It is fixed in 0.10 release >>> >>> >>> >>> Thanks >>> >>> Venkat >>> >>> >>> >>> *From: *Vishal Gupta <[email protected]> >>> *Date: *Wednesday, December 21, 2016 at 9:37 PM >>> *To: *"[email protected]" <[email protected]>, Venkat >>> Ranganathan <[email protected]> >>> >>> *Subject: *Re: Falcon issue after Kerberos implementation >>> >>> >>> >>> Hello Venkat, >>> >>> >>> >>> Thanks for quick response. Could you please guide to any contacts in HWX >>> or any online server where we can find the patch / answer. >>> >>> >>> >>> Do you know in which version this issue is being resolved. >>> >>> >>> >>> Thanks in advance. >>> >>> >>> >>> Regards, >>> >>> Vishal. >>> >>> >>> >>> >>> >>> On Thu, Dec 22, 2016 at 1:34 PM, Venkat Ranganathan < >>> [email protected]> wrote: >>> >>> > Caused by: AUTHENTICATION : Could not authenticate, GSSException: No >>> valid credentials provided (Mechanism level: Failed to find any Kerberos >>> tgt) >>> >>> This shows that the Falcon kerberos context become invalidated. There >>> was a fix in later versions and patch should be available from HWX support >>> for the older versions >>> >>> Thanks >>> Venkat >>> ________________________________________ >>> From: Vishal Gupta <[email protected]> >>> Sent: Wednesday, December 21, 2016 9:19 PM >>> To: [email protected] >>> Subject: Re: Falcon issue after Kerberos implementation >>> >>> >>> Cluster is HDP 2.3.2 >>> >>> On Thu, Dec 22, 2016 at 1:17 PM, Vishal Gupta <[email protected]> >>> wrote: >>> >>> > Hello Team, >>> > We are implementing Kerberos at one the clusters. >>> > >>> > After Kerberos implementation looks like Falcon while using SPNEGO to >>> > authenticate the user, is failing while authenticating and connecting >>> to >>> > Oozie. >>> > >>> > >>> > >>> > Any suggestions please. Details below.. >>> > >>> > >>> > 2016-12-21 07:52:42,597 ERROR - [1529601858@qtp-708533063-243 - >>> > 737162dd-6235-494c-a0b4-c8e132fc5491:ifrsdev:DELETE// >>> > entities/delete/process/edmhdpif-oozie-icbdev-th] ~ Unable to reach >>> > workflow engine for deletion or deletion failed >>> (AbstractEntityManager:265) >>> > >>> > org.apache.falcon.FalconException: IO_ERROR : java.io.IOException: >>> Error >>> > while connecting Oozie server. No of retries = 1. Exception = null >>> > >>> > at org.apache.falcon.workflow.eng >>> ine.OozieWorkflowEngine. >>> > findBundles(OozieWorkflowEngine.java:303) >>> > >>> > at org.apache.falcon.workflow.eng >>> ine.OozieWorkflowEngine. >>> > doBundleAction(OozieWorkflowEngine.java:377) >>> > >>> > at org.apache.falcon.workflow.eng >>> ine.OozieWorkflowEngine. >>> > doBundleAction(OozieWorkflowEngine.java:371) >>> > >>> > at org.apache.falcon.workflow.eng >>> ine.OozieWorkflowEngine. >>> > delete(OozieWorkflowEngine.java:355) >>> > >>> > at org.apache.falcon.resource. >>> > AbstractEntityManager.delete(AbstractEntityManager.java:252) >>> > >>> > at org.apache.falcon.resource.Con >>> figSyncService.delete( >>> > ConfigSyncService.java:62) >>> > >>> > at sun.reflect.GeneratedMethodAcc >>> essor88.invoke(Unknown >>> > Source) >>> > >>> > at sun.reflect.DelegatingMethodAccessorImpl.invoke( >>> > DelegatingMethodAccessorImpl.java:43) >>> > >>> > at java.lang.reflect.Method.invoke(Method.java:498) >>> > >>> > at org.apache.falcon.resource.cha >>> nnel.IPCChannel.invoke( >>> > IPCChannel.java:49) >>> > >>> > at org.apache.falcon.resource.proxy. >>> > SchedulableEntityManagerProxy$3.doExecute(SchedulableEntityM >>> anagerProxy. >>> > java:230) >>> > >>> > at org.apache.falcon.resource.proxy. >>> > SchedulableEntityManagerProxy$EntityProxy.execute( >>> > SchedulableEntityManagerProxy.java:575) >>> > >>> > at org.apache.falcon.resource.proxy. >>> > SchedulableEntityManagerProxy$3.execute(SchedulableEntityManagerProxy. >>> > java:219) >>> > >>> > at org.apache.falcon.resource.proxy. >>> > SchedulableEntityManagerProxy.delete_aroundBody2( >>> > SchedulableEntityManagerProxy.java:232) >>> > >>> > at org.apache.falcon.resource.proxy. >>> > SchedulableEntityManagerProxy$AjcClosure3.run( >>> > SchedulableEntityManagerProxy.java:1) >>> > >>> > at org.aspectj.runtime.reflect.JoinPointImpl.proceed( >>> > JoinPointImpl.java:149) >>> > >>> > at org.apache.falcon.aspect.AbstractFalconAspect. >>> > logAroundMonitored(AbstractFalconAspect.java:51) >>> > >>> > at org.apache.falcon.resource.proxy. >>> > SchedulableEntityManagerProxy.delete(SchedulableEntityManagerProxy. >>> > java:206) >>> > >>> > at sun.reflect.GeneratedMethodAcc >>> essor87.invoke(Unknown >>> > Source) >>> > >>> > at sun.reflect.DelegatingMethodAccessorImpl.invoke( >>> > DelegatingMethodAccessorImpl.java:43) >>> > >>> > at java.lang.reflect.Method.invoke(Method.java:498) >>> > >>> >>> > at com.sun.jersey.spi.container. >>> > JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) >>> > >>> > at com.sun.jersey.server.impl.model.method.dispatch. >>> > AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch( >>> > AbstractResourceMethodDispatchProvider.java:185) >>> > >>> > at com.sun.jersey.server.impl.model.method.dispatch. >>> > ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher. >>> > java:75) >>> > >>> > at com.sun.jersey.server.impl.uri >>> .rules.HttpMethodRule. >>> > accept(HttpMethodRule.java:288) >>> > >>> > at com.sun.jersey.server.impl.uri >>> .rules.RightHandPathRule. >>> > accept(RightHandPathRule.java:147) >>> > >>> > at com.sun.jersey.server.impl.uri >>> .rules.ResourceClassRule. >>> > accept(ResourceClassRule.java:108) >>> > >>> > at com.sun.jersey.server.impl.uri >>> .rules.RightHandPathRule. >>> > accept(RightHandPathRule.java:147) >>> > >>> > at com.sun.jersey.server.impl.uri.rules. >>> > RootResourceClassesRule.accept(RootResourceClassesRule.java:84) >>> > >>> > at com.sun.jersey.server.impl.application. >>> > WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) >>> > >>> > at com.sun.jersey.server.impl.application. >>> > WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) >>> > >>> > at com.sun.jersey.server.impl.application. >>> > WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) >>> > >>> > at com.sun.jersey.server.impl.application. >>> > WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) >>> > >>> > at com.sun.jersey.spi.container. >>> > servlet.WebComponent.service(WebComponent.java:416) >>> > >>> > at com.sun.jersey.spi.container.s >>> ervlet.ServletContainer. >>> > service(ServletContainer.java:537) >>> > >>> > at com.sun.jersey.spi.container.s >>> ervlet.ServletContainer. >>> > service(ServletContainer.java:699) >>> > >>> > at javax.servlet.http.HttpServlet.service( >>> > HttpServlet.java:820) >>> > >>> > at org.mortbay.jetty.servlet.ServletHolder.handle( >>> > ServletHolder.java:511) >>> > >>> > at org.mortbay.jetty.servlet.Serv >>> letHandler$CachedChain. >>> > doFilter(ServletHandler.java:1221) >>> > >>> > at org.apache.falcon.security.Fal >>> conAuthorizationFilter. >>> > doFilter(FalconAuthorizationFilter.java:106) >>> > >>> > at org.mortbay.jetty.servlet.Serv >>> letHandler$CachedChain. >>> > doFilter(ServletHandler.java:1212) >>> > >>> > at org.apache.falcon.security. >>> > FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFi >>> lter.java:184) >>> > >>> > at org.apache.hadoop.security.authentication.server. >>> > AuthenticationFilter.doFilter(AuthenticationFilter.java:595) >>> > >>> > at org.apache.hadoop.security.authentication.server. >>> > AuthenticationFilter.doFilter(AuthenticationFilter.java:554) >>> > >>> > at org.apache.falcon.security.Fal >>> conAuthenticationFilter. >>> > doFilter(FalconAuthenticationFilter.java:193) >>> > >>> > at org.mortbay.jetty.servlet.Serv >>> letHandler$CachedChain. >>> > doFilter(ServletHandler.java:1212) >>> > >>> > at org.apache.falcon.security.Fal >>> conAuditFilter.doFilter( >>> > FalconAuditFilter.java:64) >>> > >>> > at org.mortbay.jetty.servlet.Serv >>> letHandler$CachedChain. >>> > doFilter(ServletHandler.java:1212) >>> > >>> > at org.mortbay.jetty.servlet.ServletHandler.handle( >>> > ServletHandler.java:399) >>> > >>> > at org.mortbay.jetty.security.SecurityHandler.handle( >>> > SecurityHandler.java:216) >>> > >>> > at org.mortbay.jetty.servlet.SessionHandler.handle( >>> > SessionHandler.java:182) >>> > >>> > at org.mortbay.jetty.handler.ContextHandler.handle( >>> > ContextHandler.java:767) >>> > >>> > at org.mortbay.jetty.webapp.WebAppContext.handle( >>> > WebAppContext.java:450) >>> > >>> > at org.mortbay.jetty.handler.HandlerWrapper.handle( >>> >>> > HandlerWrapper.java:152) >>> > >>> > at org.mortbay.jetty.Server.handle(Server.java:326) >>> > >>> > at org.mortbay.jetty.HttpConnection.handleRequest( >>> > HttpConnection.java:542) >>> > >>> > at org.mortbay.jetty.HttpConnection$RequestHandler. >>> > headerComplete(HttpConnection.java:928) >>> > >>> > at org.mortbay.jetty.HttpParser. >>> > parseNext(HttpParser.java:549) >>> > >>> > at org.mortbay.jetty.HttpParser.p >>> arseAvailable(HttpParser. >>> > java:212) >>> > >>> > at org.mortbay.jetty.HttpConnection.handle( >>> > HttpConnection.java:404) >>> > >>> > at org.mortbay.jetty.bio.SocketConnector$Connection. >>> > run(SocketConnector.java:228) >>> > >>> > at org.mortbay.thread.QueuedThreadPool$PoolThread. >>> > run(QueuedThreadPool.java:582) >>> > >>> > Caused by: IO_ERROR : java.io.IOException: Error while connecting Oozie >>> > server. No of retries = 1. Exception = null >>> > >>> > at org.apache.oozie.client.OozieC >>> lient.validateWSVersion( >>> > OozieClient.java:374) >>> > >>> > at org.apache.oozie.client.OozieClient.createURL( >>> > OozieClient.java:459) >>> > >>> > at org.apache.oozie.client.OozieClient.access$000( >>> > OozieClient.java:80) >>> > >>> > at org.apache.oozie.client.OozieClient$ClientCallable. >>> > call(OozieClient.java:555) >>> > >>> > at org.apache.oozie.client.OozieC >>> lient.getBundleJobsInfo( >>> > OozieClient.java:2149) >>> > >>> > at org.apache.oozie.client.ProxyO >>> ozieClient.access$2901( >>> > ProxyOozieClient.java:48) >>> > >>> > at org.apache.oozie.client.ProxyOozieClient$29.call( >>> > ProxyOozieClient.java:598) >>> > >>> > at org.apache.oozie.client.ProxyOozieClient$29.call( >>> > ProxyOozieClient.java:596) >>> > >>> > at org.apache.oozie.client.OozieC >>> lient.doAs(OozieClient. >>> > java:244) >>> > >>> > at org.apache.oozie.client.ProxyOozieClient. >>> > getBundleJobsInfo(ProxyOozieClient.java:596) >>> > >>> > at org.apache.falcon.workflow.eng >>> ine.OozieWorkflowEngine. >>> > findBundles(OozieWorkflowEngine.java:290) >>> > >>> > ... 61 more >>> > >>> > Caused by: java.io.IOException: Error while connecting Oozie server. >>> No of >>> > retries = 1. Exception = null >>> > >>> > at org.apache.oozie.client.retry. >>> > ConnectionRetriableClient.execute(ConnectionRetriableClient.java:66) >>> > >>> > at org.apache.oozie.client.OozieClient. >>> > createRetryableConnection(OozieClient.java:504) >>> > >>> > at org.apache.oozie.client.OozieClient. >>> > getSupportedProtocolVersions(OozieClient.java:384) >>> > >>> > at org.apache.oozie.client.OozieC >>> lient.validateWSVersion( >>> > OozieClient.java:344) >>> > >>> > ... 71 more >>> > >>> > Caused by: java.lang.reflect.UndeclaredThrowableException >>> > >>> > at org.apache.hadoop.security.Use >>> rGroupInformation.doAs( >>> > UserGroupInformation.java:1672) >>> > >>> > at org.apache.oozie.client.ProxyOozieClient. >>> > createConnection(ProxyOozieClient.java:79) >>> > >>> > at org.apache.oozie.client.OozieClient$1.doExecute( >>> > OozieClient.java:507) >>> > >>> > at org.apache.oozie.client.retry. >>> > ConnectionRetriableClient.execute(ConnectionRetriableClient.java:44) >>> > >>> > ... 74 more >>> > >>> > Caused by: AUTHENTICATION : Could not authenticate, GSSException: No >>> valid >>> > credentials provided (Mechanism level: Failed to find any Kerberos tgt) >>> > >>> > at org.apache.oozie.client.AuthOozieClient. >>> > createConnection(AuthOozieClient.java:136) >>> > >>> > at org.apache.oozie.client.ProxyO >>> ozieClient.access$001( >>> > ProxyOozieClient.java:48) >>> > >>> > at org.apache.oozie.client.ProxyOozieClient$1.run( >>> > ProxyOozieClient.java:81) >>> > >>> > at org.apache.oozie.client.ProxyOozieClient$1.run( >>> > ProxyOozieClient.java:79) >>> > >>> > at java.security.AccessController.doPrivileged(Native >>> > Method) >>> > >>> > at javax.security.auth.Subject.doAs(Subject.java:422) >>> > >>> > at org.apache.hadoop.security.Use >>> rGroupInformation.doAs( >>> > UserGroupInformation.java:1657) >>> > >>> > ... 77 more >>> > >>> > Caused by: org.apache.hadoop.security.aut >>> hentication.client.AuthenticationException: >>> > GSSException: No valid credentials provided (Mechanism level: Failed to >>> > find any Kerberos tgt) >>> > >>> > at org.apache.hadoop.security.authentication.client. >>> > KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332) >>> > >>> > at org.apache.hadoop.security.authentication.client. >>> > KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205) >>> > >>> > at org.apache.hadoop.security.authentication.client. >>> > AuthenticatedURL.openConnection(AuthenticatedURL.java:215) >>> > >>> > at org.apache.oozie.client.AuthOozieClient. >>> > createConnection(AuthOozieClient.java:127) >>> > >>> > ... 83 more >>> > >>> > Caused by: GSSException: No valid credentials provided (Mechanism >>> level: >>> > Failed to find any Kerberos tgt) >>> > >>> > at sun.security.jgss.krb5.Krb5Ini >>> tCredential.getInstance( >>> > Krb5InitCredential.java:147) >>> > >>> > at sun.security.jgss.krb5.Krb5MechFactory. >>> > getCredentialElement(Krb5MechFactory.java:122) >>> > >>> > at sun.security.jgss.krb5.Krb5MechFactory. >>> > getMechanismContext(Krb5MechFactory.java:187) >>> > >>> > at sun.security.jgss.GSSManagerIm >>> pl.getMechanismContext( >>> > GSSManagerImpl.java:224) >>> > >>> > at sun.security.jgss.GSSContextImpl.initSecContext( >>> > GSSContextImpl.java:212) >>> > >>> > at sun.security.jgss.GSSContextImpl.initSecContext( >>> > GSSContextImpl.java:179) >>> > >>> > at org.apache.hadoop.security.authentication.client. >>> > KerberosAuthenticator$1.run(KerberosAuthenticator.java:311) >>> > >>> > at org.apache.hadoop.security.authentication.client. >>> > KerberosAuthenticator$1.run(KerberosAuthenticator.java:287) >>> > >>> > at java.security.AccessController.doPrivileged(Native >>> > Method) >>> > >>> > at javax.security.auth.Subject.doAs(Subject.java:422) >>> > >>> > at org.apache.hadoop.security.authentication.client. >>> > KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:287) >>> > >>> > ... 86 more >>> > >>> > >>> > >>> > >>> > >>> >>> >>> >> >> >
