James, feel free to create a jira and provide a patch. Thanks!
On Wed, Aug 13, 2014 at 10:52 AM, James Grant <[email protected]> wrote: > I'm having the same issue and was just about to send a mail when you > posted. > > I've had a look at the code and the problem seems to be when the process > has no inputs. The check in getStorageType used to short circuit if there > were no inputs but that changed as part of the changes in FALCON-443 to > only short circuit if the process has no inputs and no outputs. > > Below is a patch that will prevent the exception (currently untested). > > diff --git > a/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java > b/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java > index 59361e8..33f0aad 100644 > --- a/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java > +++ b/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java > @@ -59,11 +59,13 @@ public final class ProcessHelper { > return storageType; > } > > - for (Input input : process.getInputs().getInputs()) { > - Feed feed = EntityUtil.getEntity(EntityType.FEED, > input.getFeed()); > - storageType = FeedHelper.getStorageType(feed, cluster); > - if (Storage.TYPE.TABLE == storageType) { > - break; > + if (process.getInputs() != null) { > + for (Input input : process.getInputs().getInputs()) { > + Feed feed = EntityUtil.getEntity(EntityType.FEED, > input.getFeed()); > + storageType = FeedHelper.getStorageType(feed, cluster); > + if (Storage.TYPE.TABLE == storageType) { > + break; > + } > } > } > > James > > > On 13 August 2014 17:54, Josh Clum <[email protected]> wrote: > > > Hi, I'm getting the following NPE when i launch with "falcon entity -type > > process -schedule -name raw-cc-bp-ratio-lcms-comp-process". At the bottom > > is my process which has no inputs. > > > > If I look at > > > > > https://github.com/apache/incubator-falcon/blob/branch-0.5/common/src/main/java/org/apache/falcon/entity/ProcessHelper.java > > it seems that this should never happen because process.getInputs() == > > null should > > be true and should avoid the NPE. Any thoughts? > > > > 2014-08-13 16:49:05,152 ERROR - > > [6221571@qtp-1696932027-3 > > > :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process > > a38b5528-cae0-4b07-a8fc-482661e467a9] ~ Unable to schedule workflow > > (AbstractSchedulableEntityManager:60) > > java.lang.NullPointerException > > at > > > > > org.apache.falcon.entity.ProcessHelper.getStorageType(ProcessHelper.java:57) > > at > > > > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.isTableStorageType(OozieProcessWorkflowBuilder.java:646) > > at > > > > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.shouldSetupHiveConfiguration(OozieProcessWorkflowBuilder.java:641) > > at > > > > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createWorkflow(OozieProcessWorkflowBuilder.java:606) > > at > > > > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createDefaultCoordinator(OozieProcessWorkflowBuilder.java:291) > > at > > > > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.getCoordinators(OozieProcessWorkflowBuilder.java:206) > > at > > > > > org.apache.falcon.workflow.OozieWorkflowBuilder.map(OozieWorkflowBuilder.java:142) > > at > > > > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.newWorkflowSchedule(OozieProcessWorkflowBuilder.java:111) > > at > > > > > org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:119) > > at > > > > > org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:68) > > at > > > > > org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:57) > > at > > > > > org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:85) > > at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:48) > > at > > > > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$7.doExecute(SchedulableEntityManagerProxy.java:324) > > at > > > > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:410) > > at > > > > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule_aroundBody10(SchedulableEntityManagerProxy.java:326) > > at > > > > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure11.run(SchedulableEntityManagerProxy.java:1) > > at > > org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) > > at > > > > > org.apache.falcon.aspect.AbstractFalconAspect.logAround(AbstractFalconAspect.java:50) > > at > > > > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule(SchedulableEntityManagerProxy.java:310) > > at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > > > > > com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) > > at > > > > > com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) > > at > > > > > com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) > > at > > > > > com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) > > at > > > > > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > > at > > > > > com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) > > at > > > > > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > > at > > > > > com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) > > at > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) > > at > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) > > at > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) > > at > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) > > at > > > > > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) > > at > > > > > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) > > at > > > > > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > > at > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > > at > > > > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) > > at > > > > > org.apache.falcon.security.BasicAuthFilter$2.doFilter(BasicAuthFilter.java:183) > > at > > > > > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:392) > > at > > > > > org.apache.falcon.security.BasicAuthFilter.doFilter(BasicAuthFilter.java:221) > > at > > > > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > > at > > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > > at > > > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > > at > > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > > at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) > > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > > at org.mortbay.jetty.Server.handle(Server.java:326) > > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > > at > > > > > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) > > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) > > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) > > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > > at > > > > > org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) > > at > > > > > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > > 2014-08-13 16:49:05,154 ERROR - > > [6221571@qtp-1696932027-3 > > > :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process > > a38b5528-cae0-4b07-a8fc-482661e467a9] ~ Failure reason > > (FalconWebException:40) > > java.lang.NullPointerException > > at > > > > > org.apache.falcon.entity.ProcessHelper.getStorageType(ProcessHelper.java:57) > > at > > > > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.isTableStorageType(OozieProcessWorkflowBuilder.java:646) > > at > > > > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.shouldSetupHiveConfiguration(OozieProcessWorkflowBuilder.java:641) > > at > > > > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createWorkflow(OozieProcessWorkflowBuilder.java:606) > > at > > > > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.createDefaultCoordinator(OozieProcessWorkflowBuilder.java:291) > > at > > > > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.getCoordinators(OozieProcessWorkflowBuilder.java:206) > > at > > > > > org.apache.falcon.workflow.OozieWorkflowBuilder.map(OozieWorkflowBuilder.java:142) > > at > > > > > org.apache.falcon.workflow.OozieProcessWorkflowBuilder.newWorkflowSchedule(OozieProcessWorkflowBuilder.java:111) > > at > > > > > org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:119) > > at > > > > > org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:68) > > at > > > > > org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:57) > > at > > > > > org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:85) > > at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:48) > > at > > > > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$7.doExecute(SchedulableEntityManagerProxy.java:324) > > at > > > > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:410) > > at > > > > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule_aroundBody10(SchedulableEntityManagerProxy.java:326) > > at > > > > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure11.run(SchedulableEntityManagerProxy.java:1) > > at > > org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) > > at > > > > > org.apache.falcon.aspect.AbstractFalconAspect.logAround(AbstractFalconAspect.java:50) > > at > > > > > org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule(SchedulableEntityManagerProxy.java:310) > > at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > > > > > com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) > > at > > > > > com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) > > at > > > > > com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) > > at > > > > > com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) > > at > > > > > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > > at > > > > > com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) > > at > > > > > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > > at > > > > > com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) > > at > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) > > at > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) > > at > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) > > at > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) > > at > > > > > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) > > at > > > > > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) > > at > > > > > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > > at > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > > at > > > > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) > > at > > > > > org.apache.falcon.security.BasicAuthFilter$2.doFilter(BasicAuthFilter.java:183) > > at > > > > > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:392) > > at > > > > > org.apache.falcon.security.BasicAuthFilter.doFilter(BasicAuthFilter.java:221) > > at > > > > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > > at > > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > > at > > > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > > at > > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > > at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) > > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > > at org.mortbay.jetty.Server.handle(Server.java:326) > > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > > at > > > > > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) > > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) > > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) > > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > > at > > > > > org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) > > at > > > > > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > > 2014-08-13 16:49:05,154 ERROR - > > [6221571@qtp-1696932027-3 > > > :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process > > a38b5528-cae0-4b07-a8fc-482661e467a9] ~ Action failed: Bad Request > > Error:null > > (FalconWebException:66) > > 2014-08-13 16:49:05,155 INFO - > > [6221571@qtp-1696932027-3 > > > :vagrant:POST//entities/schedule/process/raw-cc-bp-ratio-lcms-comp-process > > a38b5528-cae0-4b07-a8fc-482661e467a9] ~ {Action:schedule, > > Dimensions:{entityType=process, colo=*, > > entityName=raw-cc-bp-ratio-lcms-comp-process}, Status: FAILED, > > Time-taken:161608565 ns} (METRIC:32) > > > > My process looks like this: > > > > <process name="raw-cc-bp-ratio-lcms-comp-process" > > xmlns="uri:falcon:process:0.1"> > > > > <tags>type=raw</tags> > > > > <clusters> > > <cluster name="minicluster"> > > <validity start="2014-08-13T00:00Z" end="2016-01-01T00:00Z"/> > > </cluster> > > </clusters> > > > > <parallel>1</parallel> > > <order>FIFO</order> > > <timeout>hours(2)</timeout> > > <frequency>days(1)</frequency> > > > > <outputs> > > <output name="output" feed="raw-cc-bp-ratio-lcms-comp-feed" > > instance="now(0,0)" /> > > </outputs> > > > > <properties> > > ... > > </properties> > > > > <workflow name="sqoop_wf" version="2.0.0" engine="oozie" > > path="/apps/huron/ingest/standard_ingest/sqoop_wf.xml" /> > > > > <retry policy="periodic" delay="minutes(10)" attempts="3" /> > > > > </process> > > > -- Regards, Venkatesh “Perfection (in design) is achieved not when there is nothing more to add, but rather when there is nothing more to take away.” - Antoine de Saint-Exupéry
