Can you check the contents of inbox folder? and probably clean it On Fri, May 8, 2015 at 1:11 PM, aarthi ranganathan <aarthiran...@gmail.com> wrote:
> I tried clearing the local state and also restarted storm. Still getting > Illegal Argument Exception. Since jar is not getting uploaded due to > illegal argument exception, the topology does not exist in the zookeeper. > The jar is getting uploaded from tomcat to nimbus inbox. Its not getting > copied to storm dist folder. Im posting the recent stack trace where look > into highlighted parts in bold. The jar uploaded starts with <stormjar-e7> > while the jar that is trying to get transferred is <stormjar-f8>. Is storm > caching old jars and trying to run? Our team also saw this storm issue > https://issues.apache.org/jira/browse/STORM-467 which is still unresolved > where they mention trying to execute non-existent jars as the issue. > > > 2015-05-08T11:58:36.122+0530 b.s.d.nimbus [INFO] Uploading file from > client to > /home/xbdev/storm/nimbus/inbox/stormjar-e7c13d9a-3aed-4232-9d20-6813e9c45c03.jar > > 2015-05-08T11:58:38.546+0530 b.s.d.nimbus [INFO] Finished uploading file > from client: > */home/xbdev/storm/nimbus/inbox/stormjar-e7c13d9a-3aed-4232-9d20-6813e9c45c03.jar* > > 2015-05-08T11:58:38.710+0530 b.s.d.nimbus [INFO] Received topology > submission for topology-1431056498295 with conf {"storm.id" > "topology-1431056498295-1-1431066518", "cassandra-config" {"cassandra.host" > "10.237.247.13", "cassandra.keyspace" ["stormks"]}, "nimbus.host" > "10.237.247.3", "storm.zookeeper.port" 2181, "storm.zookeeper.servers" > ["10.237.247.3"], "topology.acker.executors" nil, > "topology.kryo.decorators" (), "topology.name" "topology-1431056498295", > "topology.debug" true, "topology.kryo.register" nil, > "storm.elasticsearch.hosts" "10.237.247.13:9300", " > storm.elasticsearch.cluster.name" "HAP_Development_1.0", > "topology.workers" 2, "nimbus.thrift.port" 6627, "storm.thrift.transport" > "backtype.storm.security.auth.SimpleTransportPlugin", > "topology.max.task.parallelism" nil} > > 2015-05-08T11:58:38.716+0530 b.s.d.nimbus [WARN] Topology submission > exception. (topology name='topology-1431056498295') > #<IllegalArgumentException java.lang.IllegalArgumentException: > */home/xbdev/storm/nimbus/inbox/stormjar-f8982d19-81d0-420a-a847-7243eb6d4181.jar* > to copy to > /home/xbdev/storm/nimbus/stormdist/topology-1431056498295-1-1431066518 does > not exist!> > > 2015-05-08T11:58:38.722+0530 o.a.t.s.TNonblockingServer [ERROR] Unexpected > exception while invoking! > > java.lang.IllegalArgumentException: > /home/xbdev/storm/nimbus/inbox/stormjar-f8982d19-81d0-420a-a847-7243eb6d4181.jar > to copy to > /home/xbdev/storm/nimbus/stormdist/topology-1431056498295-1-1431066518 does > not exist! > > at backtype.storm.daemon.nimbus$fn__4393.invoke(nimbus.clj:1173) > ~[storm-core-0.9.4.jar:0.9.4] > > at clojure.lang.MultiFn.invoke(MultiFn.java:236) ~[clojure-1.5.1.jar:na] > > at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307) > ~[storm-core-0.9.4.jar:0.9.4] > > at > backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopologyWithOpts(nimbus.clj:953) > ~[storm-core-0.9.4.jar:0.9.4] > > at > backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopology(nimbus.clj:966) > ~[storm-core-0.9.4.jar:0.9.4] > > at > backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240) > ~[storm-core-0.9.4.jar:0.9.4] > > at > backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228) > ~[storm-core-0.9.4.jar:0.9.4] > > at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32) > ~[storm-core-0.9.4.jar:0.9.4] > > at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34) > ~[storm-core-0.9.4.jar:0.9.4] > > at > org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632) > ~[storm-core-0.9.4.jar:0.9.4] > > at > org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201) > [storm-core-0.9.4.jar:0.9.4] > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_45] > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_45] > > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > > > On Fri, May 8, 2015 at 11:49 AM, Erik Weathers <eweath...@groupon.com> > wrote: > >> If that doesn't work, then you might need to remove the state in >> zookeeper related to this topology. >> i.e., >> >> - /storm/storms/<topology-id> >> - /storm/assignments/<topology-id> >> >> - Erik >> >> On Thu, May 7, 2015 at 11:18 PM, bhargav sarvepalli <bond1k...@gmail.com> >> wrote: >> >>> Try deleting your local state folder in supervisors and storm nimbus >>> data folder, looks like its trying to find topology which does not exist. >>> >>> On Fri, May 8, 2015 at 11:08 AM, aarthi ranganathan < >>> aarthiran...@gmail.com> wrote: >>> >>>> Hi >>>> >>>> While checking nimbus log, Im getting the following stack track trace.. >>>> The previous stack trace was while running from tomcat application , which >>>> generates an illegal argument exception in nimbus logs >>>> >>>> 2015-05-06T14:27:57.718+0530 o.a.t.s.TNonblockingServer [ERROR] >>>> Unexpected exception while invoking! >>>> java.lang.IllegalArgumentException: >>>> /home/xbdev/storm/nimbus/inbox/stormjar-e9973a4f-7dc2-4cd5-a1ef-4e8c95ec9a34.jar >>>> to copy to >>>> /home/xbdev/storm/nimbus/stormdist/topology-1430892623717-3-1430902677 does >>>> not exist! >>>> at backtype.storm.daemon.nimbus$fn__4393.invoke(nimbus.clj:1173) >>>> ~[storm-core-0.9.4.jar:0.9.4] >>>> at clojure.lang.MultiFn.invoke(MultiFn.java:236) ~[clojure-1.5.1.jar:na] >>>> at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307) >>>> ~[storm-core-0.9.4.jar:0.9.4] >>>> at >>>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopologyWithOpts(nimbus.clj:953) >>>> ~[storm-core-0.9.4.jar:0.9.4] >>>> at >>>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopology(nimbus.clj:966) >>>> ~[storm-core-0.9.4.jar:0.9.4] >>>> at >>>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240) >>>> ~[storm-core-0.9.4.jar:0.9.4] >>>> at >>>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228) >>>> ~[storm-core-0.9.4.jar:0.9.4] >>>> at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32) >>>> ~[storm-core-0.9.4.jar:0.9.4] >>>> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34) >>>> ~[storm-core-0.9.4.jar:0.9.4] >>>> at >>>> org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632) >>>> ~[storm-core-0.9.4.jar:0.9.4] >>>> at >>>> org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201) >>>> [storm-core-0.9.4.jar:0.9.4] >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> [na:1.7.0_45] >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> [na:1.7.0_45] >>>> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] >>>> >>>> >>>> >>>> >>>> On Thu, May 7, 2015 at 5:36 PM, Nishu <nishuta...@gmail.com> wrote: >>>> >>>>> You can check if there are free workers available to run the topology >>>>> or not. >>>>> >>>>> Thanks, >>>>> Nishu Tayal >>>>> On 7 May 2015 17:08, "bhargav sarvepalli" <bond1k...@gmail.com> wrote: >>>>> >>>>>> Your nimbus has gone down, probably because there is no space left in >>>>>> your machine. >>>>>> >>>>>> On Thu, May 7, 2015 at 3:25 PM, aarthi ranganathan < >>>>>> aarthiran...@gmail.com> wrote: >>>>>> >>>>>>> The zookeeper exists in the same machine along with storm. So I >>>>>>> guess the ports are accessible. >>>>>>> >>>>>>> On Wed, May 6, 2015 at 6:44 PM, Aditya Rajan < >>>>>>> aditya.ra...@whizdm.com> wrote: >>>>>>> >>>>>>>> I would assume it's an issue with zookeeper. Have you checked if >>>>>>>> the relevant ports are open? >>>>>>>> On May 6, 2015 6:02 PM, "aarthi ranganathan" < >>>>>>>> aarthiran...@gmail.com> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> Hi >>>>>>>>> >>>>>>>>> We are using Storm Submitter via tomcat to submit topology to >>>>>>>>> storm cluster. The jar does not get uploaded to storm in the same >>>>>>>>> machine >>>>>>>>> as well as a different machine. We are using apache-storm -0.9.4.This >>>>>>>>> error >>>>>>>>> occurs intermittently. Following is the stack trace >>>>>>>>> >>>>>>>>> java.lang.RuntimeException: >>>>>>>>> org.apache.thrift7.transport.TTransportException >>>>>>>>> at >>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:141) >>>>>>>>> at >>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70) >>>>>>>>> at >>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.submitTopology(StormProcessor.java:148) >>>>>>>>> at >>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.access$000(StormProcessor.java:48) >>>>>>>>> at >>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor$1.call(StormProcessor.java:111) >>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>>>>> ... 3 more >>>>>>>>> Caused by: org.apache.thrift7.transport.TTransportException >>>>>>>>> at >>>>>>>>> org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) >>>>>>>>> at >>>>>>>>> org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84) >>>>>>>>> at >>>>>>>>> org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129) >>>>>>>>> at >>>>>>>>> org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101) >>>>>>>>> at >>>>>>>>> org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84) >>>>>>>>> at >>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) >>>>>>>>> at >>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) >>>>>>>>> at >>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) >>>>>>>>> at >>>>>>>>> org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69) >>>>>>>>> at >>>>>>>>> backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:162) >>>>>>>>> at >>>>>>>>> backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:146) >>>>>>>>> at >>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:127) >>>>>>>>> ... 8 more >>>>>>>>> >>>>>>>>> We are not able to reproduce the issue for a particular scenario. >>>>>>>>> It occurs randomly. Can u throw some light on the issue? We have >>>>>>>>> checked >>>>>>>>> the firewall settings and there is nothing blocked. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Aarthi R >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Aarthi R >>>>>>> >>>>>> >>>>>> >>>> >>>> >>>> -- >>>> Aarthi R >>>> >>> >>> >> > > > -- > Aarthi R >
