We did a restart by deleting the entire storm local directory folder. The inbox folder is empty. On Fri, May 8, 2015 at 1:48 PM, bhargav sarvepalli <bond1k...@gmail.com> wrote:
> Can you check the contents of inbox folder? and probably clean it > > On Fri, May 8, 2015 at 1:11 PM, aarthi ranganathan <aarthiran...@gmail.com > > wrote: > >> I tried clearing the local state and also restarted storm. Still getting >> Illegal Argument Exception. Since jar is not getting uploaded due to >> illegal argument exception, the topology does not exist in the zookeeper. >> The jar is getting uploaded from tomcat to nimbus inbox. Its not getting >> copied to storm dist folder. Im posting the recent stack trace where look >> into highlighted parts in bold. The jar uploaded starts with <stormjar-e7> >> while the jar that is trying to get transferred is <stormjar-f8>. Is storm >> caching old jars and trying to run? Our team also saw this storm issue >> https://issues.apache.org/jira/browse/STORM-467 which is still >> unresolved where they mention trying to execute non-existent jars as the >> issue. >> >> >> 2015-05-08T11:58:36.122+0530 b.s.d.nimbus [INFO] Uploading file from >> client to >> /home/xbdev/storm/nimbus/inbox/stormjar-e7c13d9a-3aed-4232-9d20-6813e9c45c03.jar >> >> 2015-05-08T11:58:38.546+0530 b.s.d.nimbus [INFO] Finished uploading file >> from client: >> */home/xbdev/storm/nimbus/inbox/stormjar-e7c13d9a-3aed-4232-9d20-6813e9c45c03.jar* >> >> 2015-05-08T11:58:38.710+0530 b.s.d.nimbus [INFO] Received topology >> submission for topology-1431056498295 with conf {"storm.id" >> "topology-1431056498295-1-1431066518", "cassandra-config" {"cassandra.host" >> "10.237.247.13", "cassandra.keyspace" ["stormks"]}, "nimbus.host" >> "10.237.247.3", "storm.zookeeper.port" 2181, "storm.zookeeper.servers" >> ["10.237.247.3"], "topology.acker.executors" nil, >> "topology.kryo.decorators" (), "topology.name" "topology-1431056498295", >> "topology.debug" true, "topology.kryo.register" nil, >> "storm.elasticsearch.hosts" "10.237.247.13:9300", " >> storm.elasticsearch.cluster.name" "HAP_Development_1.0", >> "topology.workers" 2, "nimbus.thrift.port" 6627, "storm.thrift.transport" >> "backtype.storm.security.auth.SimpleTransportPlugin", >> "topology.max.task.parallelism" nil} >> >> 2015-05-08T11:58:38.716+0530 b.s.d.nimbus [WARN] Topology submission >> exception. (topology name='topology-1431056498295') >> #<IllegalArgumentException java.lang.IllegalArgumentException: >> */home/xbdev/storm/nimbus/inbox/stormjar-f8982d19-81d0-420a-a847-7243eb6d4181.jar* >> to copy to >> /home/xbdev/storm/nimbus/stormdist/topology-1431056498295-1-1431066518 does >> not exist!> >> >> 2015-05-08T11:58:38.722+0530 o.a.t.s.TNonblockingServer [ERROR] >> Unexpected exception while invoking! >> >> java.lang.IllegalArgumentException: >> /home/xbdev/storm/nimbus/inbox/stormjar-f8982d19-81d0-420a-a847-7243eb6d4181.jar >> to copy to >> /home/xbdev/storm/nimbus/stormdist/topology-1431056498295-1-1431066518 does >> not exist! >> >> at backtype.storm.daemon.nimbus$fn__4393.invoke(nimbus.clj:1173) >> ~[storm-core-0.9.4.jar:0.9.4] >> >> at clojure.lang.MultiFn.invoke(MultiFn.java:236) ~[clojure-1.5.1.jar:na] >> >> at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307) >> ~[storm-core-0.9.4.jar:0.9.4] >> >> at >> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopologyWithOpts(nimbus.clj:953) >> ~[storm-core-0.9.4.jar:0.9.4] >> >> at >> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopology(nimbus.clj:966) >> ~[storm-core-0.9.4.jar:0.9.4] >> >> at >> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240) >> ~[storm-core-0.9.4.jar:0.9.4] >> >> at >> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228) >> ~[storm-core-0.9.4.jar:0.9.4] >> >> at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32) >> ~[storm-core-0.9.4.jar:0.9.4] >> >> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34) >> ~[storm-core-0.9.4.jar:0.9.4] >> >> at >> org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632) >> ~[storm-core-0.9.4.jar:0.9.4] >> >> at >> org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201) >> [storm-core-0.9.4.jar:0.9.4] >> >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> [na:1.7.0_45] >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> [na:1.7.0_45] >> >> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] >> >> >> On Fri, May 8, 2015 at 11:49 AM, Erik Weathers <eweath...@groupon.com> >> wrote: >> >>> If that doesn't work, then you might need to remove the state in >>> zookeeper related to this topology. >>> i.e., >>> >>> - /storm/storms/<topology-id> >>> - /storm/assignments/<topology-id> >>> >>> - Erik >>> >>> On Thu, May 7, 2015 at 11:18 PM, bhargav sarvepalli <bond1k...@gmail.com >>> > wrote: >>> >>>> Try deleting your local state folder in supervisors and storm nimbus >>>> data folder, looks like its trying to find topology which does not exist. >>>> >>>> On Fri, May 8, 2015 at 11:08 AM, aarthi ranganathan < >>>> aarthiran...@gmail.com> wrote: >>>> >>>>> Hi >>>>> >>>>> While checking nimbus log, Im getting the following stack track >>>>> trace.. The previous stack trace was while running from tomcat application >>>>> , which generates an illegal argument exception in nimbus logs >>>>> >>>>> 2015-05-06T14:27:57.718+0530 o.a.t.s.TNonblockingServer [ERROR] >>>>> Unexpected exception while invoking! >>>>> java.lang.IllegalArgumentException: >>>>> /home/xbdev/storm/nimbus/inbox/stormjar-e9973a4f-7dc2-4cd5-a1ef-4e8c95ec9a34.jar >>>>> to copy to >>>>> /home/xbdev/storm/nimbus/stormdist/topology-1430892623717-3-1430902677 >>>>> does >>>>> not exist! >>>>> at backtype.storm.daemon.nimbus$fn__4393.invoke(nimbus.clj:1173) >>>>> ~[storm-core-0.9.4.jar:0.9.4] >>>>> at clojure.lang.MultiFn.invoke(MultiFn.java:236) >>>>> ~[clojure-1.5.1.jar:na] >>>>> at >>>>> backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307) >>>>> ~[storm-core-0.9.4.jar:0.9.4] >>>>> at >>>>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopologyWithOpts(nimbus.clj:953) >>>>> ~[storm-core-0.9.4.jar:0.9.4] >>>>> at >>>>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopology(nimbus.clj:966) >>>>> ~[storm-core-0.9.4.jar:0.9.4] >>>>> at >>>>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240) >>>>> ~[storm-core-0.9.4.jar:0.9.4] >>>>> at >>>>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228) >>>>> ~[storm-core-0.9.4.jar:0.9.4] >>>>> at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32) >>>>> ~[storm-core-0.9.4.jar:0.9.4] >>>>> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34) >>>>> ~[storm-core-0.9.4.jar:0.9.4] >>>>> at >>>>> org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632) >>>>> ~[storm-core-0.9.4.jar:0.9.4] >>>>> at >>>>> org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201) >>>>> [storm-core-0.9.4.jar:0.9.4] >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>> [na:1.7.0_45] >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>> [na:1.7.0_45] >>>>> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, May 7, 2015 at 5:36 PM, Nishu <nishuta...@gmail.com> wrote: >>>>> >>>>>> You can check if there are free workers available to run the topology >>>>>> or not. >>>>>> >>>>>> Thanks, >>>>>> Nishu Tayal >>>>>> On 7 May 2015 17:08, "bhargav sarvepalli" <bond1k...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Your nimbus has gone down, probably because there is no space left >>>>>>> in your machine. >>>>>>> >>>>>>> On Thu, May 7, 2015 at 3:25 PM, aarthi ranganathan < >>>>>>> aarthiran...@gmail.com> wrote: >>>>>>> >>>>>>>> The zookeeper exists in the same machine along with storm. So I >>>>>>>> guess the ports are accessible. >>>>>>>> >>>>>>>> On Wed, May 6, 2015 at 6:44 PM, Aditya Rajan < >>>>>>>> aditya.ra...@whizdm.com> wrote: >>>>>>>> >>>>>>>>> I would assume it's an issue with zookeeper. Have you checked if >>>>>>>>> the relevant ports are open? >>>>>>>>> On May 6, 2015 6:02 PM, "aarthi ranganathan" < >>>>>>>>> aarthiran...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi >>>>>>>>>> >>>>>>>>>> We are using Storm Submitter via tomcat to submit topology to >>>>>>>>>> storm cluster. The jar does not get uploaded to storm in the same >>>>>>>>>> machine >>>>>>>>>> as well as a different machine. We are using apache-storm >>>>>>>>>> -0.9.4.This error >>>>>>>>>> occurs intermittently. Following is the stack trace >>>>>>>>>> >>>>>>>>>> java.lang.RuntimeException: >>>>>>>>>> org.apache.thrift7.transport.TTransportException >>>>>>>>>> at >>>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:141) >>>>>>>>>> at >>>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70) >>>>>>>>>> at >>>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.submitTopology(StormProcessor.java:148) >>>>>>>>>> at >>>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.access$000(StormProcessor.java:48) >>>>>>>>>> at >>>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor$1.call(StormProcessor.java:111) >>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>>>>>> ... 3 more >>>>>>>>>> Caused by: org.apache.thrift7.transport.TTransportException >>>>>>>>>> at >>>>>>>>>> org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) >>>>>>>>>> at >>>>>>>>>> org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84) >>>>>>>>>> at >>>>>>>>>> org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129) >>>>>>>>>> at >>>>>>>>>> org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101) >>>>>>>>>> at >>>>>>>>>> org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84) >>>>>>>>>> at >>>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) >>>>>>>>>> at >>>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) >>>>>>>>>> at >>>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) >>>>>>>>>> at >>>>>>>>>> org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69) >>>>>>>>>> at >>>>>>>>>> backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:162) >>>>>>>>>> at >>>>>>>>>> backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:146) >>>>>>>>>> at >>>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:127) >>>>>>>>>> ... 8 more >>>>>>>>>> >>>>>>>>>> We are not able to reproduce the issue for a particular scenario. >>>>>>>>>> It occurs randomly. Can u throw some light on the issue? We have >>>>>>>>>> checked >>>>>>>>>> the firewall settings and there is nothing blocked. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> Aarthi R >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Aarthi R >>>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>>>> -- >>>>> Aarthi R >>>>> >>>> >>>> >>> >> >> >> -- >> Aarthi R >> > > -- Aarthi R
