I stuck there for 2 weeks. :) Basically, I tried to debug every command that storm processes are trying to run. You can manually run the worker java command on console to check what error it is throwing. In our case there were some problems, few of them are:
1) slf4j jar conflicts 2) somehow wrong value of hostname in storm worker's configuration 3) something wrong in bolt's prepare method or spout's open method (may be waiting for some connection outside of storm) 4) ports being blocked by firewall (i added few ports in Amazon EC2 security groups to make it runnable) On Tue, Sep 9, 2014 at 12:48 PM, Pavan Jakati G <pava...@microland.com> wrote: > How did you fix it Vikas . Can you help us here . We are stuck here since > a week . L > > > > *From:* Vikas Agarwal [mailto:vi...@infoobjects.com] > *Sent:* 09 September 2014 12:39 > *To:* user@storm.incubator.apache.org > > *Subject:* Re: Running Topology on Remote Machine > > > > We faced similar errors. > > > > May be it can help > http://mail-archives.apache.org/mod_mbox/incubator-storm-user/201408.mbox/browser > > > > On Tue, Sep 9, 2014 at 11:02 AM, Pavan Jakati G <pava...@microland.com> > wrote: > > Worker log > > > > 2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt __system:(-1) > > 2014-09-08 07:19:09 b.s.d.executor [INFO] Loading executor SOIMbolt:[1 1] > > 2014-09-08 07:19:09 b.s.d.executor [INFO] Loaded executor tasks > SOIMbolt:[1 1] > > 2014-09-08 07:19:09 b.s.d.executor [INFO] Finished loading executor > SOIMbolt:[1 1] > > 2014-09-08 07:19:09 b.s.d.executor [INFO] Preparing bolt SOIMbolt:(1) > > 2014-09-08 07:19:09 b.s.d.worker [INFO] Launching receive-thread for > 51fb6b55-0522-437c-a18e-72212192db0f:6703 > > 2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt SOIMbolt:(1) > > 2014-09-08 07:19:09 b.s.m.n.Server [INFO] Create Netty Server > Netty-server-localhost-6703, buffer_size: 5242880, maxWorkers: 1 > > 2014-09-08 07:19:09 b.s.m.loader [INFO] Starting receive-thread: [stormId: > PaVan-1-1410160741, port: 6703, thread-id: 0 ] > > 2014-09-08 07:19:09 b.s.d.worker [INFO] Worker has topology config {" > storm.id" "PaVan-1-1410160741", "dev.zookeeper.path" > "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, > "topology.builtin.metrics.bucket.size.secs" 60, > "topology.fall.back.on.java.serialization" true, > "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, > "topology.skip.missing.kryo.registrations" false, > "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", > "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, > "topology.trident.batch.emit.interval.millis" 500, "drpc.servers" > ["192.168.10.7"], "storm.messaging.netty.flush.check.interval.ms" 10, > "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", > "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", > "topology.executor.send.buffer.size" 1024, "storm.local.dir" "storm-local", > "storm.messaging.netty.buffer_size" 5242880, > "supervisor.worker.start.timeout.secs" 120, > "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" > 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, > "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "192.168.10.7", > "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, > "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" > 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" > "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, > "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, > "storm.zookeeper.servers" ["192.168.10.7"], "transactional.zookeeper.root" > "/transactional", "topology.acker.executors" nil, > "topology.kryo.decorators" (), "topology.name" "PaVan", > "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, > "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", > "supervisor.heartbeat.frequency.secs" 5, > "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, > "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", > "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, > "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, > "topology.spout.wait.strategy" > "backtype.storm.spout.SleepSpoutWaitStrategy", > "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, > "storm.zookeeper.retry.interval" 1000, " > topology.sleep.spout.wait.strategy.time.ms" 1, > "nimbus.topology.validator" > "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" > [6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs" > 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil, > "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, > "topology.workers" 1, "supervisor.childopts" "-Xmx256m", > "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, > "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" > "backtype.storm.serialization.types.ListDelegateSerializer", > "topology.disruptor.wait.strategy" > "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" > "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, > "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" > "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" > 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" > 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" > "backtype.storm.security.auth.SimpleTransportPlugin", > "topology.state.synchronization.timeout.secs" 60, > "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" > 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", " > logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, > "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" > 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "distributed", > "topology.max.task.parallelism" nil, > "storm.messaging.netty.transfer.batch.size" 262144} > > 2014-09-08 07:19:09 b.s.d.worker [INFO] Worker > 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 for storm PaVan-1-1410160741 on > 51fb6b55-0522-437c-a18e-72212192db0f:6703 has finished loading > > > > Supervisor Log: > > > > b/json-simple-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-io-2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/usr/local/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/conf:storm-local/supervisor/stormdist/PaVan-1-1410160741/stormjar.jar' > 'backtype.storm.daemon.worker' 'PaVan-1-1410160741' > '51fb6b55-0522-437c-a18e-72212192db0f' '6703' > '2c12526f-d588-4dcc-99b9-b26e8c1d1d70' > > 2014-09-08 07:19:02 b.s.d.supervisor [INFO] > 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started > > 2014-09-08 07:19:02 b.s.d.supervisor [INFO] > 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started > > 2014-09-08 07:19:03 b.s.d.supervisor [INFO] > 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started > > 2014-09-08 07:19:03 b.s.d.supervisor [INFO] > 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started > > 2014-09-08 07:19:04 b.s.d.supervisor [INFO] > 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started > > 2014-09-08 07:19:04 b.s.d.supervisor [INFO] > 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started > > 2014-09-08 07:19:05 b.s.d.supervisor [INFO] > 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started > > 2014-09-08 07:19:05 b.s.d.supervisor [INFO] > 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started > > 2014-09-08 07:19:06 b.s.d.supervisor [INFO] > 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started > > 2014-09-08 07:19:06 b.s.d.supervisor [INFO] > 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started > > 2014-09-08 07:19:07 b.s.d.supervisor [INFO] > 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started > > 2014-09-08 07:19:07 b.s.d.supervisor [INFO] > 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started > > 2014-09-08 07:19:08 b.s.d.supervisor [INFO] > 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started > > 2014-09-08 07:19:08 b.s.d.supervisor [INFO] > 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started > > > > *From:* Raghavendra Nandagopal [mailto:askaboutst...@gmail.com] > *Sent:* 09 September 2014 10:59 > *To:* user@storm.incubator.apache.org > *Subject:* RE: Running Topology on Remote Machine > > > > Hi Sridhar, > Can you check the worker logs and see if there are any exceptions? > > Thanks, > Raghav > > On Sep 8, 2014 10:25 PM, "Sridhar G" <sridh...@microland.com> wrote: > > Hi Dhaval, > > We have tried with the below config but no luck . Anything else to add to > it ? > > Regards > Sridhar Gadi > > > -----Original Message----- > From: Dhaval Modi [mailto:dhavalmod...@gmail.com <dhavalmod...@gmail.com>] > Sent: Tue 9/9/2014 9:45 AM > To: user > Cc: Pavan Jakati G > Subject: Re: Running Topology on Remote Machine > > Hi Sridhar, > > Did you try commenting " conf.setMaxTaskParallelism(10);" ? > > Regards, > Dhaval Modi > dhavalmod...@gmail.com > > On 9 September 2014 09:13, Sridhar G <sridh...@microland.com> wrote: > > > > > > > > > > > Hi All, > > > > > > > > We are stuck with running our topology on Remote Machine. We have be able > > to run our topology on local machine which emits tuples. But when we try > to > > run it on Remote Machine it does not emit tuples. Whereas the Formation > of > > Topology is shown on STORM UI under TOPOLOGY SUMMARY. TOPOLOGY STATS , > > SPOUTS and BOLTS are visible with ZERO emits. > > > > > > > > The Code which we have used for Running Topology on REMOTE Machine is: > > > > > > > > LIBRARIES IMPORTS ARE - > > > > > > > > import backtype.storm.Config; > > > > import backtype.storm.LocalCluster; > > > > import backtype.storm.topology.TopologyBuilder; > > > > import backtype.storm.utils.Utils; > > > > > > > > > > > > import org.json.simple.JSONValue; > > > > import backtype.storm.StormSubmitter; > > > > import backtype.storm.generated.AlreadyAliveException; > > > > import backtype.storm.generated.Nimbus.Client; > > > > import backtype.storm.topology.TopologyBuilder; > > > > import backtype.storm.utils.NimbusClient; > > > > import java.util.Map; > > > > > > > ------------------------------------------------------------------------------------------------------------------------- > > > > *TOPOLOGY PROGRAM -* > > > > > > > > TopologyBuilder builder = new TopologyBuilder(); > > > > builder.setSpout("POCspout", new POCSpout(),3); > > > > builder.setBolt("POCbolt", new > > POCBolt(),3).shuffleGrouping("POCspout"); > > > > > > > > Config conf = new Config(); > > > > conf.put(Config.NIMBUS_HOST, "192.xxx.xx.x"); > > > > conf.setDebug(true); > > > > conf.setNumWorkers(1); > > > > conf.setMaxSpoutPending(5000); > > > > conf.setNumAckers(1); > > > > conf.setMaxTaskParallelism(10); > > > > Map storm_conf = Utils.readStormConfig(); > > > > > > > > storm_conf.put("nimbus.host", "192.xxx.xx.x"); > > > > Client client = > > NimbusClient.getConfiguredClient(storm_conf).getClient(); > > > > > > > > String inputJar = > > "/root/SOIM_PaVan/target/SOIMTopology-1.0-SNAPSHOT.jar"; > > > > NimbusClient nimbus = new NimbusClient(storm_conf, > > "192.xxx.xx.x",6627); > > > > String uploadedJarLocation = > > StormSubmitter.submitJar(storm_conf,inputJar); > > > > > > > > try { > > > > String jsonConf = JSONValue.toJSONString(storm_conf); > > > > > > > nimbus.getClient().submitTopology("TOPOLOGYPOC",uploadedJarLocation,jsonConf,builder.createTopology()); > > > > } > > > > catch (AlreadyAliveException ae) > > > > { > > > > ae.printStackTrace(); > > > > } > > > > > > > > > > > > > > > > *For any Correction in code let us know.* > > > > > > > > > > > > Regards, > > > > Sridhar > > > > *Believe in the impossible and remove the improbable* > > > > The information contained in this transmission may contain privileged and > > confidential information of Microland Limited, including information > > protected by privacy laws. It is intended only for the use of Microland > > Limited. If you are not the intended recipient, you are hereby notified > > that any review, dissemination, distribution, or duplication of this > > communication is strictly prohibited. If you are not the intended > > recipient, please contact the sender by reply email and destroy all > copies > > of the original message. > > > > The information contained in this transmission may contain privileged and > confidential information of Microland Limited, including information > protected by privacy laws. It is intended only for the use of Microland > Limited. If you are not the intended recipient, you are hereby notified > that any review, dissemination, distribution, or duplication of this > communication is strictly prohibited. If you are not the intended > recipient, please contact the sender by reply email and destroy all copies > of the original message. > > > > > > -- > Regards, > Vikas Agarwal > 91 – 9928301411 > > InfoObjects, Inc. > Execution Matters > http://www.infoobjects.com > 2041 Mission College Boulevard, #280 > Santa Clara, CA 95054 > +1 (408) 988-2000 Work > +1 (408) 716-2726 Fax > -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 +1 (408) 988-2000 Work +1 (408) 716-2726 Fax
