RE: Running Topology on Remote Machine

Hi Sridhar, Can you check the worker logs and see if there are any exceptions? Thanks, Raghav On Sep 8, 2014 10:25 PM, "Sridhar G" wrote: > Hi Dhaval, > > We have tried with the below config but no luck . Anything else to add to > it ? > > Regards > Sridhar Gadi > > > -Original Message--

RE: Running Topology on Remote Machine

Worker log 2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt __system:(-1) 2014-09-08 07:19:09 b.s.d.executor [INFO] Loading executor SOIMbolt:[1 1] 2014-09-08 07:19:09 b.s.d.executor [INFO] Loaded executor tasks SOIMbolt:[1 1] 2014-09-08 07:19:09 b.s.d.executor [INFO] Finished loading

RE: Running Topology on Remote Machine

Hi Dhaval, We have tried with the below config but no luck . Anything else to add to it ? Regards Sridhar Gadi -Original Message- From: Dhaval Modi [mailto:dhavalmod...@gmail.com] Sent: Tue 9/9/2014 9:45 AM To: user Cc: Pavan Jakati G Subject: Re: Running Topology on Remote Machine Hi

How big a cluster Nimbus to handle

Hi All, What's the recommended size Nimbus can handle without any management/performance implications? In terms of supervisors, workers. We have already a few hundreds of workers , a few tens of topologies in the cluster We observe that sometimes (not too rarely) Nimbus getting stuck during re-bal

Re: Running Topology on Remote Machine

Hi Sridhar, Did you try commenting " conf.setMaxTaskParallelism(10);" ? Regards, Dhaval Modi dhavalmod...@gmail.com On 9 September 2014 09:13, Sridhar G wrote: > > > > > Hi All, > > > > We are stuck with running our topology on Remote Machine. We have be able > to run our topology on local ma

Running Topology on Remote Machine

Hi All, We are stuck with running our topology on Remote Machine. We have be able to run our topology on local machine which emits tuples. But when we try to run it on Remote Machine it does not emit tuples. Whereas the Formation of Topology is shown on STORM UI under TOPOLOGY SUMMARY. TO

Re: How Do Workers Connect To Nimbus

Although, implementing the Storm cluster manually would be really nice learning, I would suggest using HortonWorks distribution which comes with Storm as OOTB solution and you can configure everything from Ambari UI. We are using Storm on Amazon EC2 machine, though it is right now in beta stage. We

Re: How Do Workers Connect To Nimbus

Stephen, I am not able to reach that IP. But you shouldn't modify the default.yaml just change storm.yaml under conf "Will the storm.yaml be the same on my worker and nimbus machine?" it should be the same on both machines. Make sure your zookeeper also running on that ip. And check fo

Re: How Do Workers Connect To Nimbus

All, I implemented the suggestions given by Parh and Harsha. I am now using the default.yaml but I changed the storm.zookeeper.servers to the nimbus machine's ip address: 54.68.149.181. I also changed the nimbus.host to 54.68.149.181. I also opened up port 6627. Now, the UI web page gives the fo

designing topologies

When you design a topology do you still use MVC design patterns? B.

Re: How Do Workers Connect To Nimbus

I'm getting kind of confused by the storm.yaml file. Should I be using the default.yaml and just modify the zookeeper and nimbus ip, or should I use a bran new storm.yaml? My nimbus machine has the ip address: 54.68.149.181. My zookeeper is on the nimbus machine. what should the storm.yaml look li

Re: How Do Workers Connect To Nimbus

All Thanks so much for your help. I cannot tell you how much I appreciate it. I'm going to try out your suggestions and keep banging my head again the wall : D. I've spent an enormous amount of time trying to get this to work. I'll let you know what happens after I try to implement your suggestio

Re: How Do Workers Connect To Nimbus

The worker connects to the thrift port and not the ui port. You need to open port 6627 or whatever is the value being set in storm.yaml using property “nimbus.thrift.port”. Based on the configuration that you have pointed so far it seems your nimbus host has nimbus,ui,supervisor working becaus

Re: How Do Workers Connect To Nimbus

I am accesing http://54.68.149.181:8080/ using chrome. But somehow ping doesn't work says "could not find host http://54.68.149.181:8080/"; On Mon, Sep 8, 2014 at 6:49 PM, Stephen Hartzell wrote: > I'll go ahead and implement your suggestion for the .yaml. > > Your other question is also confusi

Re: How Do Workers Connect To Nimbus

I'll go ahead and implement your suggestion for the .yaml. Your other question is also confusing to me. I can access the storm UI from my personal computer using http://54.68.149.181:8080/, but my personal computer cannot ping it nor can my worker AWS machine. On Mon, Sep 8, 2014 at 6:42 PM, Hars

Re: How Do Workers Connect To Nimbus

storm.zookeeper.servers: - "127.0.0.1" nimbus.host: "127.0.0.1" ( 127.0.0.1 causes to bind a loopback interface , instead either use your public ip or 0.0.0.0) storm.local.dir: /tmp/storm ( I recommend this to move to a different folder probably /home/storm, /tmp/storm will get deleted if your

Re: How Do Workers Connect To Nimbus

Also, to answer Harsha's question. I have no errors in any of my logs on the nimbus or worker machines. On Mon, Sep 8, 2014 at 6:39 PM, Stephen Hartzell wrote: > I opened up port 8080 so that I could access the UI page. Do I need to > open up something beyond that? > > On Mon, Sep 8, 2014 at 6:3

Re: How Do Workers Connect To Nimbus

I opened up port 8080 so that I could access the UI page. Do I need to open up something beyond that? On Mon, Sep 8, 2014 at 6:37 PM, Parth Brahmbhatt < pbrahmbh...@hortonworks.com> wrote: > One thing that is generally needed on EC2 hosts is security group setting > > > http://docs.aws.amazon.com

Re: How Do Workers Connect To Nimbus

One thing that is generally needed on EC2 hosts is security group setting http://docs.aws.amazon.com/gettingstarted/latest/computebasics/getting-started-security-group.html If you can not ping the host I am guessing this might be one of the things that you need to setup correctly. On Sep 8, 20

Re: How Do Workers Connect To Nimbus

Harsha, The storm.yaml on the host machine looks like this: storm.zookeeper.servers: - "127.0.0.1" nimbus.host: "127.0.0.1" storm.local.dir: /tmp/storm The storm.yaml on the worker machine looks like this: storm.zookeeper.servers: - "54.68.149.181" nimbus.host: "54.68.149.18

Re: How Do Workers Connect To Nimbus

There aren't any errors in worker machine supervisor logs. Are you using the same storm.yaml for both the machines and also are you able to ping your nimbus host or connect to zookeeper on nimbus host. -Harsha On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote: Harsha, Thanks so muc

Re: How Do Workers Connect To Nimbus

Also, one of the first things that I had tried to do is follow that tutorial. The only problem is that tutorial appears to gloss over the setup of the worker nodes. It just says that they will automatically register with zookeeper. But there must be some setup or configuration to tell the machine w

Re: How Do Workers Connect To Nimbus

Harsha, Thanks so much for getting back with me. I will check the logs, but I don't seem to get any error messages. I have a nimbus AWS machine with zookeeper on it and a worker AWS machine. On the nimbus machine I start the zookeeper and then I run: bin/storm nimbus & bin/storm supervisor & b

Re: How Do Workers Connect To Nimbus

Hi Stephen, What are the issues you are seeing. "How do worker machines "know" how to connect to nimbus? Is it in the storm configuration file" Yes. make sure you the supervisor(worker) , nimbus nodes are able to connect to your zookeeper cluster. Check your logs under storm_inst/logs/

How Do Workers Connect To Nimbus

All, I would greatly appreciate any help that anyone would afford. I've been trying to setup a storm cluster on AWS for a few weeks now on centOS EC2 machines. So far, I haven't been able to get a cluster built. I can get a supervisor and nimbus to run on a single machine, but I can't figure out h

Re: Test timeout

Thanks Taylor. I have created a JIRA . I do not have experience with clojure, so I was wondering if I could override this value, even if the method is not very straightforward like changing it using reflection. On Mon, Sep 8, 2014 at 4:50 PM, P. Tay

Re: Test timeout

We should change the fact that it is hard coded. I'd rather see it look for a system property (with a default value). That way it can be overridden easily, for example in resource-constrained environments (e.g. Jenkins). Feel free to open a JIRA ticket for this. I'd be happy to do it as well. -

Re: Test timeout

I can see that the test time out is set to 5000ms in https://github.com/apache/incubator-storm/blob/master/storm-core/src/clj/backtype/storm/testing.clj. Is there anyway to override this value? On Mon, Sep 8, 2014 at 1:39 PM, Devika Nair wrote: > Hi all, > > > I am trying to follow the example o

Test timeout

Hi all, I am trying to follow the example on https://github.com/xumingming/storm-lib/blob/master/src/jvm/storm/TestingApiDemo.java and writing tests for my topology. However, my tests(similar to testBasicTopology()) fail due to timeout when I add a bolt(the prepare for this bolt takes quite lo

How to consume from last offset when topology restarts(STORM-KAFKA)

Any update or resolution for this issue in storm kafka spout Regards Sai

Re: Supervisor unable to connect to zookeeper

It appears to be trying to connect to zk on 127.0.01:2181 which is incorrect if you have zookeeper running on it's own server (as you mentioned above). I would guess that you have not configured a zookeeper server or it is not configured correctly. On Mon, Sep 8, 2014 at 8:16 AM, ​Chandrahas Gurr

Re: Supervisor unable to connect to zookeeper

yes, based on that, i have configured servers. But could you give me any solution for the error I have mentioned. ThankYou, G V Chandrahas Raj On Mon, Sep 8, 2014 at 4:09 PM, Vikas Agarwal wrote: > Have you followed the doc > https://storm.incubator.apache.org/documentation/Setting-up-a-Storm-

Re: Is there a Tweeter Streaming Spout?

Just to note its an example spout . But I am not sure why it wouldn't allow parallelism of more than 1. For twitter api its http calls thats by increasing the parallelism of the spout you are making more calls to twitter api. I think twitter rate limits based on appid thats the limitation I can see

Re: Is there a Tweeter Streaming Spout?

That is interesting. However, it won't allow spout parallelism more than 1, right? On Mon, Sep 8, 2014 at 6:56 PM, Harsha wrote: > Hi Connie, > You can take a look at twittersamplespout in examples > https://github.com/apache/incubator-storm/blob/master/examples/storm-starter/src/jvm

Re: Is there a Tweeter Streaming Spout?

Hi Connie, You can take a look at twittersamplespout in examples [1]https://github.com/apache/incubator-storm/blob/mast er/examples/storm-starter/src/jvm/storm/starter/spout/TwitterSa mpleSpout.java It uses twitter4j to read the api you can make changes to fit your needs. -Harsha

Re: When does nimbus rebalance a topology?

Look at the supervisors logs. There could be a timeout that make a supervisor to force a worker to shut down, triggering a rebalancing. 2014-09-08 7:17 GMT-04:00 Palak Shah : > I have a topology that uses a Kafka spout to read values from a Kafka > queue. I used the kafkaSpout that came with st

Re: questions of setNumWorkers and declareOutputFields function

On Mon, Sep 8, 2014 at 12:59 AM, Jiang Jacky wrote: > Hi, Guys > I just read the document about the apache storm, my question is the number > of workers should be equal to the number of supervisor node, right? because > workers control all treads across the topology, so we d better evenly > distr

When does nimbus rebalance a topology?

I have a topology that uses a Kafka spout to read values from a Kafka queue. I used the kafkaSpout that came with storm-0.9.2-incubating. I observed that the nimbus rebalances the topology very often. The topology suddenly shuts down and starts again with the tasks running on different machines. I

Re: Supervisor unable to connect to zookeeper

Have you followed the doc https://storm.incubator.apache.org/documentation/Setting-up-a-Storm-cluster.html On Mon, Sep 8, 2014 at 1:21 PM, ​Chandrahas Gurram wrote: > Hello, > > I have manually installed zookeeper, superisors and nimbus on different > servers. But when I run supervisors they ar

Supervisor unable to connect to zookeeper

Hello, I have manually installed zookeeper, superisors and nimbus on different servers. But when I run supervisors they are unable to connect to zookeeper. The following error is thrown in supervisor.log continuosuly.. 2014-09-04 15:56:21 o.a.z.ClientCnxn [INFO] Opening socket connection to serv

Re: Is there a Tweeter Streaming Spout?

I guess no and it won't make sense to have one because it would limit the parallelism of spout. Twitter stream allows only single connection to the stream. You can use threading to have parallelism in stream consumption but it would be difficult to manage it with spouts. Better solution would be to