Re: Suspending a Spout

Hi Phil, I believe you can call Nimbus via it's Thrift interface and call the "Deactivate" and "Activate" functions, which pause and resume the topology spouts, respectively. But in your case, I believe a better architecture would be to store the data into an intermediate queue, like Kafka. That

Re: Suspending a Spout

Using storm facility for that purpose seems ill suited; for a starter, by the time you successfully suspended the spout, you probably would have violated the rate limit a bit etc. You should be able to do operations that use one token within one thread, so you could simply backoff on rate limit no

Re: Suspending a Spout

may be this is useful https://github.com/ptgoetz/storm-signals On Fri, Feb 21, 2014 at 10:02 AM, Enno Shioji wrote: > Using storm facility for that purpose seems ill suited; for a starter, by > the time you successfully suspended the spout, you probably would have > violated the rate limit a b

RE: Problems with a coordinator and emitter of an error-handling IPartitionedTridentSpout

Unfortunately, that is not the case in 0.9. isReady is called both before a new batch and a batch replay. You can see this if you look at the code - isReady is called in MasterBatchCoordinator, and whether a batch is emitted or reemitted is determined by PartitionedTridentSpoutExecutor, called b

Error while using storm-kafka spout

Hi, I am trying to run a simple program that reads messages from Kafka topic using storm-kafka spout. Here is the snip of the simple program -> List hosts = new ArrayList(); hosts.add("127.0.0.1:9092");//Kafka test server running on this port SpoutConfig kafkaConf = new S

Re: Error while using storm-kafka spout

The default number of kafka partions is 10. You can either change it to 10 in your code or modify your partions number in the server.properties file or use KafkaConfig.ZkHosts for dynamic detection of kafka brokers and partitions On Fri, Feb 21, 2014 at 12:44 PM, Saurabh Agarwal (BLOOMBERG/ 731 L

Re: Error while using storm-kafka spout

Hi, Looks like it is not able to find any topic name *test-topic*.Can you check and confirm if a topic named *test-topic* actually exist on Kafka. Thanks Bijoy On Fri, Feb 21, 2014 at 5:14 PM, Saurabh Agarwal (BLOOMBERG/ 731 LEXIN) < sagarwal...@bloomberg.net> wrote: > Hi, > > I am trying to r

Re: Error while using storm-kafka spout

correction. The default number kafka partition is 1 not 10. On Fri, Feb 21, 2014 at 1:22 PM, bijoy deb wrote: > Hi, > > Looks like it is not able to find any topic name *test-topic*.Can you > check and confirm if a topic named *test-topic* actually exist on Kafka. > > Thanks > Bijoy > > > On Fri

Is there a time period for spout/bolt to wait for open()/prepare() to complete?

Hi, I have one query related to the order in which open()-nextTuple() and prepare()-execute() methods are invoked by storm. In one of my use cases the amount of work that I do in open() or prepare() is more. It may take more time for it to get completed. Is there a guarantee that storm will not i

Deciding on which node a spout runs

In a cluster, is it possible to decide where a certain spout is being run? I would need to have a spout that tails a (log) file and emits events when new data is available. - Matti

Re: Error while using storm-kafka spout

I had tried with passing partition 1 as well. Same error...:( - Original Message - From: user@storm.incubator.apache.org To: user@storm.incubator.apache.org At: Feb 21 2014 07:48:03 correction. The default number kafka partition is 1 not 10. On Fri, Feb 21, 2014 at 1:22 PM, bijoy deb wr

Setting storm config to LocalCluster

I launch a LocalCluster for my unit tests, and I'd like to set parameters to the config. In particular, storm.zookeeper.port such that preoccupation of the default 2000 won't fail the test. Can someone advise? LocalCluster doesn't seem to take a storm Config when launched. -- Lin Zhao https://w

Re: Deciding on which node a spout runs

This might be of some help to you: http://xumingming.sinaapp.com/885/twitter-storm-how-to-develop-a-pluggable-scheduler/

Re: Suspending a Spout

Thanks very much for the replies; all great information and suggestions. One last quick question. Outside of tuples, is there any other mechanism for bolts and spouts to communicate with each other out-of-the-box? On Fri, Feb 21, 2014 at 4:44 AM, Vinoth Kumar Kannan wrote: > may be this is usef