Re: Kafka Rebalance exceptions in Storm

[padma priya chitturi]

The same exception i encountered when tried to integrate storm spout with
kafka high level consumer. This was resolved when i used SimpleConsumer.


On Thu, Apr 3, 2014 at 8:26 AM, Kashyap Mhaisekar wrote:

> Hi,
> I consistently get the following exception when I submit a storm topology
> using StormSubmitter. I am using 0.8.2 Storm.
> The spout I use is a Kafka High Level consumer thats custom written.
>
> When I submit the same topology using LocalCluster, it works fine. Am I
> missing anything
>
> *kafka.common.ConsumerRebalanceFailedException:
> Sample_Kafka_client_abcd_com-1396492916541-2f28228c can't rebalance after 4
> retries*
> *at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
> ~[stormjar.jar:na]*
> *at
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
> ~[stormjar.jar:na]*
> *at
> kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:212)
> ~[stormjar.jar:na]*
> *at
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:80)
> ~[stormjar.jar:na]*
> *at
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:92)
> ~[stormjar.jar:na]*
>
>
> *2014-04-02 21:41:58 b.s.util [INFO] Halting process: ("Worker died") *
>
> Thanks in advance
> Regards,
> Kashyap
>

Re: Basic storm question

[padma priya chitturi]

Hi,

This is how you should run nimbus/supervisor:

/bin$./storm nimbus
/bin$./storm supervisor


On Wed, Apr 2, 2014 at 11:42 PM, Leonardo Bohac wrote:

> Hello, I've downloaded the last version of storm at
> http://storm.incubator.apache.org/downloads.html and when I try to do the 
> */bin/storm
> nimbus* command I get the following message:
>
> *The storm client can only be run from within a release. You appear to be
> trying to run the client from a checkout of Storm's source code.*
>
> *You can download a Storm release
> at http://storm-project.net/downloads.html
> *
>
>
>
>  I don't know whats missing...
>
>
> Thanks!
>
>
> 2014-04-02 15:05 GMT-03:00 Nathan Leung :
>
> No, it creates an extra executor to deal with processing the ack messages
>> that are sent by the bolts after processing tuples.  See the following for
>> details on how acking works in storm:
>> https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing.
>>  By default storm will create 1 acker per worker you have in your topology.
>>
>>
>> On Wed, Apr 2, 2014 at 2:01 PM, Huiliang Zhang  wrote:
>>
>>> Hi Nathan,
>>>
>>> The last bolt just emits the tuples and no more bolt in the topology
>>> will consume and ack the tuples. Do you mean that storm automatically
>>> creates an extra executor to deal with the tuples?
>>>
>>> Thanks,
>>> Huiliang
>>>
>>>
>>> On Wed, Apr 2, 2014 at 8:31 AM, Nathan Leung  wrote:
>>>
 the extra task/executor is the acker thread.


 On Tue, Apr 1, 2014 at 9:23 PM, Huiliang Zhang wrote:

> I just submitted ExclamationTopology for testing.
>
> builder.setSpout("word", new TestWordSpout(), 10);
>
> builder.setBolt("exclaim1", new ExclamationBolt(),
> 3).shuffleGrouping("word");
>
> builder.setBolt("exclaim2", new ExclamationBolt(),
> 2).shuffleGrouping("exclaim1");
>
> I am supposed to see 15 executors. However, I see 16 executors and 16
> tasks on topology summary on storm UI. The numbers of executors are 
> correct
> for the specific spout and bolts and aggregate to 15. Is that a bug in
> displaying topology summary?
>
> My cluster consists of 2 supervisors and each has 4 workers defined.
>
> Thanks.
>
>
> On Tue, Apr 1, 2014 at 1:43 PM, Nathan Leung wrote:
>
>> By default supervisor nodes can run up to 4 workers.  This is
>> configurable in storm.yaml (for example see supervisor.slots.ports here:
>> https://github.com/nathanmarz/storm/blob/master/conf/defaults.yaml).
>>  Memory should be split between the workers.  It's a typical Java heap, 
>> so
>> anything running on that worker process shares the heap.
>>
>>
>> On Tue, Apr 1, 2014 at 4:10 PM, David Crossland <
>> da...@elastacloud.com> wrote:
>>
>>>  On said subject, how does memory allocation work I these cases?
>>> Assuming 1 worker per node would you just dump all the memory available
>>> into worker.childopts? I guess the memory pool would be shared between 
>>> the
>>> spawned threads as appropriate to their needs?
>>>
>>>  I'm assuming the equivalent options for supervisor/nimbus are fine
>>> left at defaults.  Given that the workers/spouts/bolts are the working
>>> parts of the topology these would where I should target available 
>>> memory?
>>>
>>>  D
>>>
>>>   *From:* Huiliang Zhang 
>>> *Sent:* Tuesday, 1 April 2014 19:47
>>> *To:* user@storm.incubator.apache.org
>>>
>>>  Thanks. It should be good if there exist some example figures
>>> explaining the relationship between tasks, workers, and threads.
>>>
>>>
>>> On Sat, Mar 29, 2014 at 6:34 AM, Susheel Kumar Gadalay <
>>> skgada...@gmail.com> wrote:
>>>
 No, a single worker is dedicated to a single topology no matter how
 many threads it spawns for different bolts/spouts.
 A single worker cannot be shared across multiple topologies.

 On 3/29/14, Nathan Leung  wrote:
 > From what I have seen, the second topology is run with 1 worker
 until you
 > kill the first topology or add more worker slots to your cluster.
 >
 >
 > On Sat, Mar 29, 2014 at 2:57 AM, Huiliang Zhang 
 wrote:
 >
 >> Thanks. I am still not clear.
 >>
 >> Do you mean that in a single worker process, there will be
 multiple
 >> threads and each thread will handle part of a topology? If so,
 what does
 >> the number of workers mean when submitting topology?
 >>
 >>
 >> On Fri, Mar 28, 2014 at 11:18 PM, padma priya chitturi <
 >> padmapriy...@gmail.com> wrote:
 >>
 >>> Hi,
 >>>
 >>> No, its not the case. No matter how many topologies you submit,
 the
 >>> workers will be shared among 

回复： topology doesn't run before shutting down storm(0.9.0.1)

[hisunwei]

Try to delete the supervisor's state info on the disk.

--- 原始邮件 ---

发件人： "Rob Punkunus" 
已发： 2014年4月3日 上午3:51
收件人： user@storm.incubator.apache.org
主题： topology doesn't run before shutting down storm(0.9.0.1)

Hi,

I am running a topology in storm 0.9.0.1 and it looks like the supervisor
is shutting it down 1 second after starting it.  I am including supervisor
logs below.  Anyone see this or have ideas on the next steps to debug this?

Thanks
Rob

2014-04-01 13:18:13 b.s.d.supervisor [INFO] Starting supervisor with id
mesos01-peak1.prod.internal-artemis-116-1396313446 at host
mesos01-peak1.prod.internal
2014-04-01 13:18:14 b.s.d.supervisor [INFO] Shutting down and clearing
state for id ab9e7eb5-d3e6-45d7-8fa6-d244bba6545a. Current supervisor time:
1396358294. State: :disallowed, Heartbeat: nil
2014-04-01 13:18:14 b.s.d.supervisor [INFO] Shutting down
mesos01-peak1.prod.internal-artemis-116-1396313446:ab9e7eb5-d3e6-45d7-8fa6-d244bba6545a

Kafka Rebalance exceptions in Storm

[Kashyap Mhaisekar]

Hi,
I consistently get the following exception when I submit a storm topology
using StormSubmitter. I am using 0.8.2 Storm.
The spout I use is a Kafka High Level consumer thats custom written.

When I submit the same topology using LocalCluster, it works fine. Am I
missing anything

*kafka.common.ConsumerRebalanceFailedException:
Sample_Kafka_client_abcd_com-1396492916541-2f28228c can't rebalance after 4
retries*
*at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
~[stormjar.jar:na]*
*at
kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
~[stormjar.jar:na]*
*at
kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:212)
~[stormjar.jar:na]*
*at
kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:80)
~[stormjar.jar:na]*
*at
kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:92)
~[stormjar.jar:na]*


*2014-04-02 21:41:58 b.s.util [INFO] Halting process: ("Worker died")*

Thanks in advance
Regards,
Kashyap

topology doesn't run before shutting down storm(0.9.0.1)

[Rob Punkunus]

Hi,

I am running a topology in storm 0.9.0.1 and it looks like the supervisor
is shutting it down 1 second after starting it.  I am including supervisor
logs below.  Anyone see this or have ideas on the next steps to debug this?

Thanks
Rob

2014-04-01 13:18:13 b.s.d.supervisor [INFO] Starting supervisor with id
mesos01-peak1.prod.internal-artemis-116-1396313446 at host
mesos01-peak1.prod.internal
2014-04-01 13:18:14 b.s.d.supervisor [INFO] Shutting down and clearing
state for id ab9e7eb5-d3e6-45d7-8fa6-d244bba6545a. Current supervisor time:
1396358294. State: :disallowed, Heartbeat: nil
2014-04-01 13:18:14 b.s.d.supervisor [INFO] Shutting down
mesos01-peak1.prod.internal-artemis-116-1396313446:ab9e7eb5-d3e6-45d7-8fa6-d244bba6545a

workers restarting very shortly after startup

[Khaled Elmeleegy]

Hi,
Workers are restarting very quickly after startup (30s-40s). I am running a 
topology on my laptop for testing, so I am guessing tasks are running into some 
error. The problem is I don't see any error reported on the log, even though i 
have the logging level set to debug. Please see a snippet of the worker log 
below.
Note that the bolts are reported to be starting, then there is 37s (23:21:31 - 
23:22:08) pause in the log, afterwards the worker seems to restart. Can anyone 
guess what might be going on given what's in the log? The second question is 
there something one can do to make the log more informative and capture 
whatever error that might have happened?
Thanks,Khaled



2014-04-01 23:21:30 b.s.d.executor [INFO] Loading executor IngaBolt_01_003:[2 
2]2014-04-01 23:21:31 b.s.d.task [INFO] Emitting: IngaBolt_01_003 __system 
["startup"]2014-04-01 23:21:31 b.s.d.executor [INFO] Loaded executor tasks 
IngaBolt_01_003:[2 2]2014-04-01 23:21:31 b.s.d.executor [INFO] Finished loading 
executor IngaBolt_01_003:[2 2]2014-04-01 23:21:31 o.a.z.ClientCnxn [DEBUG] 
Reading reply sessionid:0x144ebfa68472986, packet:: clientPath:null 
serverPath:null finished:false header:: 20,3  replyHeader:: 20,74078,0  
request:: '/storm/assignments,F  response:: 
s{47,47,1395528580739,1395528580739,0,61,0,0,1,1,74071} 2014-04-01 23:21:31 
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet:: 
clientPath:null serverPath:null finished:false header:: 21,3  replyHeader:: 
21,74078,0  request:: '/storm/storms,F  response:: 
s{48,48,1395528580741,1395528580741,0,61,0,0,1,1,74070} 2014-04-01 23:21:31 
b.s.d.executor [INFO] Preparing bolt IngaBolt_01_003:(2)2014-04-01 23:21:31 
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet:: 
clientPath:null serverPath:null finished:false header:: 22,3  replyHeader:: 
22,74078,0  request:: '/storm/supervisors,F  response:: 
s{49,49,1395528580743,1395528580743,0,107,0,0,1,1,74061} 2014-04-01 23:21:31 
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet:: 
clientPath:null serverPath:null finished:false header:: 23,3  replyHeader:: 
23,74078,0  request:: '/storm/workerbeats,F  response:: 
s{50,50,1395528580745,1395528580745,0,85,0,0,1,1,74069} 2014-04-01 23:21:31 
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet:: 
clientPath:null serverPath:null finished:false header:: 24,3  replyHeader:: 
24,74078,0  request:: '/storm/errors,F  response:: 
s{51,51,1395528580747,1395528580747,0,26,0,0,1,0,70999} 2014-04-01 23:21:31 
b.s.d.executor [INFO] Loading executor IngaBolt_01_048:[3 3]2014-04-01 23:21:31 
b.s.d.task [INFO] Emitting: IngaBolt_01_048 __system ["startup"]2014-04-01 
23:21:31 b.s.d.executor [INFO] Loaded executor tasks IngaBolt_01_048:[3 
3]2014-04-01 23:21:31 b.s.d.executor [INFO] Finished loading executor 
IngaBolt_01_048:[3 3]2014-04-01 23:21:31 b.s.d.executor [INFO] Preparing bolt 
IngaBolt_01_048:(3)2014-04-01 23:21:31 o.a.z.ClientCnxn [DEBUG] Reading reply 
sessionid:0x144ebfa68472986, packet:: clientPath:null serverPath:null 
finished:false header:: 25,3  replyHeader:: 25,74078,0  request:: 
'/storm/assignments,F  response:: 
s{47,47,1395528580739,1395528580739,0,61,0,0,1,1,74071} 2014-04-01 23:21:31 
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet:: 
clientPath:null serverPath:null finished:false header:: 26,3  replyHeader:: 
26,74078,0  request:: '/storm/storms,F  response:: 
s{48,48,1395528580741,1395528580741,0,61,0,0,1,1,74070} 2014-04-01 23:21:31 
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet:: 
clientPath:null serverPath:null finished:false header:: 27,3  replyHeader:: 
27,74078,0  request:: '/storm/supervisors,F  response:: 
s{49,49,1395528580743,1395528580743,0,107,0,0,1,1,74061} 2014-04-01 23:21:31 
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet:: 
clientPath:null serverPath:null finished:false header:: 28,3  replyHeader:: 
28,74078,0  request:: '/storm/workerbeats,F  response:: 
s{50,50,1395528580745,1395528580745,0,85,0,0,1,1,74069} 2014-04-01 23:21:31 
o.a.z.ClientCnxn [DEBUG] Reading reply sessionid:0x144ebfa68472986, packet:: 
clientPath:null serverPath:null finished:false header:: 29,3  replyHeader:: 
29,74078,0  request:: '/storm/errors,F  response:: 
s{51,51,1395528580747,1395528580747,0,26,0,0,1,0,70999} 2014-04-01 23:21:31 
b.s.d.executor [INFO] Loading executor IngaBolt_01_048:[4 4]2014-04-01 23:21:31 
b.s.d.task [INFO] Emitting: IngaBolt_01_048 __system ["startup"]2014-04-01 
23:21:31 b.s.d.executor [INFO] Loaded executor tasks IngaBolt_01_048:[4 
4]2014-04-01 23:21:31 b.s.d.executor [INFO] Finished loading executor 
IngaBolt_01_048:[4 4]2014-04-01 23:21:31 b.s.d.executor [INFO] Preparing bolt 
IngaBolt_01_048:(4)2014-04-01 23:21:31 o.a.z.ClientCnxn [DEBUG] Reading reply 
sessionid:0x144ebfa68472986, packet:: clientPath:null serverPath:null 
finished:false header:: 30,3  replyHeader:

Re: Basic storm question

[Leonardo Bohac]

Hello, I've downloaded the last version of storm at
http://storm.incubator.apache.org/downloads.html and when I try to do
the */bin/storm
nimbus* command I get the following message:

*The storm client can only be run from within a release. You appear to be
trying to run the client from a checkout of Storm's source code.*

*You can download a Storm release
at http://storm-project.net/downloads.html
*



I don't know whats missing...


Thanks!


2014-04-02 15:05 GMT-03:00 Nathan Leung :

> No, it creates an extra executor to deal with processing the ack messages
> that are sent by the bolts after processing tuples.  See the following for
> details on how acking works in storm:
> https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing.
>  By default storm will create 1 acker per worker you have in your topology.
>
>
> On Wed, Apr 2, 2014 at 2:01 PM, Huiliang Zhang  wrote:
>
>> Hi Nathan,
>>
>> The last bolt just emits the tuples and no more bolt in the topology will
>> consume and ack the tuples. Do you mean that storm automatically creates an
>> extra executor to deal with the tuples?
>>
>> Thanks,
>> Huiliang
>>
>>
>> On Wed, Apr 2, 2014 at 8:31 AM, Nathan Leung  wrote:
>>
>>> the extra task/executor is the acker thread.
>>>
>>>
>>> On Tue, Apr 1, 2014 at 9:23 PM, Huiliang Zhang  wrote:
>>>
 I just submitted ExclamationTopology for testing.

 builder.setSpout("word", new TestWordSpout(), 10);

 builder.setBolt("exclaim1", new ExclamationBolt(),
 3).shuffleGrouping("word");

 builder.setBolt("exclaim2", new ExclamationBolt(),
 2).shuffleGrouping("exclaim1");

 I am supposed to see 15 executors. However, I see 16 executors and 16
 tasks on topology summary on storm UI. The numbers of executors are correct
 for the specific spout and bolts and aggregate to 15. Is that a bug in
 displaying topology summary?

 My cluster consists of 2 supervisors and each has 4 workers defined.

 Thanks.


 On Tue, Apr 1, 2014 at 1:43 PM, Nathan Leung  wrote:

> By default supervisor nodes can run up to 4 workers.  This is
> configurable in storm.yaml (for example see supervisor.slots.ports here:
> https://github.com/nathanmarz/storm/blob/master/conf/defaults.yaml).
>  Memory should be split between the workers.  It's a typical Java heap, so
> anything running on that worker process shares the heap.
>
>
> On Tue, Apr 1, 2014 at 4:10 PM, David Crossland  > wrote:
>
>>  On said subject, how does memory allocation work I these cases?
>> Assuming 1 worker per node would you just dump all the memory available
>> into worker.childopts? I guess the memory pool would be shared between 
>> the
>> spawned threads as appropriate to their needs?
>>
>>  I'm assuming the equivalent options for supervisor/nimbus are fine
>> left at defaults.  Given that the workers/spouts/bolts are the working
>> parts of the topology these would where I should target available memory?
>>
>>  D
>>
>>   *From:* Huiliang Zhang 
>> *Sent:* Tuesday, 1 April 2014 19:47
>> *To:* user@storm.incubator.apache.org
>>
>>  Thanks. It should be good if there exist some example figures
>> explaining the relationship between tasks, workers, and threads.
>>
>>
>> On Sat, Mar 29, 2014 at 6:34 AM, Susheel Kumar Gadalay <
>> skgada...@gmail.com> wrote:
>>
>>> No, a single worker is dedicated to a single topology no matter how
>>> many threads it spawns for different bolts/spouts.
>>> A single worker cannot be shared across multiple topologies.
>>>
>>> On 3/29/14, Nathan Leung  wrote:
>>> > From what I have seen, the second topology is run with 1 worker
>>> until you
>>> > kill the first topology or add more worker slots to your cluster.
>>> >
>>> >
>>> > On Sat, Mar 29, 2014 at 2:57 AM, Huiliang Zhang 
>>> wrote:
>>> >
>>> >> Thanks. I am still not clear.
>>> >>
>>> >> Do you mean that in a single worker process, there will be
>>> multiple
>>> >> threads and each thread will handle part of a topology? If so,
>>> what does
>>> >> the number of workers mean when submitting topology?
>>> >>
>>> >>
>>> >> On Fri, Mar 28, 2014 at 11:18 PM, padma priya chitturi <
>>> >> padmapriy...@gmail.com> wrote:
>>> >>
>>> >>> Hi,
>>> >>>
>>> >>> No, its not the case. No matter how many topologies you submit,
>>> the
>>> >>> workers will be shared among the topologies.
>>> >>>
>>> >>> Thanks,
>>> >>> Padma Ch
>>> >>>
>>> >>>
>>> >>> On Sat, Mar 29, 2014 at 5:11 AM, Huiliang Zhang <
>>> zhl...@gmail.com>
>>> >>> wrote:
>>> >>>
>>>  Hi,
>>> 
>>>  I have a simple question about storm.
>>> 
>>>  My cluster has just 1 super

Re: Basic storm question

[Nathan Leung]

No, it creates an extra executor to deal with processing the ack messages
that are sent by the bolts after processing tuples.  See the following for
details on how acking works in storm:
https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing.
 By default storm will create 1 acker per worker you have in your topology.


On Wed, Apr 2, 2014 at 2:01 PM, Huiliang Zhang  wrote:

> Hi Nathan,
>
> The last bolt just emits the tuples and no more bolt in the topology will
> consume and ack the tuples. Do you mean that storm automatically creates an
> extra executor to deal with the tuples?
>
> Thanks,
> Huiliang
>
>
> On Wed, Apr 2, 2014 at 8:31 AM, Nathan Leung  wrote:
>
>> the extra task/executor is the acker thread.
>>
>>
>> On Tue, Apr 1, 2014 at 9:23 PM, Huiliang Zhang  wrote:
>>
>>> I just submitted ExclamationTopology for testing.
>>>
>>> builder.setSpout("word", new TestWordSpout(), 10);
>>>
>>> builder.setBolt("exclaim1", new ExclamationBolt(),
>>> 3).shuffleGrouping("word");
>>>
>>> builder.setBolt("exclaim2", new ExclamationBolt(),
>>> 2).shuffleGrouping("exclaim1");
>>>
>>> I am supposed to see 15 executors. However, I see 16 executors and 16
>>> tasks on topology summary on storm UI. The numbers of executors are correct
>>> for the specific spout and bolts and aggregate to 15. Is that a bug in
>>> displaying topology summary?
>>>
>>> My cluster consists of 2 supervisors and each has 4 workers defined.
>>>
>>> Thanks.
>>>
>>>
>>> On Tue, Apr 1, 2014 at 1:43 PM, Nathan Leung  wrote:
>>>
 By default supervisor nodes can run up to 4 workers.  This is
 configurable in storm.yaml (for example see supervisor.slots.ports here:
 https://github.com/nathanmarz/storm/blob/master/conf/defaults.yaml).
  Memory should be split between the workers.  It's a typical Java heap, so
 anything running on that worker process shares the heap.


 On Tue, Apr 1, 2014 at 4:10 PM, David Crossland 
 wrote:

>  On said subject, how does memory allocation work I these cases?
> Assuming 1 worker per node would you just dump all the memory available
> into worker.childopts? I guess the memory pool would be shared between the
> spawned threads as appropriate to their needs?
>
>  I'm assuming the equivalent options for supervisor/nimbus are fine
> left at defaults.  Given that the workers/spouts/bolts are the working
> parts of the topology these would where I should target available memory?
>
>  D
>
>   *From:* Huiliang Zhang 
> *Sent:* Tuesday, 1 April 2014 19:47
> *To:* user@storm.incubator.apache.org
>
>  Thanks. It should be good if there exist some example figures
> explaining the relationship between tasks, workers, and threads.
>
>
> On Sat, Mar 29, 2014 at 6:34 AM, Susheel Kumar Gadalay <
> skgada...@gmail.com> wrote:
>
>> No, a single worker is dedicated to a single topology no matter how
>> many threads it spawns for different bolts/spouts.
>> A single worker cannot be shared across multiple topologies.
>>
>> On 3/29/14, Nathan Leung  wrote:
>> > From what I have seen, the second topology is run with 1 worker
>> until you
>> > kill the first topology or add more worker slots to your cluster.
>> >
>> >
>> > On Sat, Mar 29, 2014 at 2:57 AM, Huiliang Zhang 
>> wrote:
>> >
>> >> Thanks. I am still not clear.
>> >>
>> >> Do you mean that in a single worker process, there will be multiple
>> >> threads and each thread will handle part of a topology? If so,
>> what does
>> >> the number of workers mean when submitting topology?
>> >>
>> >>
>> >> On Fri, Mar 28, 2014 at 11:18 PM, padma priya chitturi <
>> >> padmapriy...@gmail.com> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> No, its not the case. No matter how many topologies you submit,
>> the
>> >>> workers will be shared among the topologies.
>> >>>
>> >>> Thanks,
>> >>> Padma Ch
>> >>>
>> >>>
>> >>> On Sat, Mar 29, 2014 at 5:11 AM, Huiliang Zhang > >
>> >>> wrote:
>> >>>
>>  Hi,
>> 
>>  I have a simple question about storm.
>> 
>>  My cluster has just 1 supervisor and 4 ports are defined to run 4
>>  workers. I first submit a topology which needs 3 workers. Then I
>> submit
>>  another topology which needs 2 workers. Does this mean that the
>> 2nd
>>  topology will never be run?
>> 
>>  Thanks,
>>  Huiliang
>> 
>> >>>
>> >>>
>> >>
>> >
>>
>
>

>>>
>>
>

Re: Basic storm question

[Huiliang Zhang]

Hi Nathan,

The last bolt just emits the tuples and no more bolt in the topology will
consume and ack the tuples. Do you mean that storm automatically creates an
extra executor to deal with the tuples?

Thanks,
Huiliang


On Wed, Apr 2, 2014 at 8:31 AM, Nathan Leung  wrote:

> the extra task/executor is the acker thread.
>
>
> On Tue, Apr 1, 2014 at 9:23 PM, Huiliang Zhang  wrote:
>
>> I just submitted ExclamationTopology for testing.
>>
>> builder.setSpout("word", new TestWordSpout(), 10);
>>
>> builder.setBolt("exclaim1", new ExclamationBolt(),
>> 3).shuffleGrouping("word");
>>
>> builder.setBolt("exclaim2", new ExclamationBolt(),
>> 2).shuffleGrouping("exclaim1");
>>
>> I am supposed to see 15 executors. However, I see 16 executors and 16
>> tasks on topology summary on storm UI. The numbers of executors are correct
>> for the specific spout and bolts and aggregate to 15. Is that a bug in
>> displaying topology summary?
>>
>> My cluster consists of 2 supervisors and each has 4 workers defined.
>>
>> Thanks.
>>
>>
>> On Tue, Apr 1, 2014 at 1:43 PM, Nathan Leung  wrote:
>>
>>> By default supervisor nodes can run up to 4 workers.  This is
>>> configurable in storm.yaml (for example see supervisor.slots.ports here:
>>> https://github.com/nathanmarz/storm/blob/master/conf/defaults.yaml).
>>>  Memory should be split between the workers.  It's a typical Java heap, so
>>> anything running on that worker process shares the heap.
>>>
>>>
>>> On Tue, Apr 1, 2014 at 4:10 PM, David Crossland 
>>> wrote:
>>>
  On said subject, how does memory allocation work I these cases?
 Assuming 1 worker per node would you just dump all the memory available
 into worker.childopts? I guess the memory pool would be shared between the
 spawned threads as appropriate to their needs?

  I'm assuming the equivalent options for supervisor/nimbus are fine
 left at defaults.  Given that the workers/spouts/bolts are the working
 parts of the topology these would where I should target available memory?

  D

   *From:* Huiliang Zhang 
 *Sent:* Tuesday, 1 April 2014 19:47
 *To:* user@storm.incubator.apache.org

  Thanks. It should be good if there exist some example figures
 explaining the relationship between tasks, workers, and threads.


 On Sat, Mar 29, 2014 at 6:34 AM, Susheel Kumar Gadalay <
 skgada...@gmail.com> wrote:

> No, a single worker is dedicated to a single topology no matter how
> many threads it spawns for different bolts/spouts.
> A single worker cannot be shared across multiple topologies.
>
> On 3/29/14, Nathan Leung  wrote:
> > From what I have seen, the second topology is run with 1 worker
> until you
> > kill the first topology or add more worker slots to your cluster.
> >
> >
> > On Sat, Mar 29, 2014 at 2:57 AM, Huiliang Zhang 
> wrote:
> >
> >> Thanks. I am still not clear.
> >>
> >> Do you mean that in a single worker process, there will be multiple
> >> threads and each thread will handle part of a topology? If so, what
> does
> >> the number of workers mean when submitting topology?
> >>
> >>
> >> On Fri, Mar 28, 2014 at 11:18 PM, padma priya chitturi <
> >> padmapriy...@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> No, its not the case. No matter how many topologies you submit, the
> >>> workers will be shared among the topologies.
> >>>
> >>> Thanks,
> >>> Padma Ch
> >>>
> >>>
> >>> On Sat, Mar 29, 2014 at 5:11 AM, Huiliang Zhang 
> >>> wrote:
> >>>
>  Hi,
> 
>  I have a simple question about storm.
> 
>  My cluster has just 1 supervisor and 4 ports are defined to run 4
>  workers. I first submit a topology which needs 3 workers. Then I
> submit
>  another topology which needs 2 workers. Does this mean that the
> 2nd
>  topology will never be run?
> 
>  Thanks,
>  Huiliang
> 
> >>>
> >>>
> >>
> >
>


>>>
>>
>

Re: Basic storm question

[Nathan Leung]

the extra task/executor is the acker thread.


On Tue, Apr 1, 2014 at 9:23 PM, Huiliang Zhang  wrote:

> I just submitted ExclamationTopology for testing.
>
> builder.setSpout("word", new TestWordSpout(), 10);
>
> builder.setBolt("exclaim1", new ExclamationBolt(), 3).shuffleGrouping(
> "word");
>
> builder.setBolt("exclaim2", new ExclamationBolt(), 2).shuffleGrouping(
> "exclaim1");
>
> I am supposed to see 15 executors. However, I see 16 executors and 16
> tasks on topology summary on storm UI. The numbers of executors are correct
> for the specific spout and bolts and aggregate to 15. Is that a bug in
> displaying topology summary?
>
> My cluster consists of 2 supervisors and each has 4 workers defined.
>
> Thanks.
>
>
> On Tue, Apr 1, 2014 at 1:43 PM, Nathan Leung  wrote:
>
>> By default supervisor nodes can run up to 4 workers.  This is
>> configurable in storm.yaml (for example see supervisor.slots.ports here:
>> https://github.com/nathanmarz/storm/blob/master/conf/defaults.yaml).
>>  Memory should be split between the workers.  It's a typical Java heap, so
>> anything running on that worker process shares the heap.
>>
>>
>> On Tue, Apr 1, 2014 at 4:10 PM, David Crossland wrote:
>>
>>>  On said subject, how does memory allocation work I these cases?
>>> Assuming 1 worker per node would you just dump all the memory available
>>> into worker.childopts? I guess the memory pool would be shared between the
>>> spawned threads as appropriate to their needs?
>>>
>>>  I'm assuming the equivalent options for supervisor/nimbus are fine
>>> left at defaults.  Given that the workers/spouts/bolts are the working
>>> parts of the topology these would where I should target available memory?
>>>
>>>  D
>>>
>>>   *From:* Huiliang Zhang 
>>> *Sent:* Tuesday, 1 April 2014 19:47
>>> *To:* user@storm.incubator.apache.org
>>>
>>>  Thanks. It should be good if there exist some example figures
>>> explaining the relationship between tasks, workers, and threads.
>>>
>>>
>>> On Sat, Mar 29, 2014 at 6:34 AM, Susheel Kumar Gadalay <
>>> skgada...@gmail.com> wrote:
>>>
 No, a single worker is dedicated to a single topology no matter how
 many threads it spawns for different bolts/spouts.
 A single worker cannot be shared across multiple topologies.

 On 3/29/14, Nathan Leung  wrote:
 > From what I have seen, the second topology is run with 1 worker until
 you
 > kill the first topology or add more worker slots to your cluster.
 >
 >
 > On Sat, Mar 29, 2014 at 2:57 AM, Huiliang Zhang 
 wrote:
 >
 >> Thanks. I am still not clear.
 >>
 >> Do you mean that in a single worker process, there will be multiple
 >> threads and each thread will handle part of a topology? If so, what
 does
 >> the number of workers mean when submitting topology?
 >>
 >>
 >> On Fri, Mar 28, 2014 at 11:18 PM, padma priya chitturi <
 >> padmapriy...@gmail.com> wrote:
 >>
 >>> Hi,
 >>>
 >>> No, its not the case. No matter how many topologies you submit, the
 >>> workers will be shared among the topologies.
 >>>
 >>> Thanks,
 >>> Padma Ch
 >>>
 >>>
 >>> On Sat, Mar 29, 2014 at 5:11 AM, Huiliang Zhang 
 >>> wrote:
 >>>
  Hi,
 
  I have a simple question about storm.
 
  My cluster has just 1 supervisor and 4 ports are defined to run 4
  workers. I first submit a topology which needs 3 workers. Then I
 submit
  another topology which needs 2 workers. Does this mean that the 2nd
  topology will never be run?
 
  Thanks,
  Huiliang
 
 >>>
 >>>
 >>
 >

>>>
>>>
>>
>

Re: Whole topology restarts when a worker crash

[Anuj Kumar]

Great!


On Wed, Apr 2, 2014 at 4:48 PM, Quentin de G. wrote:

> OK, seems all this was a misunderstanding of how Trident works.
>
> I allowed 2 supevisor slots on each machine, but kept 1 worker per
> machine. When I killed machine A, worker from machine A took the unused
> slot on machine B. First worker slot didn't restarted. I then restarted
> machine A and rebalanced the topology. Only one worker was moved.
> This is the behaviour I expected.
>
> To put it in a nutshell : Always reserve empty slots for fault-tolerance.
> Thanks again for the time you took Anuj :)
>
>

Re: java.lang.IllegalArgumentException: timeout value is negative - seen in Worker logs

[Derek Dagit]

This should be fixed with STORM-187 and STORM-259.

They are both merged to master as of March 20.

The two relevant commits are:

d9d637eab5f3761a4768fc960b2574d45962b466
d5dee0ef5f8ad403d1eff05f569c8e9b1e44508c



Introduce your preferred values in a binary search pattern to track down the 
problem configuration item.


The bug fixed with STORM-187 occurs probabilistically, so I would recommend 
pulling in the above fixes if you can rather than doing a binary search of 
config values.
--
Derek

On 4/2/14, 4:13, David Crossland wrote:

This is my netty config, perhaps this will help? Swap yours for this, does the 
problem persist?  Introduce your preferred values in a binary search pattern to 
track down the problem configuration item.  Hopefully that specified below will 
get your topology working again..

storm.messaging.transport: "backtype.storm.messaging.netty.Context"
storm.messaging.netty.buffer_size: 16384
storm.messaging.netty.max_retries: 10
storm.messaging.netty.min_wait_ms: 1000
storm.messaging.netty.max_wait_ms: 5000


D

From: Binita Bharati
Sent: ?Wednesday?, ?2? ?April? ?2014 ?08?:?18
To: user@storm.incubator.apache.org

The same issue is still observed in worker logs. Anyone has any other inputs ?

Thanks

On Tue, Apr 1, 2014 at 8:25 AM, Binita Bharati  wrote:

Thanks , will give it a shot..

On Fri, Mar 28, 2014 at 9:27 AM, bijoy deb  wrote:

Hi,

Try modifying  "storm.messaging.netty.max_retries: 100" to a lower
value,e.g. "storm.messaging.netty.max_retries: 10".

Thanks
Bijoy


On Fri, Mar 28, 2014 at 8:48 AM, Binita Bharati 
wrote:


Hi all,

Am using storm-0.9.0.1.

The following error is seen in the worker logs:

2014-03-25 16:18:24 STDIO [ERROR] Mar 25, 2014 4:18:24 PM
org.jboss.netty.channel.DefaultChannelPipeline
WARNING: An exception was thrown by a user handler while handling an
exception event ([id: 0x8068e4b0] EXCEPTION:
java.net.ConnectException: Connection refused)
java.lang.IllegalArgumentException: timeout value is negative
 at java.lang.Thread.sleep(Native Method)
 at backtype.storm.messaging.netty.Client.reconnect(Client.java:78)
 at
backtype.storm.messaging.netty.StormClientHandler.exceptionCaught(StormClientHandler.java:108)
 at
org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
 at
org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
 at
org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:109)
 at
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:78)
 at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
 at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)


Relevant Netty config from storm.yaml:
  storm.messaging.transport: "backtype.storm.messaging.netty.Context"
  storm.messaging.netty.server_worker_threads: 1
  storm.messaging.netty.client_worker_threads: 1
  storm.messaging.netty.buffer_size: 5242880
  storm.messaging.netty.max_retries: 100
  storm.messaging.netty.max_wait_ms: 1000
  storm.messaging.netty.min_wait_ms: 100

Anyone knows why ?

Thanks
Binita

Re: Storm UI

[David Crossland]

Unluckily I've taken the cluster down now otherwise I would take some 
screenshots.

As soon as I have run one up I will provide some.  But from what I remember it 
seemed to double in Chrome, in IE it was much larger.  It just made it hard to 
know which stats to interpret as correct, although I reasoned it must be when 
the system stats are hidden.

I did not see a tooltip.. I will check that too.

I will reply probably tomorrow with the info

Thanks
David

From: Derek Dagit
Sent: ?Wednesday?, ?2? ?April? ?2014 ?15?:?42
To: user@storm.incubator.apache.org

> it displays different stats depending on whether or not I show/hide system 
> stats.

This is expected.  There should be a tool tip on that button that says 
something like "toggle inclusion of system components."


> How much the stats change seems to differ between Chrome and IE.

That button is setting a cookie's value to true/false.  The math is done on the 
server side and not in the browser, so the difference in the browser used 
should not matter beyond setting the cookie.

If the toggle is not taking effect at all for some browsers, then we should 
create a new Issue to take a look.

--
Derek

On 4/2/14, 8:04, David Crossland wrote:
> I have a curious issue with the UI, it displays different stats depending on 
> whether or not I show/hide system stats.  How much the stats change seems to 
> differ between Chrome and IE.  Is this a known issue?
>
> Thanks
> David
>

Re: Storm UI

[Derek Dagit]

it displays different stats depending on whether or not I show/hide system 
stats.


This is expected.  There should be a tool tip on that button that says something like 
"toggle inclusion of system components."



How much the stats change seems to differ between Chrome and IE.


That button is setting a cookie's value to true/false.  The math is done on the 
server side and not in the browser, so the difference in the browser used 
should not matter beyond setting the cookie.

If the toggle is not taking effect at all for some browsers, then we should 
create a new Issue to take a look.

--
Derek

On 4/2/14, 8:04, David Crossland wrote:

I have a curious issue with the UI, it displays different stats depending on 
whether or not I show/hide system stats.  How much the stats change seems to 
differ between Chrome and IE.  Is this a known issue?

Thanks
David

Storm UI

[David Crossland]

I have a curious issue with the UI, it displays different stats depending on 
whether or not I show/hide system stats.  How much the stats change seems to 
differ between Chrome and IE.  Is this a known issue?

Thanks
David

Tuples lost in Storm 0.9.1

[Daria Mayorova]

 Hi everyone,

We are having some issues with the Storm topology. The problem is that some
tuples are being lost somewhere in the topology. Just after the topology is
deployed, it goes pretty well, but after several hours it starts to loose a
significant amount of tuples.

>From what we've found out from the logs, the thing is that the tuples exit
one bolt/spout, and never enter the next bolt.

Here is some info about the topology:

   - The version is 0.9.1, and netty is used as transport
   - The spout is extending BaseRichSpout, and the bolts extend
   BaseBasicBolt
   - The spout is using Kestrel message queue
   - The cluster consists of 2 nodes: zookeeper, nimbus and ui are running
   on one node, and the workers run on another node. I am attaching the
   content of the config files below. We have also tried running the workers
   on another node (the same where nimbus and zookeeper are), and also on both
   nodes, but the behavior is the same.

According to the Storm UI there are no Failed tuples. Can anybody give any
idea of what might be the reason of the tuples getting lost?

Thanks.

*Storm config (storm.yaml)*
(In case both nodes have workers running, the configuration is the same on
both nodes, just the "storm.local.hostname" parameter changes)

storm.zookeeper.servers:
 - "zkserver1"
nimbus.host: "nimbusserver"
storm.local.dir: "/mnt/storm"
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
storm.local.hostname: "storm1server"

nimbus.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true"
ui.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true"
supervisor.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true"
worker.childopts: "-Xmx3548m -Djava.net.preferIPv4Stack=true"

storm.cluster.mode: "distributed"
storm.local.mode.zmq: false
storm.thrift.transport: "backtype.storm.security.auth.SimpleTransportPlugin"

storm.messaging.transport: "backtype.storm.messaging.netty.Context"

storm.messaging.netty.server_worker_threads: 1
storm.messaging.netty.client_worker_threads: 1
storm.messaging.netty.buffer_size: 5242880 #5MB buffer
storm.messaging.netty.max_retries: 30
storm.messaging.netty.max_wait_ms: 1000
storm.messaging.netty.min_wait_ms: 100

*Zookeeper config (zoo.cfg):*
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/zookeeper
clientPort=2181
autopurge.purgeInterval=24
autopurge.snapRetainCount=5
server.1=localhost:2888:3888

*Topology configuration* passed to the StormSubmitter:
Config conf = new Config();
conf.setNumAckers(6);
conf.setNumWorkers(4);
conf.setMaxSpoutPending(100);


Best regards,
Daria Mayorova

Re: Whole topology restarts when a worker crash

[Quentin de G.]

OK, seems all this was a misunderstanding of how Trident works.

I allowed 2 supevisor slots on each machine, but kept 1 worker per machine.
When I killed machine A, worker from machine A took the unused slot on
machine B. First worker slot didn't restarted. I then restarted machine A
and rebalanced the topology. Only one worker was moved.
This is the behaviour I expected.

To put it in a nutshell : Always reserve empty slots for fault-tolerance.
Thanks again for the time you took Anuj :)

Re: java.lang.IllegalArgumentException: timeout value is negative - seen in Worker logs

[David Crossland]

This is my netty config, perhaps this will help? Swap yours for this, does the 
problem persist?  Introduce your preferred values in a binary search pattern to 
track down the problem configuration item.  Hopefully that specified below will 
get your topology working again..

storm.messaging.transport: "backtype.storm.messaging.netty.Context"
storm.messaging.netty.buffer_size: 16384
storm.messaging.netty.max_retries: 10
storm.messaging.netty.min_wait_ms: 1000
storm.messaging.netty.max_wait_ms: 5000


D

From: Binita Bharati
Sent: ?Wednesday?, ?2? ?April? ?2014 ?08?:?18
To: user@storm.incubator.apache.org

The same issue is still observed in worker logs. Anyone has any other inputs ?

Thanks

On Tue, Apr 1, 2014 at 8:25 AM, Binita Bharati  wrote:
> Thanks , will give it a shot..
>
> On Fri, Mar 28, 2014 at 9:27 AM, bijoy deb  wrote:
>> Hi,
>>
>> Try modifying  "storm.messaging.netty.max_retries: 100" to a lower
>> value,e.g. "storm.messaging.netty.max_retries: 10".
>>
>> Thanks
>> Bijoy
>>
>>
>> On Fri, Mar 28, 2014 at 8:48 AM, Binita Bharati 
>> wrote:
>>>
>>> Hi all,
>>>
>>> Am using storm-0.9.0.1.
>>>
>>> The following error is seen in the worker logs:
>>>
>>> 2014-03-25 16:18:24 STDIO [ERROR] Mar 25, 2014 4:18:24 PM
>>> org.jboss.netty.channel.DefaultChannelPipeline
>>> WARNING: An exception was thrown by a user handler while handling an
>>> exception event ([id: 0x8068e4b0] EXCEPTION:
>>> java.net.ConnectException: Connection refused)
>>> java.lang.IllegalArgumentException: timeout value is negative
>>> at java.lang.Thread.sleep(Native Method)
>>> at backtype.storm.messaging.netty.Client.reconnect(Client.java:78)
>>> at
>>> backtype.storm.messaging.netty.StormClientHandler.exceptionCaught(StormClientHandler.java:108)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
>>> at
>>> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:109)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:78)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>>
>>> Relevant Netty config from storm.yaml:
>>>  storm.messaging.transport: "backtype.storm.messaging.netty.Context"
>>>  storm.messaging.netty.server_worker_threads: 1
>>>  storm.messaging.netty.client_worker_threads: 1
>>>  storm.messaging.netty.buffer_size: 5242880
>>>  storm.messaging.netty.max_retries: 100
>>>  storm.messaging.netty.max_wait_ms: 1000
>>>  storm.messaging.netty.min_wait_ms: 100
>>>
>>> Anyone knows why ?
>>>
>>> Thanks
>>> Binita
>>
>>

Doubt regarding handling transactional failure

[Myidealand account]

Hello

how trident handles hardware failure means is it dependent on message
timeout for failing batch or task heartbeat will inform that node has down
and previous  transaction has failed .and it will reattempt

Restart topology due to memory leak

[Mateusz Dymiński]

Hi all,

Currently I have problem with memory leak somewhere inside my topology.
When topology is up and running for longer time I noticed that is restarted
somewhere in the meantime. My question: Is a way to turn off this
feature(restarting topology when workers are slow) on Nimbus? I would like
to analyze the memory dump after the JVM crash, but when Nimbus is
restarting it, it's hard to analyze proper memory dump.

Or maybe is a option to be informed when nimbus is going to restart it ?

Thanks in advance for any help!

Br,
Mateusz

Re: java.lang.IllegalArgumentException: timeout value is negative - seen in Worker logs

[Binita Bharati]

The same issue is still observed in worker logs. Anyone has any other inputs ?

Thanks

On Tue, Apr 1, 2014 at 8:25 AM, Binita Bharati  wrote:
> Thanks , will give it a shot..
>
> On Fri, Mar 28, 2014 at 9:27 AM, bijoy deb  wrote:
>> Hi,
>>
>> Try modifying  "storm.messaging.netty.max_retries: 100" to a lower
>> value,e.g. "storm.messaging.netty.max_retries: 10".
>>
>> Thanks
>> Bijoy
>>
>>
>> On Fri, Mar 28, 2014 at 8:48 AM, Binita Bharati 
>> wrote:
>>>
>>> Hi all,
>>>
>>> Am using storm-0.9.0.1.
>>>
>>> The following error is seen in the worker logs:
>>>
>>> 2014-03-25 16:18:24 STDIO [ERROR] Mar 25, 2014 4:18:24 PM
>>> org.jboss.netty.channel.DefaultChannelPipeline
>>> WARNING: An exception was thrown by a user handler while handling an
>>> exception event ([id: 0x8068e4b0] EXCEPTION:
>>> java.net.ConnectException: Connection refused)
>>> java.lang.IllegalArgumentException: timeout value is negative
>>> at java.lang.Thread.sleep(Native Method)
>>> at backtype.storm.messaging.netty.Client.reconnect(Client.java:78)
>>> at
>>> backtype.storm.messaging.netty.StormClientHandler.exceptionCaught(StormClientHandler.java:108)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
>>> at
>>> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:109)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:78)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>>
>>> Relevant Netty config from storm.yaml:
>>>  storm.messaging.transport: "backtype.storm.messaging.netty.Context"
>>>  storm.messaging.netty.server_worker_threads: 1
>>>  storm.messaging.netty.client_worker_threads: 1
>>>  storm.messaging.netty.buffer_size: 5242880
>>>  storm.messaging.netty.max_retries: 100
>>>  storm.messaging.netty.max_wait_ms: 1000
>>>  storm.messaging.netty.min_wait_ms: 100
>>>
>>> Anyone knows why ?
>>>
>>> Thanks
>>> Binita
>>
>>

Re: Whole topology restarts when a worker crash

[Quentin de G.]

Hello again, Anuj, and thanks for your time.
I just checked the worker process ID on 169.machine.com.

If I kill a worker on 168.machine.com, there's no change on
169.machine.comworker PID, and tasks on
169.machine.com don't restart.
If I kill the supervisor on 168.machine.com, the worker PID on
169.machine.com changes and all tasks are restarted. I'm not sure if its a
normal behaviour.

Thanks.


On Tue, Apr 1, 2014 at 6:23 PM, Anuj Kumar  wrote:

> Can you check the worker process ID on the 169.machine.com. Does it
> change when you kill the worker on 168.machine.com?
>
>
> On Tue, Apr 1, 2014 at 9:41 PM, Quentin de G. wrote:
>
>> Sure, here's the url of the gist:
>> https://gist.github.com/noKid/e1fdbc582973f5c3e5d6
>>
>> Screenshots:
>>   Before:
>> Storm UI: http://postimg.org/image/dpsxjbjwv/
>> Topology: http://postimg.org/image/664wedx1r/
>> A topology component: http://postimg.org/image/nkp4mnu6n/
>>
>>   After
>> Same topology component: http://postimg.org/image/e2pdmm8i7/
>>
>>
>> I only killed 168.machine.com, yet all the tasks (even those on
>> 169.machine.com) restarted :(
>>
>>
>> On Tue, Apr 1, 2014 at 5:20 PM, Anuj Kumar  wrote:
>>
>>> Can you share the gist of your Trident topology and some screenshots of
>>> the current state and state after you kill the worker and it restarts?
>>>
>>>
>>> On Tue, Apr 1, 2014 at 8:25 PM, Quentin de G. 
>>> wrote:
>>>
 Yes.

>>>
>>>
>>
>>
>> --
>>Quentin de Grandmaison
>>
>>  / 
>> LinkedInprofile
>>  / Profil
>> Viadeo  Mon 
>> CV
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>


-- 
   Quentin de Grandmaison

 / LinkedIn 
profile
/ Profil
Viadeo  Mon CV