Error when submitting toplogy

[David Crossland]

I was investigating an issue with my storm deployment and came across this 
posting

https://groups.google.com/forum/#!msg/storm-user/RXozK0SHlp8/WDIFtp9V1lAJ

More specifically this error

java.io.IOException: . doesn't exist.
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:157)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:147)
at backtype.storm.util$exec_command_BANG_.invoke(util.clj:181)
at backtype.storm.util$extract_dir_from_jar.invoke(util.clj:186)
at backtype.storm.daemon.supervisor$fn__3963.invoke(supervisor.clj:361)
at clojure.lang.MultiFn.invoke(MultiFn.java:171)
at 
backtype.storm.daemon.supervisor$fn__3749$exec_fn__994__auto3750$this__3852.invoke(supervisor.clj:264)
at backtype.storm.event$event_manager$fn__2167$fn__2168.invoke(event.clj:25)
at backtype.storm.event$event_manager$fn__2167.invoke(event.clj:22)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:662)

This had me stumped for some time.  I tried everything suggested in the 
original post to no avail.

I eventually solved the issue.  Basically I had multiple Nimbus daemons running 
in the cluster (this is my first deployment..).

So I now have only 1 daemon running (I also only have one drpc instance running 
on the master).  And after deleting the contents of storms home folder 
(nimbus/supervisor directories) I was able to overcome this issue.

I thought it would be worth posting this (somehow..) hopefully this is the 
right way

David

Advice

[David Crossland]

Hi, I have a 3 node cluster 2 medium, 1 small instances (I'm probably going to 
up this to a medium).  10 Cores total.  My main bottleneck is a service bus 
which has approx. 3.5mil json string messages published to it in a day.  I 
don't seem to be consuming messages at a fast enough rate.

Ive tried modifying the parallelism hint to a number of values, I've tried 
8/20/64/128.. all pretty much stabs in the dark.

I'm looking for some advice as to how to configure this in my environment.  I 
assume there would be some relationship between the number of cores and the 
amount of parallelism I should specify that would ensure best performance and 
throughput.

I wonder also how the number of worker roles fits into this.  Again, I'm taking 
a bit of a stab in choosing 12, one for each slot associated with the 
supervisors.

Any pointers you can give me would be appreciated.

Thanks
David

Re: Advice

[David Crossland]

Hi,

Thanks for the reply.  I played with the topology and I seem to have 'fixed' 
it.  Still, any clarification you can make on this topic would be very useful 
to me.  It would be good to have a definitive way to calculate how many 
workers/Executors/Tasks are most efficient given a set of cores.

I'm rereading the documentation to try and get a clearer picture too.

Regards
David

From: Lajos<mailto:la...@protulae.com>
Sent: ?Sunday?, ?16? ?March? ?2014 ?13?:?31
To: user@storm.incubator.apache.org<mailto:user@storm.incubator.apache.org>

Hi David,

As general advice, you would want one thread per core. You have to then
divide up the threads according to what components have more work to do.
It seems mysterious, but just requires that you first understand what
your components are doing and then do some testing with stats collection.

For example, I've found that a 1:2 ratio works best in some of my
topologies, i.e. the parallelism of the spout is say 2, but that for the
bolt that is doing the work is 4.

Regarding worker processes versus threads, to be honest I haven't yet
seen enough data to say which is more important. At the end of the day,
you just want as little CPU contention as possible for the guys doing
more of the work.

I will post something on my site on this topic in the next day or so;
I'll reply back on this thread when I do.

Cheers,

Lajos



theconsultantcto.com
Enterprise Lucene/Solr

On 16/03/2014 12:59, David Crossland wrote:
> Hi, I have a 3 node cluster 2 medium, 1 small instances (I'm probably
> going to up this to a medium).  10 Cores total.  My main bottleneck is a
> service bus which has approx. 3.5mil json string messages published to
> it in a day.  I don't seem to be consuming messages at a fast enough rate.
>
> Ive tried modifying the parallelism hint to a number of values, I've
> tried 8/20/64/128.. all pretty much stabs in the dark.
>
> I'm looking for some advice as to how to configure this in my
> environment.  I assume there would be some relationship between the
> number of cores and the amount of parallelism I should specify that
> would ensure best performance and throughput.
>
> I wonder also how the number of worker roles fits into this.  Again, I'm
> taking a bit of a stab in choosing 12, one for each slot associated with
> the supervisors.
>
> Any pointers you can give me would be appreciated.
>
> Thanks
> David

Server load - Topology optimization

[David Crossland]

Being very new to storm I'm not sure what to expect in some regards.

Ive been playing about with the number of workers/executors/tasks trying to 
improve throughput on my cluster.  I have a 3 nodes, two 4 core and a 2 core 
node (I can't increase the 3rd node to a medium until the customer gets more 
cores..).  There is a spout that reads from a message bus and a bolt that 
filter out all but the messages we are interested in processing downstream in 
the topology.  Most messages are filtered out.

I'm assuming that these two components require the most resources as they 
should be reading/filtering messages at a constant rate, there are two further 
bolts that are invoked intermittently and hence require less.

Ive set the number of workers to 12 (in fact I've noticed it rarely seems to 
make much difference if I set this to 3/6/9 or 12, there is marginal 
improvement the higher the value).

The spout and filtering bolt I've tried a number of values for in the 
parallelism hint (I started with 1/4/8/16/20/30/40/60/128/256… ) and I can 
barely get the throughput to exceed 3500 messages per minute.  And the larger 
hints just grind the system to a halt.  Currently I have them both set to 20.

The strange thing is that no matter what I do the CPU load is very tiny and is 
typically 80-90% idle.  Suggesting that the topology isn't doing that much 
work.  And tbh I've no idea why this is.  Can anyone offer any suggestions?

If I understand how this should work, given the number of cores I would think 
the total number of executors should total 10? Spawning 1 thread per node, I 
can then set a number of tasks to say 2/4/8 per thread (I've no idea which 
would be most efficient..).  Ive tried something along these lines and my 
throughput was significantly less, approx. 2000 messages per minute.  In fact 
parallelism hint of 20/24/30 seem to have produced the most throughput.

All help/suggestions gratefully received!

Thanks
David

Re: Server load - Topology optimization

[David Crossland]

Could my issue relate to memory allocated to the JVM? Most of the setting are 
pretty much the defaults.  Are there any other settings that could be 
throttling the topology?

I'd like to be able to identify the issue without all this constant "stabbing 
in the dark"... ?

D

From: David Crossland<mailto:da...@elastacloud.com>
Sent: ?Tuesday?, ?18? ?March? ?2014 ?16?:?32
To: user@storm.incubator.apache.org<mailto:user@storm.incubator.apache.org>

Being very new to storm I'm not sure what to expect in some regards.

Ive been playing about with the number of workers/executors/tasks trying to 
improve throughput on my cluster.  I have a 3 nodes, two 4 core and a 2 core 
node (I can't increase the 3rd node to a medium until the customer gets more 
cores..).  There is a spout that reads from a message bus and a bolt that 
filter out all but the messages we are interested in processing downstream in 
the topology.  Most messages are filtered out.

I'm assuming that these two components require the most resources as they 
should be reading/filtering messages at a constant rate, there are two further 
bolts that are invoked intermittently and hence require less.

Ive set the number of workers to 12 (in fact I've noticed it rarely seems to 
make much difference if I set this to 3/6/9 or 12, there is marginal 
improvement the higher the value).

The spout and filtering bolt I've tried a number of values for in the 
parallelism hint (I started with 1/4/8/16/20/30/40/60/128/256... ) and I can 
barely get the throughput to exceed 3500 messages per minute.  And the larger 
hints just grind the system to a halt.  Currently I have them both set to 20.

The strange thing is that no matter what I do the CPU load is very tiny and is 
typically 80-90% idle.  Suggesting that the topology isn't doing that much 
work.  And tbh I've no idea why this is.  Can anyone offer any suggestions?

If I understand how this should work, given the number of cores I would think 
the total number of executors should total 10? Spawning 1 thread per node, I 
can then set a number of tasks to say 2/4/8 per thread (I've no idea which 
would be most efficient..).  Ive tried something along these lines and my 
throughput was significantly less, approx. 2000 messages per minute.  In fact 
parallelism hint of 20/24/30 seem to have produced the most throughput.

All help/suggestions gratefully received!

Thanks
David

Re: Server load - Topology optimization

[David Crossland]

A bit more information then

There are 4 components

Spout - This is reading from an azure service bus topic/subscription.  A 
connection is created in the open() method of the spout, nextTuple does a peek 
on the message, and invokes the following code;

StringWriter writer = new StringWriter();
IOUtils.copy(message.getBody(), writer);
String messageBody = writer.toString();

It then deletes the message from the queue.

Overall nothing all that exciting..

Bolt 1 - Filtering

Parses the message body (json string) and converts it to an object 
representation.  Filters out anything that isn't a monetise message.  It then 
emits the monetise message object to the next bolt.  Monetise messages account 
for ~ 0.03% of the total message volume.

Bolt 2 - transformation

Basically extracts from the monetise object the values that are interesting and 
contracts a string which it emits

Bolt 3 - Storage

Stores the transformed string in Redis using the current date/time as key.

-

Shuffle grouping is used with the topology

I ack every tuple irrespective of whether I emit the tuple or not.  It should 
not be attempting to replay tuple.

-

I don't think Bolt 2/3 are the cause of the bottleneck.  They don't have to 
process much data at all tbh.

I can accept that perhaps there is something inefficient with the spout, 
perhaps it just can't read from the service bus quickly enough. I will do some 
more research on this and have a chat with the colleague who wrote this 
component.

I suppose I'm just trying to identify if I've configured something incorrectly 
with respect to storm, whether I'm correct to relate the total number of 
executors and tasks to the total number of cores I have available.  I find it 
strange that I get a better throughput when I choose an arbitrary large number 
for the parallelism hint than if I constrain myself to a maximum that equates 
to the number of cores.

D

From: Nathan Leung<mailto:ncle...@gmail.com>
Sent: ‎Tuesday‎, ‎18‎ ‎March‎ ‎2014 ‎18‎:‎38
To: user@storm.incubator.apache.org<mailto:user@storm.incubator.apache.org>

In my experience storm is able to make good use of CPU resources, if the 
application is written appropriately.  You shouldn't require too much executor 
parallelism if your application is CPU intensive.  If your bolts are doing 
things like remote DB/NoSQL accesses, then that changes things and 
parallelizing bolts will give you more throughput.  Not knowing your 
application, the best way to pin down the problem is to simplify your topology. 
 Cut out everything except for the Spout.  How is your filtering done?  if you 
return without emitting, the latest versions of storm will sleep before trying 
again.  It may be worthwhile to loop in the spout until you receive a valid 
message, or the bus is empty.  How much throughput can you achieve from the 
spout, emitting a tuple into the ether?  Maybe the problem is your message bus. 
 Once you have achieve a level of performance you are satisfied from the spout, 
add one bolt.  What bottlenecks does the bolt introduce?  etc etc.


On Tue, Mar 18, 2014 at 2:31 PM, David Crossland 
mailto:da...@elastacloud.com>> wrote:
Could my issue relate to memory allocated to the JVM? Most of the setting are 
pretty much the defaults.  Are there any other settings that could be 
throttling the topology?

I'd like to be able to identify the issue without all this constant “stabbing 
in the dark”… 😃

D

From: David Crossland<mailto:da...@elastacloud.com>
Sent: ‎Tuesday‎, ‎18‎ ‎March‎ ‎2014 ‎16‎:‎32
To: user@storm.incubator.apache.org<mailto:user@storm.incubator.apache.org>

Being very new to storm I'm not sure what to expect in some regards.

Ive been playing about with the number of workers/executors/tasks trying to 
improve throughput on my cluster.  I have a 3 nodes, two 4 core and a 2 core 
node (I can't increase the 3rd node to a medium until the customer gets more 
cores..).  There is a spout that reads from a message bus and a bolt that 
filter out all but the messages we are interested in processing downstream in 
the topology.  Most messages are filtered out.

I'm assuming that these two components require the most resources as they 
should be reading/filtering messages at a constant rate, there are two further 
bolts that are invoked intermittently and hence require less.

Ive set the number of workers to 12 (in fact I've noticed it rarely seems to 
make much difference if I set this to 3/6/9 or 12, there is marginal 
improvement the higher the value).

The spout and filtering bolt I've tried a number of values for in the 
parallelism hint (I started with 1/4/8/16/20/30/40/60/128/256… ) and I can 
barely get the throughput to exceed 3500 messages per minute.  And the larger 
hints just grind the system to a halt.  Currently I have them both set to 20.

T

Re: Server load - Topology optimization

[David Crossland]

Message filtering occurs only in the first bolt.  The spout will emit any 
message it encounters off the queue.

It is possible that the spout will find no messages on the queue, if its been 
emptied, in which case no tuple is emitted.

D

From: Link Wang<mailto:link.wang...@gmail.com>
Sent: ‎Tuesday‎, ‎18‎ ‎March‎ ‎2014 ‎23‎:‎55
To: David Crossland<mailto:da...@elastacloud.com>
Cc: user@storm.incubator.apache.org<mailto:user@storm.incubator.apache.org>

Do you filter out message in the nextTuple method and with no tuple sent at one 
time it was called?
There's a internal mechanism that storm will sleep 1ms to call nextTuple if 
this calling of nextTuple sent nothing.

2014年3月19日 上午7:14于 David Crossland 写道：
>
> Perhaps these screenshots might shed some light? I don't think there is much 
> of a latency issue.  I'm really starting to suspect there is some consumption 
> rate issue from the topic.
>
> I set the spout to a high parallelism value as it did seem to improve 
> throughput..
>
> But if there is anything you can spot that would be grand
>
> Thanks
> David
>
> From: Nathan Leung
> Sent: ‎Tuesday‎, ‎18‎ ‎March‎ ‎2014 ‎21‎:‎14
> To: user@storm.incubator.apache.org
>
> It could be bolt 3.  What is the latency like between your worker and your 
> redis server?  Increasing the number of threads for bolt 3 will likely 
> increase your throughput.  Bolt 1 and 2 are probably CPU bound, but bolt 3 is 
> probably restricted by your network access.  Also I've found that 
> localOrShuffleGrouping can improve performance due to reduced network 
> communications.
>
>
> On Tue, Mar 18, 2014 at 3:55 PM, David Crossland  
> wrote:
>>
>> A bit more information then
>>
>> There are 4 components
>>
>> Spout - This is reading from an azure service bus topic/subscription.  A 
>> connection is created in the open() method of the spout, nextTuple does a 
>> peek on the message, and invokes the following code;
>>
>> StringWriter writer = new StringWriter();
>> IOUtils.copy(message.getBody(), writer);
>> String messageBody = writer.toString();
>>
>> It then deletes the message from the queue.
>>
>> Overall nothing all that exciting..
>>
>> Bolt 1 - Filtering
>>
>> Parses the message body (json string) and converts it to an object 
>> representation.  Filters out anything that isn't a monetise message.  It 
>> then emits the monetise message object to the next bolt.  Monetise messages 
>> account for ~ 0.03% of the total message volume.
>>
>> Bolt 2 - transformation
>>
>> Basically extracts from the monetise object the values that are interesting 
>> and contracts a string which it emits
>>
>> Bolt 3 - Storage
>>
>> Stores the transformed string in Redis using the current date/time as key.
>>
>> -
>>
>> Shuffle grouping is used with the topology
>>
>> I ack every tuple irrespective of whether I emit the tuple or not.  It 
>> should not be attempting to replay tuple.
>>
>> -
>>
>> I don't think Bolt 2/3 are the cause of the bottleneck.  They don't have to 
>> process much data at all tbh.
>>
>> I can accept that perhaps there is something inefficient with the spout, 
>> perhaps it just can't read from the service bus quickly enough. I will do 
>> some more research on this and have a chat with the colleague who wrote this 
>> component.
>>
>> I suppose I'm just trying to identify if I've configured something 
>> incorrectly with respect to storm, whether I'm correct to relate the total 
>> number of executors and tasks to the total number of cores I have available. 
>>  I find it strange that I get a better throughput when I choose an arbitrary 
>> large number for the parallelism hint than if I constrain myself to a 
>> maximum that equates to the number of cores.
>>
>> D
>>
>> From: Nathan Leung
>> Sent: ‎Tuesday‎, ‎18‎ ‎March‎ ‎2014 ‎18‎:‎38
>> To: user@storm.incubator.apache.org
>>
>> In my experience storm is able to make good use of CPU resources, if the 
>> application is written appropriately.  You shouldn't require too much 
>> executor parallelism if your application is CPU intensive.  If your bolts 
>> are doing things like remote DB/NoSQL accesses, then that changes things and 
>> parallelizing bolts will give you more throughput.  Not knowing your 
>> application, the best way to pin down the problem is to simplify your 
>> topology.  Cut out everything except for the Spout.  How is your

Re: Server load - Topology optimization

[David Crossland]

I had noticed the ack count never increased for the spout.

The spout is a BaseRichSpout, I presumed the underlying code should handle the 
ack and display appropriate metrics. But you can see from the first bolt that 
all messages have been asked.. I don't know if this is an issue.

Ive been doing a bit of reading around the service bus, if I understand 
correctly it can push 20 messages per second per connection (I need to confirm 
this with a colleague who has the experience with this..) If that's the case 
then it tallies with the throughput I've been seeing (approx…).  I suspect the 
problem isn't so much the topology, but the service bus itself.

I'll come back if this doesn't pan out.

D

From: Sean Zhong<mailto:clock...@gmail.com>
Sent: ‎Wednesday‎, ‎19‎ ‎March‎ ‎2014 ‎00‎:‎45
To: user@storm.incubator.apache.org<mailto:user@storm.incubator.apache.org>

The Spout is suspicous.

From the screenshots, you are using no-acked spout, there may also be GC issue 
there.
Here is the suggestion:

Check whether you are making connection in the context of nextTuple. If that is 
true, it means a large latency. You can check the spout latency by enabling 
acked spout(when emmit, add a messageId. e.g. emit(new Values(msg), messageId), 
and check the latency.
If this is the case, you need to create a seperate thread for data connection 
in spout, and use a queue to buffer messages, then nextTuple just pol the queue.



On Wed, Mar 19, 2014 at 8:06 AM, Michael Rose 
mailto:mich...@fullcontact.com>> wrote:
Well, I see you have 30 spouts instances and 3 bolt instances. Doesn't seem 
like it's a huge bottleneck, but it is having to send over the wire much of the 
time. Something to keep in mind for the future.

I'd be most suspicious of your spouts. All spouts on a single worker are run 
single-threaded (in an event loop calling nextTuple()), so if you have ANY 
blocking work that'll kill throughput. If AzureServiceBus is anything like SQS, 
response times are not instant. We tend to have background threads feeding a 
queue in our spouts for that reason, as we use SQS for many of our topologies. 
Given 12 workers, you'd have ~3 spouts per machine running single-threaded.

With spouts, you should attempt to maintain as little blocking work as 
possible...if you're using a queue, you should be using Queue#poll() to either 
return a value OR null, and only emit a tuple if an event is available (and 
meets your criteria). Storm will handle rate limiting the spouts with sleeps.


Michael Rose (@Xorlev<https://twitter.com/xorlev>)
Senior Platform Engineer, FullContact<http://www.fullcontact.com/>
mich...@fullcontact.com<mailto:mich...@fullcontact.com>


On Tue, Mar 18, 2014 at 5:14 PM, David Crossland 
mailto:da...@elastacloud.com>> wrote:
Perhaps these screenshots might shed some light? I don't think there is much of 
a latency issue.  I'm really starting to suspect there is some consumption rate 
issue from the topic.

I set the spout to a high parallelism value as it did seem to improve 
throughput..

But if there is anything you can spot that would be grand

Thanks
David

From: Nathan Leung<mailto:ncle...@gmail.com>
Sent: ‎Tuesday‎, ‎18‎ ‎March‎ ‎2014 ‎21‎:‎14
To: user@storm.incubator.apache.org<mailto:user@storm.incubator.apache.org>

It could be bolt 3.  What is the latency like between your worker and your 
redis server?  Increasing the number of threads for bolt 3 will likely increase 
your throughput.  Bolt 1 and 2 are probably CPU bound, but bolt 3 is probably 
restricted by your network access.  Also I've found that localOrShuffleGrouping 
can improve performance due to reduced network communications.


On Tue, Mar 18, 2014 at 3:55 PM, David Crossland 
mailto:da...@elastacloud.com>> wrote:
A bit more information then

There are 4 components

Spout - This is reading from an azure service bus topic/subscription.  A 
connection is created in the open() method of the spout, nextTuple does a peek 
on the message, and invokes the following code;

StringWriter writer = new StringWriter();
IOUtils.copy(message.getBody(), writer);
String messageBody = writer.toString();

It then deletes the message from the queue.

Overall nothing all that exciting..

Bolt 1 - Filtering

Parses the message body (json string) and converts it to an object 
representation.  Filters out anything that isn't a monetise message.  It then 
emits the monetise message object to the next bolt.  Monetise messages account 
for ~ 0.03% of the total message volume.

Bolt 2 - transformation

Basically extracts from the monetise object the values that are interesting and 
contracts a string which it emits

Bolt 3 - Storage

Stores the transformed string in Redis using the current date/time as key.

-

Shuffle grouping is used with the topology

I ack every tuple irr

Re: Server load - Topology optimization

[David Crossland]

Where is this messageId derived from?

I note there is a tuple.getMessageId() but there does not appear to be an 
overload to emit that accepts this?

There is an overload

emit<http://nathanmarz.github.io/storm/doc/backtype/storm/task/OutputCollector.html#emit(java.lang.String,%20java.util.List)>(java.lang.String
 streamId, java.util.List tuple)

I don't think this is what you are referring to however.

My bolt execute method looks like this;  (sorry if I'm being obtuse, this is 
still very new.. )

public void execute(Tuple tuple) {
String msg = (String)tuple.getValue(0);
if(msg == null)
{
logger.log(Level.ERROR, "Message is null");
//acknowledge the tuple has failed and return
_collector.fail(tuple);
return;
}

MessageProcessor processor = new MessageProcessor();
EventMessage message = processor.processMessage(msg);

if(message == null)
{
logger.log(Level.DEBUG, "Message did not conform to a known event");
logger.log(Level.DEBUG, msg);
//acknowlege the tuple, but dont do anything with it, we dont have 
to emit
_collector.ack(tuple);
}

if(message instanceof MonetiseEvent)
{
logger.log(Level.DEBUG, "recieved monetise message");
_collector.emit(new Values(message));
_collector.ack(tuple);
}
}


D

From: Sean Zhong<mailto:clock...@gmail.com>
Sent: ‎Wednesday‎, ‎19‎ ‎March‎ ‎2014 ‎15‎:‎39
To: user@storm.incubator.apache.org<mailto:user@storm.incubator.apache.org>

Hi David,


As I said, to get the ack count for spout, you need to add a message Id when 
emitting. Likes this,
collector.emit(new Values(msg), messageId)
 ^| here

If the message Id is null, then there will be ack message sent to spout.


Sean



On Wed, Mar 19, 2014 at 4:49 PM, David Crossland 
mailto:da...@elastacloud.com>> wrote:
I had noticed the ack count never increased for the spout.

The spout is a BaseRichSpout, I presumed the underlying code should handle the 
ack and display appropriate metrics. But you can see from the first bolt that 
all messages have been asked.. I don't know if this is an issue.

Ive been doing a bit of reading around the service bus, if I understand 
correctly it can push 20 messages per second per connection (I need to confirm 
this with a colleague who has the experience with this..) If that's the case 
then it tallies with the throughput I've been seeing (approx…).  I suspect the 
problem isn't so much the topology, but the service bus itself.

I'll come back if this doesn't pan out.

D

From: Sean Zhong<mailto:clock...@gmail.com>
Sent: ‎Wednesday‎, ‎19‎ ‎March‎ ‎2014 ‎00‎:‎45
To: user@storm.incubator.apache.org<mailto:user@storm.incubator.apache.org>

The Spout is suspicous.

>From the screenshots, you are using no-acked spout, there may also be GC issue 
>there.
Here is the suggestion:

Check whether you are making connection in the context of nextTuple. If that is 
true, it means a large latency. You can check the spout latency by enabling 
acked spout(when emmit, add a messageId. e.g. emit(new Values(msg), messageId), 
and check the latency.
If this is the case, you need to create a seperate thread for data connection 
in spout, and use a queue to buffer messages, then nextTuple just pol the queue.



On Wed, Mar 19, 2014 at 8:06 AM, Michael Rose 
mailto:mich...@fullcontact.com>> wrote:
Well, I see you have 30 spouts instances and 3 bolt instances. Doesn't seem 
like it's a huge bottleneck, but it is having to send over the wire much of the 
time. Something to keep in mind for the future.

I'd be most suspicious of your spouts. All spouts on a single worker are run 
single-threaded (in an event loop calling nextTuple()), so if you have ANY 
blocking work that'll kill throughput. If AzureServiceBus is anything like SQS, 
response times are not instant. We tend to have background threads feeding a 
queue in our spouts for that reason, as we use SQS for many of our topologies. 
Given 12 workers, you'd have ~3 spouts per machine running single-threaded.

With spouts, you should attempt to maintain as little blocking work as 
possible...if you're using a queue, you should be using Queue#poll() to either 
return a value OR null, and only emit a tuple if an event is available (and 
meets your criteria). Storm will handle rate limiting the spouts with sleeps.


Michael Rose (@Xorlev<https://twitter.com/xorlev>)
Senior Platform Engineer, FullContact<http://www.fullcontact.com/>
mich...@fullcontact.com<mailto:mich...@fullcontact.com>


On Tue, Mar 18, 2014 at 5:14 PM, David Crossland 
mailto:da...@elastacloud.com>> wrote:
Perhaps these screenshots might shed s

Re: Server load - Topology optimization

[David Crossland]

I was looking at the documentation last night, I understand I need only specify 
some message id in the spout. Is this correct?

I would expect every message would need a unique id, i suppose a UUID could be 
used?

David

From: Sean Zhong<mailto:clock...@gmail.com>
Sent: ?Thursday?, ?20? ?March? ?2014 ?01?:?52
To: user@storm.incubator.apache.org<mailto:user@storm.incubator.apache.org>


What is the type of _collector?

If it is SpoutOutputCollector, you can change from:
if(message instanceof MonetiseEvent)
{
logger.log(Level.DEBUG, "recieved monetise message");
_collector.emit(new Values(message));
_collector.ack(tuple);
}

to
  Integer msgId = new Integer(0); //can be any object
if(message instanceof MonetiseEvent)
{
logger.log(Level.DEBUG, "recieved monetise message");
_collector.emit(new Values(message), msgId);
_collector.ack(tuple);
}



On Thu, Mar 20, 2014 at 12:18 AM, David Crossland 
mailto:da...@elastacloud.com>> wrote:
_collector.emit(new Values(message));

Re: Server load - Topology optimization

[David Crossland]

Thanks for this, I've taken a look at this over the past couple of days.  If I 
follow it correctly (and I'm not certain I do..) I should get 20 messages per 
second per connection.  This tallies with what I see when I have a single 
executor/task.

As the service bus supports up to 100 connections experimented by raising the 
number of executors to 100, which does increase the throughput to approx. 60 
messages per second, which is not ideal.  And I imagine with so many thread 
that the amount of contention is reducing the efficiency of the system..

Ive tried in previous experiments to limit the number of executors to the 
number of cores available, but I still can't seem to raise the throughput to a 
meaningful level..

As I say I will be having a chat with a colleague who has the experience with 
the service bus to try and gain more insight into what can be done, if 
anything..

D

From: Milinda Pathirage<mailto:mpath...@umail.iu.edu>
Sent: ‎Thursday‎, ‎20‎ ‎March‎ ‎2014 ‎03‎:‎31
To: user@storm.incubator.apache.org<mailto:user@storm.incubator.apache.org>

I think its worth exploring throughput you can achieve using Azure
Service Bus Queue first. Most probably this could be a problem in the
way you access Azure Queues. This [1] may help.

Thanks
Milinda


[1] http://msdn.microsoft.com/en-us/library/windowsazure/hh528527.aspx

On Wed, Mar 19, 2014 at 9:55 PM, Michael Rose  wrote:
> messageId is any unique identifier of the message, such that when ack is
> called on your spout you're returned the identifier to then mark the work as
> complete in the source in the case it supports replay.
>
> Michael Rose (@Xorlev)
> Senior Platform Engineer, FullContact
> mich...@fullcontact.com
>
>
>
> On Wed, Mar 19, 2014 at 10:18 AM, David Crossland 
> wrote:
>>
>> Where is this messageId derived from?
>>
>> I note there is a tuple.getMessageId() but there does not appear to be an
>> overload to emit that accepts this?
>>
>> There is an overload
>>
>> emit(java.lang.String streamId, java.util.List tuple)
>>
>> I don't think this is what you are referring to however.
>>
>> My bolt execute method looks like this;  (sorry if I'm being obtuse, this
>> is still very new.. )
>>
>> public void execute(Tuple tuple) {
>> String msg = (String)tuple.getValue(0);
>> if(msg == null)
>> {
>> logger.log(Level.ERROR, "Message is null");
>> //acknowledge the tuple has failed and return
>> _collector.fail(tuple);
>> return;
>> }
>>
>> MessageProcessor processor = new MessageProcessor();
>> EventMessage message = processor.processMessage(msg);
>>
>> if(message == null)
>> {
>> logger.log(Level.DEBUG, "Message did not conform to a known
>> event");
>> logger.log(Level.DEBUG, msg);
>> //acknowlege the tuple, but dont do anything with it, we dont
>> have to emit
>> _collector.ack(tuple);
>> }
>>
>> if(message instanceof MonetiseEvent)
>> {
>> logger.log(Level.DEBUG, "recieved monetise message");
>> _collector.emit(new Values(message));
>> _collector.ack(tuple);
>> }
>> }
>>
>>
>> D
>>
>> From: Sean Zhong
>> Sent: ‎Wednesday‎, ‎19‎ ‎March‎ ‎2014 ‎15‎:‎39
>> To: user@storm.incubator.apache.org
>>
>> Hi David,
>>
>>
>> As I said, to get the ack count for spout, you need to add a message Id
>> when emitting. Likes this,
>> collector.emit(new Values(msg), messageId)
>>  ^| here
>>
>> If the message Id is null, then there will be ack message sent to spout.
>>
>>
>> Sean
>>
>>
>>
>> On Wed, Mar 19, 2014 at 4:49 PM, David Crossland 
>> wrote:
>>>
>>> I had noticed the ack count never increased for the spout.
>>>
>>> The spout is a BaseRichSpout, I presumed the underlying code should
>>> handle the ack and display appropriate metrics. But you can see from the
>>> first bolt that all messages have been asked.. I don't know if this is an
>>> issue.
>>>
>>> Ive been doing a bit of reading around the service bus, if I understand
>>> correctly it can push 20 messages per second per connection (I need to
>>> confirm this with a colleague who has the experience with this..) If that's
>>> the case then it tallies with the throughput I'v

Re: Server load - Topology optimization

[David Crossland]

I added a message id in the spout, now I get acks on my spout, thanks for 
that.. looks like I've got a 1267.486 latency on completion.. time to reduce 
the number of executors methinks..

D

From: Sean Zhong<mailto:clock...@gmail.com>
Sent: ?Thursday?, ?20? ?March? ?2014 ?01?:?52
To: user@storm.incubator.apache.org<mailto:user@storm.incubator.apache.org>


What is the type of _collector?

If it is SpoutOutputCollector, you can change from:
if(message instanceof MonetiseEvent)
{
logger.log(Level.DEBUG, "recieved monetise message");
_collector.emit(new Values(message));
_collector.ack(tuple);
}

to
  Integer msgId = new Integer(0); //can be any object
if(message instanceof MonetiseEvent)
{
logger.log(Level.DEBUG, "recieved monetise message");
_collector.emit(new Values(message), msgId);
_collector.ack(tuple);
}



On Thu, Mar 20, 2014 at 12:18 AM, David Crossland 
mailto:da...@elastacloud.com>> wrote:
_collector.emit(new Values(message));

Curiosity

[David Crossland]

I'm just wondering about the storm.yaml settings, should these be replicated to 
all nodes in the topology or need it only be configured on the “master” node 
where I deploy the topology?

thanks
David

Re: Curiosity

[David Crossland]

That's a pain, thanks.  Its either that then or configure it in code... either 
way I either have to recompile or touch each node..

Still thanks for the info xD

D

From: Jon Logan<mailto:jmlo...@buffalo.edu>
Sent: ?Tuesday?, ?1? ?April? ?2014 ?16?:?43
To: user@storm.incubator.apache.org<mailto:user@storm.incubator.apache.org>

All nodes.


On Tue, Apr 1, 2014 at 11:41 AM, David Crossland 
mailto:da...@elastacloud.com>> wrote:
I'm just wondering about the storm.yaml settings, should these be replicated to 
all nodes in the topology or need it only be configured on the "master" node 
where I deploy the topology?

thanks
David

Re: Curiosity

[David Crossland]

Appreciated, we will be looking at tools like this I the future.  Atm, I'm just 
trying to tune to topology, as I'll be modifying code anyway I'll just stick 
the config mods in there.

D

From: Patricio Echag?e<mailto:patric...@gmail.com>
Sent: ?Tuesday?, ?1? ?April? ?2014 ?17?:?22
To: user@storm.incubator.apache.org<mailto:user@storm.incubator.apache.org>


That's where chef or puppet come handy.

Sent from my Nexus 4.

On Apr 1, 2014 9:20 AM, "David Crossland" 
mailto:da...@elastacloud.com>> wrote:
That's a pain, thanks.  Its either that then or configure it in code... either 
way I either have to recompile or touch each node..

Still thanks for the info xD

D

From: Jon Logan<mailto:jmlo...@buffalo.edu>
Sent: Tuesday, 1 April 2014 16:43
To: user@storm.incubator.apache.org<mailto:user@storm.incubator.apache.org>

All nodes.


On Tue, Apr 1, 2014 at 11:41 AM, David Crossland 
mailto:da...@elastacloud.com>> wrote:
I'm just wondering about the storm.yaml settings, should these be replicated to 
all nodes in the topology or need it only be configured on the "master" node 
where I deploy the topology?

thanks
David

Re: Basic storm question

[David Crossland]

On said subject, how does memory allocation work I these cases? Assuming 1 
worker per node would you just dump all the memory available into 
worker.childopts? I guess the memory pool would be shared between the spawned 
threads as appropriate to their needs?

I'm assuming the equivalent options for supervisor/nimbus are fine left at 
defaults.  Given that the workers/spouts/bolts are the working parts of the 
topology these would where I should target available memory?

D

From: Huiliang Zhang
Sent: ?Tuesday?, ?1? ?April? ?2014 ?19?:?47
To: user@storm.incubator.apache.org

Thanks. It should be good if there exist some example figures explaining the 
relationship between tasks, workers, and threads.


On Sat, Mar 29, 2014 at 6:34 AM, Susheel Kumar Gadalay 
mailto:skgada...@gmail.com>> wrote:
No, a single worker is dedicated to a single topology no matter how
many threads it spawns for different bolts/spouts.
A single worker cannot be shared across multiple topologies.

On 3/29/14, Nathan Leung mailto:ncle...@gmail.com>> wrote:
> From what I have seen, the second topology is run with 1 worker until you
> kill the first topology or add more worker slots to your cluster.
>
>
> On Sat, Mar 29, 2014 at 2:57 AM, Huiliang Zhang 
> mailto:zhl...@gmail.com>> wrote:
>
>> Thanks. I am still not clear.
>>
>> Do you mean that in a single worker process, there will be multiple
>> threads and each thread will handle part of a topology? If so, what does
>> the number of workers mean when submitting topology?
>>
>>
>> On Fri, Mar 28, 2014 at 11:18 PM, padma priya chitturi <
>> padmapriy...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> No, its not the case. No matter how many topologies you submit, the
>>> workers will be shared among the topologies.
>>>
>>> Thanks,
>>> Padma Ch
>>>
>>>
>>> On Sat, Mar 29, 2014 at 5:11 AM, Huiliang Zhang 
>>> mailto:zhl...@gmail.com>>
>>> wrote:
>>>
 Hi,

 I have a simple question about storm.

 My cluster has just 1 supervisor and 4 ports are defined to run 4
 workers. I first submit a topology which needs 3 workers. Then I submit
 another topology which needs 2 workers. Does this mean that the 2nd
 topology will never be run?

 Thanks,
 Huiliang

>>>
>>>
>>
>

Re: java.lang.IllegalArgumentException: timeout value is negative - seen in Worker logs

[David Crossland]

This is my netty config, perhaps this will help? Swap yours for this, does the 
problem persist?  Introduce your preferred values in a binary search pattern to 
track down the problem configuration item.  Hopefully that specified below will 
get your topology working again..

storm.messaging.transport: "backtype.storm.messaging.netty.Context"
storm.messaging.netty.buffer_size: 16384
storm.messaging.netty.max_retries: 10
storm.messaging.netty.min_wait_ms: 1000
storm.messaging.netty.max_wait_ms: 5000


D

From: Binita Bharati
Sent: ?Wednesday?, ?2? ?April? ?2014 ?08?:?18
To: user@storm.incubator.apache.org

The same issue is still observed in worker logs. Anyone has any other inputs ?

Thanks

On Tue, Apr 1, 2014 at 8:25 AM, Binita Bharati  wrote:
> Thanks , will give it a shot..
>
> On Fri, Mar 28, 2014 at 9:27 AM, bijoy deb  wrote:
>> Hi,
>>
>> Try modifying  "storm.messaging.netty.max_retries: 100" to a lower
>> value,e.g. "storm.messaging.netty.max_retries: 10".
>>
>> Thanks
>> Bijoy
>>
>>
>> On Fri, Mar 28, 2014 at 8:48 AM, Binita Bharati 
>> wrote:
>>>
>>> Hi all,
>>>
>>> Am using storm-0.9.0.1.
>>>
>>> The following error is seen in the worker logs:
>>>
>>> 2014-03-25 16:18:24 STDIO [ERROR] Mar 25, 2014 4:18:24 PM
>>> org.jboss.netty.channel.DefaultChannelPipeline
>>> WARNING: An exception was thrown by a user handler while handling an
>>> exception event ([id: 0x8068e4b0] EXCEPTION:
>>> java.net.ConnectException: Connection refused)
>>> java.lang.IllegalArgumentException: timeout value is negative
>>> at java.lang.Thread.sleep(Native Method)
>>> at backtype.storm.messaging.netty.Client.reconnect(Client.java:78)
>>> at
>>> backtype.storm.messaging.netty.StormClientHandler.exceptionCaught(StormClientHandler.java:108)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
>>> at
>>> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:109)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:78)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>>
>>> Relevant Netty config from storm.yaml:
>>>  storm.messaging.transport: "backtype.storm.messaging.netty.Context"
>>>  storm.messaging.netty.server_worker_threads: 1
>>>  storm.messaging.netty.client_worker_threads: 1
>>>  storm.messaging.netty.buffer_size: 5242880
>>>  storm.messaging.netty.max_retries: 100
>>>  storm.messaging.netty.max_wait_ms: 1000
>>>  storm.messaging.netty.min_wait_ms: 100
>>>
>>> Anyone knows why ?
>>>
>>> Thanks
>>> Binita
>>
>>

Storm UI

[David Crossland]

I have a curious issue with the UI, it displays different stats depending on 
whether or not I show/hide system stats.  How much the stats change seems to 
differ between Chrome and IE.  Is this a known issue?

Thanks
David

Re: Storm UI

[David Crossland]

Unluckily I've taken the cluster down now otherwise I would take some 
screenshots.

As soon as I have run one up I will provide some.  But from what I remember it 
seemed to double in Chrome, in IE it was much larger.  It just made it hard to 
know which stats to interpret as correct, although I reasoned it must be when 
the system stats are hidden.

I did not see a tooltip.. I will check that too.

I will reply probably tomorrow with the info

Thanks
David

From: Derek Dagit<mailto:der...@yahoo-inc.com>
Sent: ?Wednesday?, ?2? ?April? ?2014 ?15?:?42
To: user@storm.incubator.apache.org<mailto:user@storm.incubator.apache.org>

> it displays different stats depending on whether or not I show/hide system 
> stats.

This is expected.  There should be a tool tip on that button that says 
something like "toggle inclusion of system components."


> How much the stats change seems to differ between Chrome and IE.

That button is setting a cookie's value to true/false.  The math is done on the 
server side and not in the browser, so the difference in the browser used 
should not matter beyond setting the cookie.

If the toggle is not taking effect at all for some browsers, then we should 
create a new Issue to take a look.

--
Derek

On 4/2/14, 8:04, David Crossland wrote:
> I have a curious issue with the UI, it displays different stats depending on 
> whether or not I show/hide system stats.  How much the stats change seems to 
> differ between Chrome and IE.  Is this a known issue?
>
> Thanks
> David
>

Re: storm.yml conflict, which one to use?

[David Crossland]

To my (rather limited) knowledge storm.yaml should be consistent across all 
locations. It can be overridden in your topology jar.

D

From: howard chen
Sent: ?Tuesday?, ?8? ?April? ?2014 ?14?:?28
To: user@storm.incubator.apache.org

Hi,

I deployed storm into different machines

a. nimbus
b. ui
c. supervisor

Let say I want to set the "worker.childopts", if my value of this
setting in the nimbus' storm.yml and supervisor storm.yml are
different, which value will storm cluster use?

I ask this because I have seen that in the storm ui, it is showing the
nimbus value instead of the supervisor value, which is quite
unexpected.

Re: storm.yml conflict, which one to use?

[David Crossland]

In that sense then fair enough, as you are only running UI in one location you 
probably need only configure that parameter on that node.

You mentioned worker.childopts, I assume you have a worker on the same node as 
nimbus then? Even so I would expect this value to be consistent across all 
nodes.

But tbh, you might as well duplicate the same yaml to all nodes, the node will 
ignore any parameters it doesn't need.  And the values it does use will be 
consistent across all nodes.

Again I reiterate I probably don't have a clue what I'm talking about.  To make 
my life simpler I just duplicated the yaml file to all nodes.  This stopped me 
having to manage multiple versions.  I like to make things easy for myself 
where possible.
D

From: howard chen<mailto:howac...@gmail.com>
Sent: ?Tuesday?, ?8? ?April? ?2014 ?15?:?12
To: user@storm.incubator.apache.org<mailto:user@storm.incubator.apache.org>

Hi

Because it does not make sense to duplicate, for example, ui.childopts
in supervisor'storm.yaml right?

So I want to know how the storm.yaml is being used, so I can remove
these duplicates if they are really dummy and not used.

On Tue, Apr 8, 2014 at 9:38 PM, David Crossland  wrote:
> To my (rather limited) knowledge storm.yaml should be consistent across all
> locations. It can be overridden in your topology jar.
>
> D
>
> From: howard chen
> Sent: Tuesday, 8 April 2014 14:28
> To: user@storm.incubator.apache.org
>
> Hi,
>
> I deployed storm into different machines
>
> a. nimbus
> b. ui
> c. supervisor
>
> Let say I want to set the "worker.childopts", if my value of this
> setting in the nimbus' storm.yml and supervisor storm.yml are
> different, which value will storm cluster use?
>
> I ask this because I have seen that in the storm ui, it is showing the
> nimbus value instead of the supervisor value, which is quite
> unexpected.

Re: compiling storm code with maven and using storm client

[David Crossland]

I'm not sure about 1

But 2, you can just copy any topology dependant jars to storms lib directory

D

From: Xing Yong
Sent: ?Monday?, ?14? ?April? ?2014 ?12?:?25
To: user@storm.incubator.apache.org

1. when I compile the storm-0.9.1 release source code with maven, always get 
this error, how to fix it ? thank you.

Compiling backtype.storm.ui.core to 
/home/yongxing/infra-git/storm-0.9.1-incubating/storm-core/target/classes
[INFO] 
[INFO] Reactor Summary:
[INFO]
[INFO] Storm . SUCCESS [3:53.213s]
[INFO] maven-shade-clojure-transformer ... SUCCESS [4.043s]
[INFO] Storm Core  FAILURE [1:57.586s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 5:58.106s
[INFO] Finished at: Mon Apr 14 12:28:08 CST 2014
[INFO] Final Memory: 27M/2030M
[INFO] 
[ERROR] Failed to execute goal 
com.theoryinpractise:clojure-maven-plugin:1.3.18:compile (compile-clojure) on 
project storm-core: Clojure failed. -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
com.theoryinpractise:clojure-maven-plugin:1.3.18:compile (compile-clojure) on 
project storm-core: Clojure failed.
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at 
org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: org.apache.maven.plugin.MojoExecutionException: Clojure failed.
at 
com.theoryinpractise.clojure.AbstractClojureCompilerMojo.callClojureWith(AbstractClojureCompilerMojo.java:451)
at 
com.theoryinpractise.clojure.AbstractClojureCompilerMojo.callClojureWith(AbstractClojureCompilerMojo.java:367)
at 
com.theoryinpractise.clojure.AbstractClojureCompilerMojo.callClojureWith(AbstractClojureCompilerMojo.java:344)
at 
com.theoryinpractise.clojure.ClojureCompilerMojo.execute(ClojureCompilerMojo.java:47)
at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
... 19 more
[ERROR]
[ERROR]

2. Using storm client, how to submit additional files to server, does storm 
client support this operation? for example, i want to transfer some files from 
client to server which my topology will use, someone have any idea to share 
with ?

thank you