Re: storm performance tunning

[Anis Nasir]

https://storm.incubator.apache.org/documentation/FAQ.html


On Sun, Jul 6, 2014 at 1:26 AM, 唐 思成  wrote:

> that really helps，thx a lot
> 在 2014年7月5日，下午8:35，Eugene  写道：
>
> The best tool for topology tuning is Storm UI. Here you can learn your
> process latency and capacity of your storm components.
> Start with some baseline, for example, having all parallelism setting to
> 1. Then gradually increase parallelism until you see capacity numbers well
> below 1.0.
> In general, try to increase number of executors first, before increasing
> number of workers. When increasing number of executors (threads) does not
> give any
> improvements, then start adding additional workers. Also, if your
> components more IO bound then increasing number of task helps as well.
>
> In addition to Storm UI,and resources Francesco pointed out,  you can use
> storm metrics API to send data to graphite.
> http://www.michael-noll.com/blog/2013/11/06/sending-metrics-from-storm-to-graphite/
>
> Upcoming book, Storm Applied   has a
> chapter about tuning Storm (chapter 6).
>
> Thanks
> Eugene Dvorkin
>
>
>
>
>
>
>
> On Sat, Jul 5, 2014 at 7:19 AM,  wrote:
>
>>  i ' ve found some good advices reading these articles
>>
>>
>> http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/
>>
>>
>> http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/
>>
>>
>>
>>
>>
>> Il 2014-07-05 12:12 唐思成 ha scritto:
>>
>> everyone, do you have some best practice or pattern for tunning the storm
>> cluster, like how to set the number of workers, the numbers of spout and
>> bolts, etc..
>> Any advice is welcome.
>>
>> 2014-07-05
>> --
>> 唐思成
>>
>>
>>
>>
>
>
>
> --
> Eugene Dvorkin
> Software Engineer
> New York City Storm User Group - organizer
> WebMD
> email: edvor...@gmail.com
> phone: 646-250-9649
> eugenedvorkin.com
> Connect with me on:
> LinkedIn  Twitter
> 
>
>
>

Re: Unable to read from multiple kafkaSpout

[Anis Nasir]

yes. My question is about parallelism in a Spout. I see only one spout
reading, even when I set the parallelism to more than one.

Regards
Anis


On Sun, Jun 22, 2014 at 1:54 PM,  wrote:

> It's not make sense. I know one SpoutKafka can fetch from one topic
> only(due to config). So I use 12 KafkaSpouts to fetch from 12 different
> topics, and each for parallelism 1 or 2 only. It works(I can see 12 spouts
> in storm UI. Notice that, you have to set different zookeeper paths for
> different topics, and they store offset for each topics in different
> location of zookeeper.
>
> SpoutConfig(BrokerHosts hosts, String topic, String zkRoot, String id)
>
> You should give different *id* for different topics.
>
> James
>
>
> Anis Nasir  於 2014/6/22 下午7:43 寫道：
>
> I have four instances of KafkaSpout in my topology .I can see four
> executors running on storm UI, as i have set number of instances to four.
> However, only one of the spouts is able to fetch messages from kafka.
>
> cheers
> Anis
>
>
> On Sun, Jun 22, 2014 at 1:40 PM,  wrote:
>
>> Hi,
>>
>> How many instances of KafkaSpout in your topology? If you have one only,
>> I think you can see one spout in storm UI. However, if you set parallelism
>> for 4, you will see "executers" for 4 in storm UI.
>>
>> James
>>
>> > Anis Nasir  於 2014/6/22 下午7:33 寫道：
>> >
>> > Dear all,
>> >
>> > I configured my kafka topic to have four partitions and I tried to set
>> the spout task parallelism to four as well. However, in storm UI, I can
>> only see one spout reading from the kafka topic. Can someone observed the
>> same pattern? or am I doing something wrong?
>> >
>> > Regards
>> > Anis
>>
>
>

Re: Unable to read from multiple kafkaSpout

[Anis Nasir]

I have four instances of KafkaSpout in my topology .I can see four
executors running on storm UI, as i have set number of instances to four.
However, only one of the spouts is able to fetch messages from kafka.

cheers
Anis


On Sun, Jun 22, 2014 at 1:40 PM,  wrote:

> Hi,
>
> How many instances of KafkaSpout in your topology? If you have one only, I
> think you can see one spout in storm UI. However, if you set parallelism
> for 4, you will see "executers" for 4 in storm UI.
>
> James
>
> > Anis Nasir  於 2014/6/22 下午7:33 寫道：
> >
> > Dear all,
> >
> > I configured my kafka topic to have four partitions and I tried to set
> the spout task parallelism to four as well. However, in storm UI, I can
> only see one spout reading from the kafka topic. Can someone observed the
> same pattern? or am I doing something wrong?
> >
> > Regards
> > Anis
>

Unable to read from multiple kafkaSpout

[Anis Nasir]

Dear all,

I configured my kafka topic to have four partitions and I tried to set the
spout task parallelism to four as well. However, in storm UI, I can only
see one spout reading from the kafka topic. Can someone observed the same
pattern? or am I doing something wrong?

Regards
Anis

Extracting Performance Metrics

[Anis Nasir]

Dear all,

I am running a cluster with 1 kafka + 1 nimbus + 10 supervisor + 1
zookeeper nodes. I am executing multiple topologies on the cluster and I
want to extract different metrics that I am mentioning below. Can someone
help me by recommending tools that I can use to extract this information.


Per Topology
 - Throughput
 - Latency

Per Spout or Bolt
 - Throughput
 - Latency
 - Execution Time
 - Queuing Time
 - Number of Messages Processed

Regards
Anis

Re: Getting Exception in kafka-storm

[Anis Nasir]

Thank you very much Michael.I downgraded to the older version (0.3.0)
of wurstmeister's
code that was using curator-framework 1.0.1. It is working for me now.

Cheers,
Anis


On Sat, Jun 14, 2014 at 11:15 PM, Michael G. Noll <
michael+st...@michael-noll.com> wrote:

> You may consider trying the build setup (dependencies as well as
> excludes) in kafka-storm-starter until Storm 0.9.2 -- which includes
> storm-kafka 0.9.2 -- is released.
>
> https://github.com/miguno/kafka-storm-starter/blob/develop/build.sbt
>
> Here I am using a re-packaged version of wurstmeister's spout.  The
> repackaged version is built against Java 6 and uses the latest version
> of wurstmeister's code, i.e. it's newer than the last 0.4 version
> released by wurstmeister.
>
> Details here:
> https://clojars.org/com.miguno/storm-kafka-0.8-plus_2.10
> https://github.com/miguno/storm-kafka-0.8-plus/tree/miguno_clojars
>
> 
>   com.miguno
>   storm-kafka-0.8-plus_2.10
>   0.5.0-SNAPSHOT
> 
>
> Again, I consider this as a short-term workaround until Storm 0.9.2 is
> released.
>
> Hope this helps,
> Michael
>
>
> On 06/12/2014 12:58 PM, Anis Nasir wrote:
> > I am still stuck on this issue. Can someone please help in upgrading the
> > storm cluster 0.9.0.1 to use curator-framework 1.3.3 and zookeeper 3.4.5
> >
> > Regards
> > Anis
> >
> >
> > On Wed, Jun 11, 2014 at 11:38 PM, Anis Nasir  > <mailto:aadi.a...@gmail.com>> wrote:
> >
> > Regarding, 0.9.2-incubating-SNAPSHOT, it comes with the interface
> > ILocalCluster. I can not find any example using it.
> >
> > Regards
> > Anis
> >
> >
> > On Wed, Jun 11, 2014 at 11:14 PM, Anis Nasir  > <mailto:aadi.a...@gmail.com>> wrote:
> >
> > Dear Jonas,
> >
> > I am getting the same exception running this build
> >
> > https://github.com/wurstmeister/storm-kafka-0.8-plus-test
> >
> >
> >
> > Regards
> > Anis
> >
> >
> > On Wed, Jun 11, 2014 at 11:07 PM, János Háber
> >  > <mailto:janos.ha...@finesolution.hu>> wrote:
> >
> > Anis: no, you using snapshot version of storm-kafka plugin
> > plus the old storm kafka plugin with storm 0.9.1...
> > I think storm-kafka 0.9.2-incubating-SNAPSHOT only
> > compatible with storm-core 0.9.2-incubating-SNAPSHOT
> >
> > In the 0.9.2-incubating-SNAPSHOT old curator framework
> > (1.0.1) will be changed to more recent version...
> >
> >     as you see you using two version of storm-kafka. Remove the
> > old one (the 0.4.0 version) and increase the storm-core
> > version to 0.9.2-incubating-SNAPSHOT...
> >
> > maybe it's help...
> >
> > b0c1
> >
> > János Háber
> > Fine Solution Ltd
> >
> >
> >
> > On Wed, Jun 11, 2014 at 11:03 PM, Anis Nasir
> > mailto:aadi.a...@gmail.com>> wrote:
> >
> > Dear Michael,
> >
> > I am using storm0.9.0.1 with kafka 0.8.1.1 and below
> > mentioned are the dependencies. I tried with both
> > storm-kafka 0.9.2-incubating-SNAPSHOT
> > and storm-kafka-0.8-plus 0.4.0. The zookeeper version is
> > 3.3.6.
> >
> > 
> >   org.apache.storm
> >   storm-core
> >   0.9.1-incubating
> > 
> >
> >  
> >  org.apache.storm
> >  storm-kafka
> >  0.9.2-incubating-SNAPSHOT
> > 
> >
> >  
> > net.wurstmeister.storm
> > storm-kafka-0.8-plus
> > 0.4.0
> >   
> >
> > Regards
> > Anis
> >
> >
> >
> > On Wed, Jun 11, 2014 at 10:51 PM, Michael G. Noll
> >  > <mailto:michael+st...@michael-noll.com>> wrote:
> >
> > Can you share your build depencies, notably the
> > versions of Storm and
> > storm-kafka and their respect

Re: Getting Exception in kafka-storm

[Anis Nasir]

I am still stuck on this issue. Can someone please help in upgrading the
storm cluster 0.9.0.1 to use curator-framework 1.3.3 and zookeeper 3.4.5

Regards
Anis


On Wed, Jun 11, 2014 at 11:38 PM, Anis Nasir  wrote:

> Regarding, 0.9.2-incubating-SNAPSHOT, it comes with the interface
> ILocalCluster. I can not find any example using it.
>
> Regards
> Anis
>
>
> On Wed, Jun 11, 2014 at 11:14 PM, Anis Nasir  wrote:
>
>> Dear Jonas,
>>
>> I am getting the same exception running this build
>>
>> https://github.com/wurstmeister/storm-kafka-0.8-plus-test
>>
>>
>>
>> Regards
>> Anis
>>
>>
>> On Wed, Jun 11, 2014 at 11:07 PM, János Háber <
>> janos.ha...@finesolution.hu> wrote:
>>
>>> Anis: no, you using snapshot version of storm-kafka plugin plus the old
>>> storm kafka plugin with storm 0.9.1...
>>> I think storm-kafka 0.9.2-incubating-SNAPSHOT only compatible with
>>> storm-core 0.9.2-incubating-SNAPSHOT
>>>
>>> In the 0.9.2-incubating-SNAPSHOT old curator framework (1.0.1) will be
>>> changed to more recent version...
>>>
>>> as you see you using two version of storm-kafka. Remove the old one (the
>>> 0.4.0 version) and increase the storm-core version
>>> to 0.9.2-incubating-SNAPSHOT...
>>>
>>> maybe it's help...
>>>
>>> b0c1
>>>
>>> János Háber
>>> Fine Solution Ltd
>>>
>>>
>>>
>>> On Wed, Jun 11, 2014 at 11:03 PM, Anis Nasir 
>>> wrote:
>>>
>>>> Dear Michael,
>>>>
>>>> I am using storm0.9.0.1 with kafka 0.8.1.1 and below mentioned are the
>>>> dependencies. I tried with both storm-kafka 0.9.2-incubating-SNAPSHOT
>>>> and storm-kafka-0.8-plus 0.4.0. The zookeeper version is 3.3.6.
>>>>
>>>> 
>>>>   org.apache.storm
>>>>   storm-core
>>>>   0.9.1-incubating
>>>> 
>>>>
>>>>  
>>>>  org.apache.storm
>>>>  storm-kafka
>>>>  0.9.2-incubating-SNAPSHOT
>>>> 
>>>>
>>>>   
>>>> net.wurstmeister.storm
>>>> storm-kafka-0.8-plus
>>>> 0.4.0
>>>>   
>>>>
>>>> Regards
>>>> Anis
>>>>
>>>>
>>>>
>>>> On Wed, Jun 11, 2014 at 10:51 PM, Michael G. Noll <
>>>> michael+st...@michael-noll.com> wrote:
>>>>
>>>>> Can you share your build depencies, notably the versions of Storm and
>>>>> storm-kafka and their respective artifactId and groupId?   That is, the
>>>>> snippets relevant to storm-core and storm-kafka from your pom.xml,
>>>>> build.gradle, build.sbt, or similar build configuration file.  Also,
>>>>> please add from which Maven repositories you are downloading storm-core
>>>>> and storm-kafka, respectively.
>>>>>
>>>>> Best,
>>>>> Michael
>>>>>
>>>>>
>>>>>
>>>>> On 06/11/2014 09:52 PM, Anis Nasir wrote:
>>>>> > Dear all,
>>>>> >
>>>>> >
>>>>> > I am getting below mentioned exception while running storm-kafka.
>>>>> Does
>>>>> > anyone have seen this one before?
>>>>> >
>>>>> >
>>>>> > 7800 [Thread-27-words] ERROR backtype.storm.util - Async loop died!
>>>>> > java.lang.NoSuchMethodError:
>>>>> >
>>>>> com.netflix.curator.framework.api.CreateBuilder.creatingParentsIfNeeded()Lcom/netflix/curator/framework/api/ProtectACLCreateModePathAndBytesable;
>>>>> > at storm.kafka.ZkState.writeBytes(ZkState.java:59)
>>>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>>>> > at storm.kafka.ZkState.writeJSON(ZkState.java:53)
>>>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>>>> > at storm.kafka.PartitionManager.commit(PartitionManager.java:188)
>>>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>>>> > at storm.kafka.KafkaSpout.commit(KafkaSpout.java:169)
>>>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>>>> > at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:134)
>>>>> > ~[storm-kafka-0.8-

Re: Getting Exception in kafka-storm

[Anis Nasir]

Regarding, 0.9.2-incubating-SNAPSHOT, it comes with the interface
ILocalCluster. I can not find any example using it.

Regards
Anis


On Wed, Jun 11, 2014 at 11:14 PM, Anis Nasir  wrote:

> Dear Jonas,
>
> I am getting the same exception running this build
>
> https://github.com/wurstmeister/storm-kafka-0.8-plus-test
>
>
>
> Regards
> Anis
>
>
> On Wed, Jun 11, 2014 at 11:07 PM, János Háber  > wrote:
>
>> Anis: no, you using snapshot version of storm-kafka plugin plus the old
>> storm kafka plugin with storm 0.9.1...
>> I think storm-kafka 0.9.2-incubating-SNAPSHOT only compatible with
>> storm-core 0.9.2-incubating-SNAPSHOT
>>
>> In the 0.9.2-incubating-SNAPSHOT old curator framework (1.0.1) will be
>> changed to more recent version...
>>
>> as you see you using two version of storm-kafka. Remove the old one (the
>> 0.4.0 version) and increase the storm-core version
>> to 0.9.2-incubating-SNAPSHOT...
>>
>> maybe it's help...
>>
>> b0c1
>>
>> János Háber
>> Fine Solution Ltd
>>
>>
>>
>> On Wed, Jun 11, 2014 at 11:03 PM, Anis Nasir  wrote:
>>
>>> Dear Michael,
>>>
>>> I am using storm0.9.0.1 with kafka 0.8.1.1 and below mentioned are the
>>> dependencies. I tried with both storm-kafka 0.9.2-incubating-SNAPSHOT
>>> and storm-kafka-0.8-plus 0.4.0. The zookeeper version is 3.3.6.
>>>
>>> 
>>>   org.apache.storm
>>>   storm-core
>>>   0.9.1-incubating
>>> 
>>>
>>>  
>>>  org.apache.storm
>>>  storm-kafka
>>>  0.9.2-incubating-SNAPSHOT
>>> 
>>>
>>>   
>>> net.wurstmeister.storm
>>> storm-kafka-0.8-plus
>>> 0.4.0
>>>   
>>>
>>> Regards
>>> Anis
>>>
>>>
>>>
>>> On Wed, Jun 11, 2014 at 10:51 PM, Michael G. Noll <
>>> michael+st...@michael-noll.com> wrote:
>>>
>>>> Can you share your build depencies, notably the versions of Storm and
>>>> storm-kafka and their respective artifactId and groupId?   That is, the
>>>> snippets relevant to storm-core and storm-kafka from your pom.xml,
>>>> build.gradle, build.sbt, or similar build configuration file.  Also,
>>>> please add from which Maven repositories you are downloading storm-core
>>>> and storm-kafka, respectively.
>>>>
>>>> Best,
>>>> Michael
>>>>
>>>>
>>>>
>>>> On 06/11/2014 09:52 PM, Anis Nasir wrote:
>>>> > Dear all,
>>>> >
>>>> >
>>>> > I am getting below mentioned exception while running storm-kafka. Does
>>>> > anyone have seen this one before?
>>>> >
>>>> >
>>>> > 7800 [Thread-27-words] ERROR backtype.storm.util - Async loop died!
>>>> > java.lang.NoSuchMethodError:
>>>> >
>>>> com.netflix.curator.framework.api.CreateBuilder.creatingParentsIfNeeded()Lcom/netflix/curator/framework/api/ProtectACLCreateModePathAndBytesable;
>>>> > at storm.kafka.ZkState.writeBytes(ZkState.java:59)
>>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>>> > at storm.kafka.ZkState.writeJSON(ZkState.java:53)
>>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>>> > at storm.kafka.PartitionManager.commit(PartitionManager.java:188)
>>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>>> > at storm.kafka.KafkaSpout.commit(KafkaSpout.java:169)
>>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>>> > at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:134)
>>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>>> > at
>>>> >
>>>> backtype.storm.daemon.executor$fn__3430$fn__3445$fn__3474.invoke(executor.clj:547)
>>>> > ~[storm-core-0.9.0.1.jar:na]
>>>> > at backtype.storm.util$async_loop$fn__444.invoke(util.clj:403)
>>>> > ~[storm-core-0.9.0.1.jar:na]
>>>> > at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.4.0.jar:na]
>>>> > at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_30]
>>>> > 7800 [Thread-27-words] ERROR backtype.storm.daemon.executor -
>>>> > java.lang.NoSuchMethodError:
>>>> >
>>>> com.netflix.curator.framework.api.CreateBuilder.creatingParentsIfNeeded()Lcom/netflix/curator/framework/api/ProtectACLCreateModePathAndBytesable;
>>>> > at storm.kafka.ZkState.writeBytes(ZkState.java:59)
>>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>>> > at storm.kafka.ZkState.writeJSON(ZkState.java:53)
>>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>>> > at storm.kafka.PartitionManager.commit(PartitionManager.java:188)
>>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>>> > at storm.kafka.KafkaSpout.commit(KafkaSpout.java:169)
>>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>>> > at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:134)
>>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>>> > at
>>>> >
>>>> backtype.storm.daemon.executor$fn__3430$fn__3445$fn__3474.invoke(executor.clj:547)
>>>> > ~[storm-core-0.9.0.1.jar:na]
>>>> > at backtype.storm.util$async_loop$fn__444.invoke(util.clj:403)
>>>> > ~[storm-core-0.9.0.1.jar:na]
>>>> > at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.4.0.jar:na]
>>>> > at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_30]
>>>> >
>>>> > Regards
>>>> > Anis
>>>>
>>>>
>>>
>>
>

Re: Getting Exception in kafka-storm

[Anis Nasir]

Dear Jonas,

I am getting the same exception running this build

https://github.com/wurstmeister/storm-kafka-0.8-plus-test



Regards
Anis


On Wed, Jun 11, 2014 at 11:07 PM, János Háber 
wrote:

> Anis: no, you using snapshot version of storm-kafka plugin plus the old
> storm kafka plugin with storm 0.9.1...
> I think storm-kafka 0.9.2-incubating-SNAPSHOT only compatible with
> storm-core 0.9.2-incubating-SNAPSHOT
>
> In the 0.9.2-incubating-SNAPSHOT old curator framework (1.0.1) will be
> changed to more recent version...
>
> as you see you using two version of storm-kafka. Remove the old one (the
> 0.4.0 version) and increase the storm-core version
> to 0.9.2-incubating-SNAPSHOT...
>
> maybe it's help...
>
> b0c1
>
> János Háber
> Fine Solution Ltd
>
>
>
> On Wed, Jun 11, 2014 at 11:03 PM, Anis Nasir  wrote:
>
>> Dear Michael,
>>
>> I am using storm0.9.0.1 with kafka 0.8.1.1 and below mentioned are the
>> dependencies. I tried with both storm-kafka 0.9.2-incubating-SNAPSHOT
>> and storm-kafka-0.8-plus 0.4.0. The zookeeper version is 3.3.6.
>>
>> 
>>   org.apache.storm
>>   storm-core
>>   0.9.1-incubating
>> 
>>
>>  
>>  org.apache.storm
>>  storm-kafka
>>  0.9.2-incubating-SNAPSHOT
>> 
>>
>>   
>> net.wurstmeister.storm
>> storm-kafka-0.8-plus
>> 0.4.0
>>   
>>
>> Regards
>> Anis
>>
>>
>>
>> On Wed, Jun 11, 2014 at 10:51 PM, Michael G. Noll <
>> michael+st...@michael-noll.com> wrote:
>>
>>> Can you share your build depencies, notably the versions of Storm and
>>> storm-kafka and their respective artifactId and groupId?   That is, the
>>> snippets relevant to storm-core and storm-kafka from your pom.xml,
>>> build.gradle, build.sbt, or similar build configuration file.  Also,
>>> please add from which Maven repositories you are downloading storm-core
>>> and storm-kafka, respectively.
>>>
>>> Best,
>>> Michael
>>>
>>>
>>>
>>> On 06/11/2014 09:52 PM, Anis Nasir wrote:
>>> > Dear all,
>>> >
>>> >
>>> > I am getting below mentioned exception while running storm-kafka. Does
>>> > anyone have seen this one before?
>>> >
>>> >
>>> > 7800 [Thread-27-words] ERROR backtype.storm.util - Async loop died!
>>> > java.lang.NoSuchMethodError:
>>> >
>>> com.netflix.curator.framework.api.CreateBuilder.creatingParentsIfNeeded()Lcom/netflix/curator/framework/api/ProtectACLCreateModePathAndBytesable;
>>> > at storm.kafka.ZkState.writeBytes(ZkState.java:59)
>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>> > at storm.kafka.ZkState.writeJSON(ZkState.java:53)
>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>> > at storm.kafka.PartitionManager.commit(PartitionManager.java:188)
>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>> > at storm.kafka.KafkaSpout.commit(KafkaSpout.java:169)
>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>> > at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:134)
>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>> > at
>>> >
>>> backtype.storm.daemon.executor$fn__3430$fn__3445$fn__3474.invoke(executor.clj:547)
>>> > ~[storm-core-0.9.0.1.jar:na]
>>> > at backtype.storm.util$async_loop$fn__444.invoke(util.clj:403)
>>> > ~[storm-core-0.9.0.1.jar:na]
>>> > at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.4.0.jar:na]
>>> > at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_30]
>>> > 7800 [Thread-27-words] ERROR backtype.storm.daemon.executor -
>>> > java.lang.NoSuchMethodError:
>>> >
>>> com.netflix.curator.framework.api.CreateBuilder.creatingParentsIfNeeded()Lcom/netflix/curator/framework/api/ProtectACLCreateModePathAndBytesable;
>>> > at storm.kafka.ZkState.writeBytes(ZkState.java:59)
>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>> > at storm.kafka.ZkState.writeJSON(ZkState.java:53)
>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>> > at storm.kafka.PartitionManager.commit(PartitionManager.java:188)
>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>> > at storm.kafka.KafkaSpout.commit(KafkaSpout.java:169)
>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>> > at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:134)
>>> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
>>> > at
>>> >
>>> backtype.storm.daemon.executor$fn__3430$fn__3445$fn__3474.invoke(executor.clj:547)
>>> > ~[storm-core-0.9.0.1.jar:na]
>>> > at backtype.storm.util$async_loop$fn__444.invoke(util.clj:403)
>>> > ~[storm-core-0.9.0.1.jar:na]
>>> > at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.4.0.jar:na]
>>> > at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_30]
>>> >
>>> > Regards
>>> > Anis
>>>
>>>
>>
>

Re: Getting Exception in kafka-storm

[Anis Nasir]

Dear Michael,

I am using storm0.9.0.1 with kafka 0.8.1.1 and below mentioned are the
dependencies. I tried with both storm-kafka 0.9.2-incubating-SNAPSHOT
and storm-kafka-0.8-plus
0.4.0. The zookeeper version is 3.3.6.


  org.apache.storm
  storm-core
  0.9.1-incubating


 
 org.apache.storm
 storm-kafka
 0.9.2-incubating-SNAPSHOT


 
net.wurstmeister.storm
storm-kafka-0.8-plus
0.4.0
  

Regards
Anis



On Wed, Jun 11, 2014 at 10:51 PM, Michael G. Noll <
michael+st...@michael-noll.com> wrote:

> Can you share your build depencies, notably the versions of Storm and
> storm-kafka and their respective artifactId and groupId?   That is, the
> snippets relevant to storm-core and storm-kafka from your pom.xml,
> build.gradle, build.sbt, or similar build configuration file.  Also,
> please add from which Maven repositories you are downloading storm-core
> and storm-kafka, respectively.
>
> Best,
> Michael
>
>
>
> On 06/11/2014 09:52 PM, Anis Nasir wrote:
> > Dear all,
> >
> >
> > I am getting below mentioned exception while running storm-kafka. Does
> > anyone have seen this one before?
> >
> >
> > 7800 [Thread-27-words] ERROR backtype.storm.util - Async loop died!
> > java.lang.NoSuchMethodError:
> >
> com.netflix.curator.framework.api.CreateBuilder.creatingParentsIfNeeded()Lcom/netflix/curator/framework/api/ProtectACLCreateModePathAndBytesable;
> > at storm.kafka.ZkState.writeBytes(ZkState.java:59)
> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
> > at storm.kafka.ZkState.writeJSON(ZkState.java:53)
> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
> > at storm.kafka.PartitionManager.commit(PartitionManager.java:188)
> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
> > at storm.kafka.KafkaSpout.commit(KafkaSpout.java:169)
> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
> > at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:134)
> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
> > at
> >
> backtype.storm.daemon.executor$fn__3430$fn__3445$fn__3474.invoke(executor.clj:547)
> > ~[storm-core-0.9.0.1.jar:na]
> > at backtype.storm.util$async_loop$fn__444.invoke(util.clj:403)
> > ~[storm-core-0.9.0.1.jar:na]
> > at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.4.0.jar:na]
> > at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_30]
> > 7800 [Thread-27-words] ERROR backtype.storm.daemon.executor -
> > java.lang.NoSuchMethodError:
> >
> com.netflix.curator.framework.api.CreateBuilder.creatingParentsIfNeeded()Lcom/netflix/curator/framework/api/ProtectACLCreateModePathAndBytesable;
> > at storm.kafka.ZkState.writeBytes(ZkState.java:59)
> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
> > at storm.kafka.ZkState.writeJSON(ZkState.java:53)
> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
> > at storm.kafka.PartitionManager.commit(PartitionManager.java:188)
> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
> > at storm.kafka.KafkaSpout.commit(KafkaSpout.java:169)
> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
> > at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:134)
> > ~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
> > at
> >
> backtype.storm.daemon.executor$fn__3430$fn__3445$fn__3474.invoke(executor.clj:547)
> > ~[storm-core-0.9.0.1.jar:na]
> > at backtype.storm.util$async_loop$fn__444.invoke(util.clj:403)
> > ~[storm-core-0.9.0.1.jar:na]
> > at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.4.0.jar:na]
> > at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_30]
> >
> > Regards
> > Anis
>
>

Getting Exception in kafka-storm

[Anis Nasir]

Dear all,


I am getting below mentioned exception while running storm-kafka. Does
anyone have seen this one before?


7800 [Thread-27-words] ERROR backtype.storm.util - Async loop died!
java.lang.NoSuchMethodError: com.netflix.curator.framework.
api.CreateBuilder.creatingParentsIfNeeded()Lcom/
netflix/curator/framework/api/ProtectACLCreateModePathAndBytesable;
at storm.kafka.ZkState.writeBytes(ZkState.java:59)
~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
at storm.kafka.ZkState.writeJSON(ZkState.java:53)
~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
at storm.kafka.PartitionManager.commit(PartitionManager.java:188)
~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
at storm.kafka.KafkaSpout.commit(KafkaSpout.java:169)
~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:134)
~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
at 
backtype.storm.daemon.executor$fn__3430$fn__3445$fn__3474.invoke(executor.clj:547)
~[storm-core-0.9.0.1.jar:na]
at backtype.storm.util$async_loop$fn__444.invoke(util.clj:403)
~[storm-core-0.9.0.1.jar:na]
at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.4.0.jar:na]
at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_30]
7800 [Thread-27-words] ERROR backtype.storm.daemon.executor -
java.lang.NoSuchMethodError: com.netflix.curator.framework.
api.CreateBuilder.creatingParentsIfNeeded()Lcom/
netflix/curator/framework/api/ProtectACLCreateModePathAndBytesable;
at storm.kafka.ZkState.writeBytes(ZkState.java:59)
~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
at storm.kafka.ZkState.writeJSON(ZkState.java:53)
~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
at storm.kafka.PartitionManager.commit(PartitionManager.java:188)
~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
at storm.kafka.KafkaSpout.commit(KafkaSpout.java:169)
~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:134)
~[storm-kafka-0.8-plus-0.5.0-SNAPSHOT.jar:na]
at 
backtype.storm.daemon.executor$fn__3430$fn__3445$fn__3474.invoke(executor.clj:547)
~[storm-core-0.9.0.1.jar:na]
at backtype.storm.util$async_loop$fn__444.invoke(util.clj:403)
~[storm-core-0.9.0.1.jar:na]
at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.4.0.jar:na]
at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_30]

Regards
Anis

Re: Storm-kafka Integration

[Anis Nasir]

Dear Komal,

I used the storm-kafka version mentioned in the link.

http://anisnasir.wordpress.com/2014/05/25/apache-storm-kafka-a-nice-choice/


Regards
Anis



On Mon, Jun 2, 2014 at 1:49 PM, Komal Thombare 
wrote:

> Hi Andres,
>
> Still i am getting the same error.
>
>
> Thanks and Regards,
>
> Komal Thombare
>
>
> -Andres Gomez  wrote: -
>
> To: user@storm.incubator.apache.org
> From: Andres Gomez 
> Date: 06/02/2014 04:29PM
>
> Subject: Re: Storm-kafka Integration
>
> Hi again,
>
> I advise you to use the repo kafka happens to you:
>
>
> https://github.com/apache/incubator-storm/tree/master/external/storm-kafka
>
>
> 1. make a clone of this project
> 2. mvm package — generate a jar of project
> 3. mvn install — install the project on local repository
>
> Regards,
>
> Andres
>
> El 02/06/2014, a las 12:56, Komal Thombare 
> escribió:
>
>  Hi,
>
> I have downloaded the storm-kafka jar from the below link:
> http://repo1.maven.org/maven2/com/n3twork/storm/storm-kafka/20140521/
>
> Now I have made changes in the storm word count topology and used
> KafkaSpout.
> While running topology in Local mode I get following error:
>
> java.lang.NoClassDefFoundError: org/apache/curator/RetryPolicy
> at storm.kafka.KafkaSpout.open(KafkaSpout.java:85)
> ~[storm-kafka-20140521.jar:na]
> at
> backtype.storm.daemon.executor$eval5100$fn__5101$fn__5116.invoke(executor.clj:519)
> ~[na:na]
> at backtype.storm.util$async_loop$fn__390.invoke(util.clj:431) ~[na:na]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
> at java.lang.Thread.run(Thread.java:636) [na:1.6.0_17]
> Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy
> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
> ~[na:1.6.0_17]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[na:1.6.0_17]
> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
> ~[na:1.6.0_17]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:319) ~[na:1.6.0_17]
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
> ~[na:1.6.0_17]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:264) ~[na:1.6.0_17]
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:332)
> ~[na:1.6.0_17]
>
> Though the curator-client-1.0.1.jar is present in the lib of storm and I
> have included same in my buildpath.
>
>
> Thanks and Regards,
>
> Komal Thombare
>
>
> -Andres Gomez wrote: -
>
> To: user@storm.incubator.apache.org
> From: Andres Gomez 
> Date: 06/02/2014 02:33PM
> Subject: Re: Storm-kafka Integration
>
> You should make a clone of this project:
>
> https://github.com/apache/incubator-storm/tree/master/external/storm-kafka
>
> and do “mvn install”, I suposse you use kafka 0.8.+
>
> and then you do this:
>
> SpoutConfig spoutConfig = new SpoutConfig();
>
> spoutConfig.zkServers = “localhost”; //zookeeper Host
> spoutConfig.zkPort = 2181;   //zookeeper port
> spoutConfig.zkRoot = “/kafkaStorm/“;  // zookeeper path
> spoutConfig.id = “storm”; // zookeeper id
>
>
> // This config is a example!!
>
> KafkaSpout spout = new KafkaSpout(spoutConfig);
>
>
> El 02/06/2014, a las 10:56, Joe Stein  escribió:
>
> Please take a look at
> http://www.michael-noll.com/blog/2014/05/27/kafka-storm-integration-example-tutorial/#state-of-the-integration-game
>
> There is a github project too
> https://github.com/miguno/kafka-storm-starter
>
> This covers latest Storm, Kafka and Avro.
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>
>
> On Mon, Jun 2, 2014 at 4:55 AM, Komal Thombare 
> wrote:
>
>>  Hi Deepak,
>>
>> Yes i have. I have also got the storm-contrib source code, but then I am
>> unaware of how to compile it.
>>
>>
>> Thanks and Regards,
>>
>> Komal Thombare
>> Tata Consultancy Services Limited
>> Ph:- 086-55388772
>> Mail-to: komal.thomb...@tcs.com
>> Website: http://www.tcs.com
>> 
>> Experience certainty. IT Services
>> Business Solutions
>> Consulting
>> 
>>
>> -Deepak Sharma  wrote: -
>>
>> To: user 
>> From: Deepak Sharma 
>> Date: 06/02/2014 02:22PM
>> Subject: Re: Storm-kafka Integration
>>
>>
>> Hi Komal
>> Have you looked at KafkaSpout?
>>
>> Thanks
>> Deepak
>>
>>
>> On Mon, Jun 2, 2014 at 2:13 PM, Komal Thombare 
>> wrote:
>>
>>> Hi,
>>>
>>> I am trying to integrate storm with kafka, but did not get any proper
>>> documentation to do so.
>>> Can anyone please help me for getting started with the integration.
>>>
>>> Thanks and Regards,
>>>
>>> Komal Thombare
>>>
>>> =-=-=
>>> Notice: The information contained in this e-mail
>>> message and/or attachments to it may contain
>>> conf

Unable to read from beginning of the topic storm-kafka-0.8-plus

[Anis Nasir]

Dear all,

I am using Storm 0.9.1-incubating, along with kafka 0.8.1.1 with
storm-kafka-0.8-plus. Initially, the library worked fine and was able to
fetch messages from a kafka topic. However, after running the topology few
times, now I am unable to fetch any message from the kafka topic. I believe
it is a problem to set the offset from the beginning of the topic. I have
tried setting

kafkaConfig.startOffsetTime = -2;
and also
kafkaConfig.forceFromStart = true;

However, none of these parameters force the spout to read the messages from
beginning of the topic. I would really appreciate any help in this regard.

Regards

Anis

Re: compatible version of kafka and storm-kafka

[Anis Nasir]

It worked. Thanks a lot Irek and Joe. I really appreciate it.


On Sat, May 24, 2014 at 6:36 PM, Joe Stein  wrote:

> You should also upgrade Kafka to 0.8.1.1 some bug fixes.
>
>
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop
> /
>
>
> On May 24, 2014, at 10:42 AM, Irek Khasyanov  wrote:
>
> Anis, storm-kafka-plus 0.4 works fine for with same versions
>
>
> On 24 May 2014 18:34, Anis Nasir  wrote:
>
>> Thank you very much for the response Joe.
>>
>> I am running a Strom 0.9.1 cluster and Kafka 0.8.1 cluster. Both are
>> sharing a zookeeper version 3.3.6. I am looking for a storm-kafka that can
>> help me to connect both these clusters.
>>
>> cheers,
>> Anis
>>
>>
>> On Sat, May 24, 2014 at 2:17 PM, Joe Stein  wrote:
>>
>>> Michael Noll recently put together a Kafka/Storm starter which is a good
>>> place to begin https://github.com/miguno/kafka-storm-starter
>>>
>>> /***
>>>  Joe Stein
>>>  Founder, Principal Consultant
>>>  Big Data Open Source Security LLC
>>>  http://www.stealth.ly
>>>  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
>>> /
>>>
>>>
>>> On Sat, May 24, 2014 at 8:12 AM, Anis Nasir  wrote:
>>>
>>>> Dear all,
>>>>
>>>> Can someone suggest me a stable version of kafka and storm kafka that
>>>> works with Storm 0.9.1 incubating. Thanking you in advance.
>>>>
>>>> Regards
>>>> Anis
>>>>
>>>
>>>
>>
>
>
> --
> With best regards, Irek Khasyanov.
>
>

Re: compatible version of kafka and storm-kafka

[Anis Nasir]

Thank you very much for the response Joe.

I am running a Strom 0.9.1 cluster and Kafka 0.8.1 cluster. Both are
sharing a zookeeper version 3.3.6. I am looking for a storm-kafka that can
help me to connect both these clusters.

cheers,
Anis


On Sat, May 24, 2014 at 2:17 PM, Joe Stein  wrote:

> Michael Noll recently put together a Kafka/Storm starter which is a good
> place to begin https://github.com/miguno/kafka-storm-starter
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> /
>
>
> On Sat, May 24, 2014 at 8:12 AM, Anis Nasir  wrote:
>
>> Dear all,
>>
>> Can someone suggest me a stable version of kafka and storm kafka that
>> works with Storm 0.9.1 incubating. Thanking you in advance.
>>
>> Regards
>> Anis
>>
>
>

compatible version of kafka and storm-kafka

[Anis Nasir]

Dear all,

Can someone suggest me a stable version of kafka and storm kafka that works
with Storm 0.9.1 incubating. Thanking you in advance.

Regards
Anis

[Storm-user]- Fetching and logging in HDFS

[Anis Nasir]

Dear all,

Is there a spout or a bolt available to read and write data from a HDFS
cluster?


Regards
Anis