Re: Submitting jar to storm cluster using java

StormSubmitter.submitJar https://github.com/apache/incubator-storm/blob/master/storm-core/src/jvm/backtype/storm/StormSubmitter.java#L230 On 22 May 2014, at 3:44 pm, Komal Thombare wrote: > Hi, > > We submit storm jar to storm cluster using commandline, as 'storm jar'. > > But how could it

Submitting jar to storm cluster using java

Hi, We submit storm jar to storm cluster using commandline, as 'storm jar'. But how could it be done with the java client. Thanks and Regards, Komal Thombare =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain confidential o

Re: Which daemon tool for Storm nodes?

I always like hearing the question: "What happened to the supervisor supervisor?!" -Cody On Wed, May 21, 2014 at 7:48 PM, P. Taylor Goetz wrote: > Exactly what Michael said... > > Any good process monitoring tool will work. > > The only issue I've had with supervisord is not with the software,

Re: Which daemon tool for Storm nodes?

Exactly what Michael said... Any good process monitoring tool will work. The only issue I've had with supervisord is not with the software, but rather the name conflict with the Storm supervisor process -- it can confuse the hell out of some people. -Taylor > On May 21, 2014, at 9:29 PM, Mich

Re: Sample project using Storm and Kafka (using Scala 2.10)

János, I sense and appreciate your frustration. The initial goal with bringing the storm-kafka project under the Apache Storm project umbrella is to provide a definitive source for the code and eliminate fragmentation, which has been an issue in the past. This is a good first step for many rea

Re: Which daemon tool for Storm nodes?

We use upstart. Supervisord would also work. Just anything to keep an eye on it and restart it if it dies (a very rare occurrence). Michael Rose (@Xorlev ) Senior Platform Engineer, FullContact mich...@fullcontact.com On Wed, May 21, 2014

Which daemon tool for Storm nodes?

Hi, Is there a recommended supervisor or daemon tool for Storm nodes? I'm using Supervisor for Storm and Zookeeper. I heard some concern in using Supervisor, but I haven't researched into the matter. Any comments on this? Thanks, Connie

Re: Sample project using Storm and Kafka (using Scala 2.10)

Dear Taylor, I love your work, but: - I don't want to build myself - Dependent libraries (like tormenta-kafka) need a cross compiled version of storm-kafka, without this they need to clone the project, change the group id, handle every changes by hand, and publish to central repository. - I need to

$mastercoord-bg0

what does the $mastercoord-bg0 represent ? It seems to have much less work that my bolt spout. Also how can I set the parallelism of this master spout ? when my other bolts are emitting and acking millions of tuples, this master spout only emits/acks a couple dozen. Also currently something in my

Re:RE: Recovering From Zookeeper Failure

Thanks. - Original Message - From: Simon Cooper At: Wednesday, May 21, 2014 06:16 This has already been reported: https://issues.apache.org/jira/browse/STORM-307. The workaround we’ve implemented is, in our storm init scripts, we always delete the {storm.home}/su

Re: Sample project using Storm and Kafka (using Scala 2.10)

If you build yourself, you can do the following: mvn install -DscalaVersion=2.10.3 -DkafkaArtifact=kafka_2.10 - Taylor On May 21, 2014, at 5:32 PM, János Háber wrote: > https://github.com/apache/incubator-storm/blob/master/external/storm-kafka/pom.xml#L34 > And 2.10 not backward compatible wi

Re: Sample project using Storm and Kafka (using Scala 2.10)

https://github.com/apache/incubator-storm/blob/master/external/storm-kafka/pom.xml#L34 And 2.10 not backward compatible with 2.9... that's why scala applications/libraries is cross compiled with multiple scala version. (that's why SBT will handle this natively, but I think in apache sbt is not allo

Re: Sample project using Storm and Kafka (using Scala 2.10)

On Wed, May 21, 2014 at 11:27 PM, P. Taylor Goetz wrote: > ill include the Kafka spout (for Kafka 0.8.x). > Yeah, but the current maven build compile to 2.9 scala not 2.10... János Háber Fine Solution Ltd

Re: Sample project using Storm and Kafka (using Scala 2.10)

storm-kafka is now part of the Storm project. When 0.9.2 is released (shortly) it will include the Kafka spout (for Kafka 0.8.x). - Taylor On May 21, 2014, at 4:32 PM, Marco wrote: > Hi, > I'm having some troubles understanding how to boostrap a Kafka + Storm > project. > > Specifically I d

Re: Sample project using Storm and Kafka (using Scala 2.10)

Benjamin: That's great but storm-kafka-plus compiled with scala 2.9 not 2.10, you need to download the source and recompile with maven option ( https://github.com/wurstmeister/storm-kafka-0.8-plus/pull/15 ). The storm-kafka-plus project is moved to storm as external project (so I think the lat

Re: Sample project using Storm and Kafka (using Scala 2.10)

https://www.google.com/search?q=storm+kafka+0.8&oq=storm+kafka+0.8 https://kafka.apache.org/downloads.html On Wed, May 21, 2014 at 1:32 PM, Marco wrote: > Hi, > I'm having some troubles understanding how to boostrap a Kafka + Storm > project. > > Specifically I don't get if storm-kafka is indee

Sample project using Storm and Kafka (using Scala 2.10)

Hi, I'm having some troubles understanding how to boostrap a Kafka + Storm project. Specifically I don't get if storm-kafka is indeed included in Storm release or not. Also, I'm currently using Scala 2.10 but it seems that Kafka main release is only available in its Scala 2.9.2 version. Thanks

Re: Sometimes topology crashed with internal exception

You can also synchronize access to the OutputCollector so that only 1 thread is using it at a time. On Wed, May 21, 2014 at 12:39 PM, Irek Khasyanov wrote: > Hm, yes, firs bolt emitting from different thread, I did't realize that > this will be problem. Thanks! I'll try to change everything and

Re: Sometimes topology crashed with internal exception

Hm, yes, firs bolt emitting from different thread, I did't realize that this will be problem. Thanks! I'll try to change everything and see errors. On 21 May 2014 18:01, P. Taylor Goetz wrote: > Are you using the OutputCollector to emit in a separate thread (i.e. > Outside of the execute() met

Set Component Configuration after during Prepare Method

I want to set a *tick tuple* and I always do that in getComponentConfiguration() without problems. But in this case for a variety of reasons I need do it during *prepare method* and I don't know how to set tick tuple at this point (getComponentConfiguration has already called). Is it possible cha

Re: [Storm-user]- Fetching and logging in HDFS

Hi, I think you're looking for https://github.com/ptgoetz/storm-hdfs Hope this helps. Cheers, Suparno On 21 May 2014 15:08, Anis Nasir wrote: > Dear all, > > Is there a spout or a bolt available to read and write data from a HDFS > cluster? > > > Regards > Anis > -- Suparno Datta

Re: Sometimes topology crashed with internal exception

Are you using the OutputCollector to emit in a separate thread (i.e. Outside of the execute() method.)? As the wiki states, this will cause the problem you are seeing? -Taylor > On May 21, 2014, at 7:24 AM, Irek Khasyanov wrote: > > Hello. > > I have strange problem with by topology, sometim

Re: Sometimes topology crashed with internal exception

With null value topology crashed with different exception: java.lang.NullPointerException: null at backtype.storm.utils.Utils.toByteArray(Utils.java:383) ~[storm-core-0.9.0.1.jar:na] at storm.kafka.KafkaUtils.generateTuples(KafkaUtils.java:177) ~[storm-kafka-0.8-plus-0.4.0.jar:na] at storm.kafka.P

[Storm-user]- Fetching and logging in HDFS

Dear all, Is there a spout or a bolt available to read and write data from a HDFS cluster? Regards Anis

Re: Sometimes topology crashed with internal exception

Interesting question, I did't look into this direction. I will check it. On 21 May 2014 16:49, Nathan Leung wrote: > Are you sure that you're not passing a null value in your tuple? > On May 21, 2014 7:25 AM, "Irek Khasyanov" wrote: > >> Hello. >> >> I have strange problem with by topology, so

Re: Sometimes topology crashed with internal exception

Are you sure that you're not passing a null value in your tuple? On May 21, 2014 7:25 AM, "Irek Khasyanov" wrote: > Hello. > > I have strange problem with by topology, sometimes everything crashed with > exception: > > java.lang.RuntimeException: java.lang.NullPointerException > at > backtype.st

Sometimes topology crashed with internal exception

Hello. I have strange problem with by topology, sometimes everything crashed with exception: java.lang.RuntimeException: java.lang.NullPointerException at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:107) at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvai

RE: [VOTE] Storm Logo Contest - Round 1

#10: 3pts #6: 2pts From: jose farfan [mailto:josef...@gmail.com] Sent: 21 May 2014 11:38 To: user@storm.incubator.apache.org Subject: Re: [VOTE] Storm Logo Contest - Round 1 #6 - 5 pts On Thu, May 15, 2014 at 6:28 PM, P. Taylor Goetz mailto:ptgo...@gmail.com>> wrote: This is a call to vote on s

Re: [VOTE] Storm Logo Contest - Round 1

6 - 1 pt 9 - 2 pts 11 - 2 pts On 5/15/14, P. Taylor Goetz wrote: > This is a call to vote on selecting the top 3 Storm logos from the 11 > entries received. This is the first of two rounds of voting. In the first > round the top 3 entries will be selected to move onto the second round where > the

Re: [VOTE] Storm Logo Contest - Round 1

#6 - 5 pts On Thu, May 15, 2014 at 6:28 PM, P. Taylor Goetz wrote: > This is a call to vote on selecting the top 3 Storm logos from the 11 > entries received. This is the first of two rounds of voting. In the first > round the top 3 entries will be selected to move onto the second round > where

Re: [VOTE] Storm Logo Contest - Round 1

#1 - 5pts -- Gianmarco On 20 May 2014 19:33, Megan Kearl wrote: > #1 - 5pts > > > On Tue, May 20, 2014 at 11:46 AM, Gaspar Muñoz wrote: > >> #9 - 5pts >> >> >> 2014-05-20 18:43 GMT+02:00 Tom Brown : >> >> #9 - 5pts >>> >>> >>> >>> >>> On Tue, May 20, 2014 at 7:20 AM, Milinda Pathirage < >>> m

RE: Recovering From Zookeeper Failure

This has already been reported: https://issues.apache.org/jira/browse/STORM-307. The workaround we’ve implemented is, in our storm init scripts, we always delete the {storm.home}/supervisor and {storm.home}/workers directories before starting the supervisor. From: Saurabh Agarwal (BLOOMBERG/ 73

Re: Recovering From Zookeeper Failure

We also had the same issue. We also need to delete supervisor and worker directories in order to restart the supervisor processes after these processes die. Is this known bug? Is there any cleaner way to restart? Thanks , Saurabh. - Original Message - From: Ryan Chan At: Friday, M

Re: storm to HDFS and lambda architecture

Answered my own question, after a deeper search in google, the point is Storm-on-YARN, is designed for use in the batch layer (over a HDFS cluster), so the Storm-to-HDFS connector has sense in this batch layer. On the other hand, the Storm in the speed layer has no sense the use of HDFS connector.