ATTENTION - storm-u...@googlegroups.com is now deprecated.

2013-11-05 Thread P. Taylor Goetz
The google groups account storm-u...@googlegroups.com is now officially deprecated in favor of the Apache-hosted user/dev mailing lists. ## Storm Users Storm users should now send messages and subscribe to u...@storm.incubator.apache.com. You can subscribe to this list by sending an email to u

Re: [storm-user] ATTENTION - storm-u...@googlegroups.com is now deprecated.

2013-11-07 Thread P. Taylor Goetz
PR08MB315.namprd08.prod.outlook.com > > u...@storm.incubator.apache.com > Remote Server returned '550 5.7.0 Proxy session setup failed on Frontend with > '554 5.4.4 SMTPSEND.DNS.NonExistentDomain; nonexistent domain'' > > > Le mardi 5 novembre 2

[DISCUSS] Storm Release 0.9.0-rc3

2013-11-08 Thread P. Taylor Goetz
I think it’s time for another release candidate. Several bug fixes have been merged and there are a few outstanding pull requests waiting for a 3rd +1 from committers. The list of open pull requests targeted for 0.9.0 can be found here: https://github.com/nathanmarz/storm/issues?milestone=1&pag

Re: Future of Logback?

2013-11-11 Thread P. Taylor Goetz
No, and I doubt there would be any pressure to switch to log4j. If you look at the logback website[1] there are many Apache projects using it. [1] http://logback.qos.ch On Nov 11, 2013, at 4:09 PM, Jon wrote: > With the switch of Storm to Apache, are there any plans to move away from > Logba

Re: [storm-user] [SOLVED] Heartbeat deadlocking?

2013-11-13 Thread P. Taylor Goetz
This turned out to be a red herring (I work with Brian). The root cause was that GC logging for workers had been turned on without specifying an output file. As a result the GC logging went to standard output without being redirected to logback. Eventually the buffer filled and the JVM would ha

Re: [DISCUSS] Storm Release 0.9.0-rc3

2013-11-15 Thread P. Taylor Goetz
le to push > out the first Apache Storm release. > > > On Fri, Nov 8, 2013 at 3:03 PM, P. Taylor Goetz wrote: > I think it’s time for another release candidate. Several bug fixes have been > merged and there are a few outstanding pull requests waiting for a 3rd +1 > from commi

[RELEASE] Storm 0.9.0-rc3

2013-11-18 Thread P. Taylor Goetz
The Storm team is pleased to announce the release of Storm version 0.9.0-rc3. NOTE: This release is neither a top-level Apache release nor an Apache Incubator release. Once 0.9.0 (final) has been released, Storm migrate to the Apache release process. Version 0.9.1 will be the first Apache incuba

Cluster Deployment with Apache Whirr

2013-11-21 Thread P. Taylor Goetz
For a project I’m working on, I had to use Apache Whirr to deploy to EC2. Since Whirr didn’t have a Storm service, I put one together: https://github.com/ptgoetz/whirr-storm Functionally it is similar to storm-deploy. One benefit of Whirr is the existing support for services that often go hand-

Re: Get errors when trying to run the storm-jms example topology

2013-11-21 Thread P. Taylor Goetz
Hi Pete, I’ve updated the version information in the poms, so users won’t have to go through that process anymore. I tried to reproduce the issue you are seeing but running with a fresh download/install of ActiveMQ 5.9.0 (no changes to the poms were necessary). The only way I could reproduce t

Re: [storm-user] Re: [RELEASE] Storm 0.9.0-rc3

2013-11-26 Thread P. Taylor Goetz
0.9.0-rc3 artifacts are there: https://clojars.org/search?q=Storm - Taylor > On Nov 26, 2013, at 4:54 AM, Nicolas wrote: > > Hi, > > It seems that it's not available on clojars (still showing 0.9.0-rc2 for me). > Can you check please ? > > Thanks :) > Nicolas > -- > You received this messa

Re: [storm-user] [RELEASE] Storm 0.9.0-rc3

2013-11-26 Thread P. Taylor Goetz
gt; On 2013年11月26日, at 下午9:39, P. Taylor Goetz wrote: > >> 0.9.0-rc3 artifacts are there: >> >> https://clojars.org/search?q=Storm >> >> - Taylor >> >> On Nov 26, 2013, at 4:54 AM, Nicolas wrote: >> >>> Hi, >>> >>>

[RELEASE] Storm 0.9.0

2013-12-04 Thread P. Taylor Goetz
The Storm team is pleased to announce the release of Storm version 0.9.0. NOTE: This release is neither a top-level Apache release nor an Apache Incubator release. Storm will now migrate to the Apache release process. Version 0.9.1 will be the first Apache incubator release. Storm is a distribu

Re: [RELEASE] Storm 0.9.0

2013-12-04 Thread P. Taylor Goetz
tes? Past releases had notes under > the "Blog" section <http://storm-project.net/blog.html> of the website. > > > On Wed, Dec 4, 2013 at 8:03 PM, P. Taylor Goetz wrote: > >> The Storm team is pleased to announce the release of Storm version 0.9.0. >>

[CORRECTION] [RELEASE] Storm 0.9.0

2013-12-04 Thread P. Taylor Goetz
CORRECTION: The HREF for the release tag link was wrong, and there was a paragraph erroneously describing this release as a release candidate. This is a FINAL release. Corrected announcement below: The Storm team is pleased to announce the release of Storm version 0.9.0. NOTE: This release is

Re: Finding a home for storm-graph, storm-elastic-search, and storm-cassandra?

2013-12-04 Thread P. Taylor Goetz
(shifting discussion to dev@) Brian, That’s a good question. There’s been some discussion in this respect, specifically with regard to storm-kafka, since that is a widely used module within the storm community. The short answer is we don’t know quite yet. I think the near-term goal is to get

Re: [storm-user] Re: [CORRECTION] [RELEASE] Storm 0.9.0

2013-12-05 Thread P. Taylor Goetz
Guillaume, Thank you for calling this out. I inadvertently built the release against Java 7. Storm 0.9.0 should work with both Java 6 and Java 7. I will rectify this today by releasing what will likely be version 0.9.0.1. There will be no code changes, it will simply restore compatibility with

Re: [CORRECTION] [RELEASE] Storm 0.9.0

2013-12-05 Thread P. Taylor Goetz
s in this release? The last activity on that branch was PR #722, and per > Github, it's quite far from the master. The last change was nearly 3 weeks > ago... > > https://github.com/nathanmarz/storm/tree/0.9.0 > > > > On Wed, Dec 4, 2013 at 8:51 PM, P. Taylor Goetz

Re: [CORRECTION] [RELEASE] Storm 0.9.0

2013-12-05 Thread P. Taylor Goetz
The 0.9.0 tag has been corrected: https://github.com/nathanmarz/storm/tree/0.9.0 - Taylor On Dec 5, 2013, at 8:21 AM, P. Taylor Goetz wrote: > Yep. Something is wrong with that tag, at least on Github. I suspect it has > something to do with a name collision between the 0.9.0 *tag*

[RELEASE] Storm 0.9.0.1

2013-12-05 Thread P. Taylor Goetz
The Storm team is pleased to announce the release of Storm version 0.9.0.1. This release simply fixes binary compatibility with Java 1.6. There are no functional changes. NOTE: This release is neither a top-level Apache release nor an Apache Incubator release. Storm will now migrate to the Apac

What's new in Storm 0.9.0

2013-12-08 Thread P. Taylor Goetz
http://storm-project.net/2013/12/08/storm090-released.html signature.asc Description: Message signed with OpenPGP using GPGMail

Re: What's new in Storm 0.9.0

2013-12-08 Thread P. Taylor Goetz
be emphasized more, rather than simply listing it in the changelog? It causes > major headaches if you include the log4j adapter in your deployed jar. > > > On Sun, Dec 8, 2013 at 11:16 PM, P. Taylor Goetz wrote: > http://storm-project.net/2013/12/08/storm090-released.html >

Test

2013-12-12 Thread P. Taylor Goetz
Test

Re: [storm-user] Nimbus goes out of memory

2013-12-13 Thread P. Taylor Goetz
cant provide much information about the process. But it basically involves > sending some random packets to check for vulnerabilities. It would be a tough > process to get the security group add exceptions.. > > Thanks, > Kiran > > On Thursday, December 12, 2013 12:52:3

Re: New kafka spout

2013-12-19 Thread P. Taylor Goetz
David, I had the same questions and came to the same conclusions (also just a cursory look over the code). That’s why the storm-kafka implementation uses Kafka’s low-level API — so it can keep track of the offsets and not have to store messages in memory. Mattijs — I’d be interested in heari

Re: Is it possible to find out the active supervisor nodes through a program?

2014-01-07 Thread P. Taylor Goetz
Richards, Yes, the Nimbus thrift interface provides an API for querying a Storm cluster, and is exactly what Storm UI uses to get the information it displays. As far as “active” supervisors, it depends on what you mean by “active”. If by active you mean “up and known to Nimbus” then the list r

Re: Logging

2014-01-13 Thread P. Taylor Goetz
Hi Klaus, If you put a logback.xml file on the class path it should work for local mode. - Taylor > On Jan 13, 2014, at 11:37 AM, Klausen Schaefersinho > wrote: > > Hi, > > I am using the maven dependency to develop so its not using the installation > directory. > > Cheers, > > Klaus > >

New Build System

2014-01-14 Thread P. Taylor Goetz
As of today the Storm master branch now uses Maven (as opposed to Leiningen) for it’s build system. For those familiar with Maven, the transition should be relatively easy. For those transitioning from Leiningen, here are a few Leiningen --> Maven mappings: “lein sub install”--> “mvn install”

Re: quick question ab failed tuple on only 1 of the 2 bolts

2014-01-23 Thread P. Taylor Goetz
Correct. The spout will replay the tuple, so it will go through the entire graph. - Taylor > On Jan 23, 2014, at 10:46 AM, Adrian Mocanu wrote: > > Any takers on this? > > I’m under the impression that storm will resend to both tuples not only to > the one that send the fail signal. >

Re: trident simplest aggregator into cassandra

2014-02-03 Thread P. Taylor Goetz
Which project did the CassandraState implementation come from? On Feb 3, 2014, at 5:09 PM, Adrian Mocanu wrote: > Hi > I'm using Trident to perform some aggregations and store the results into > cassandra. > > I've looked at IBackingMap and specifically at some tutorials on trident site > a

Re: trident simplest aggregator into cassandra

2014-02-03 Thread P. Taylor Goetz
0.4.0-rc4") > which is very different. I did not use that one – I thought that was the > older version. > > Thanks > A > From: P. Taylor Goetz [mailto:ptgo...@gmail.com] > Sent: February-03-14 5:21 PM > To: user@storm.incubator.apache.org > Subject: Re:

Storm Incubator Website

2014-02-04 Thread P. Taylor Goetz
The storm incubator website has been published: http://storm.incubator.apache.org It is basically a merging of storm-project.net with the content of the storm wiki, with the necessary incubator branding/notice applied. I will follow up with instructions for how committers/contributors can updat

Re: http-client version conflict

2014-02-05 Thread P. Taylor Goetz
Your best bet is probably to use the shade plugin to relocate the http-client package so it doesn’t conflict with the version storm uses. Storm does this with the libtrhift dependency in storm-core: https://github.com/apache/incubator-storm/blob/master/storm-core/pom.xml#L220 (You can ignore t

Re: Rebalance topology throws java.lang.StringIndexOutOfBoundsException: String index out of range: -1

2014-02-05 Thread P. Taylor Goetz
Hi Florin, I opened an issue to track this: https://issues.apache.org/jira/browse/STORM-214 Regards, Taylor On Feb 5, 2014, at 4:30 AM, Spico Florin wrote: > > Hello! > In the mentioned version for Windows, the rebalance command arguments should > not be passed as indicated in > > https

Re: Svend's blog - several questions

2014-02-05 Thread P. Taylor Goetz
Hi Adrian, I’ll apologize up-front for not answering your questions now, but I’ll try to follow up later when I have a little more bandwidth. In the meantime, check out the storm documentation on the new Storm website: http://storm.incubator.apache.org, which includes the latest javadoc for the

Re: Svend's blog - several questions

2014-02-06 Thread P. Taylor Goetz
Thanks Svend. Good explanation. Adrian, The storm-cassandra documentation could be better in terms of explaining how to use the MapState implementation, but theres a unit test that demonstrates basic usage: https://github.com/hmsonline/storm-cassandra/blob/master/src/test/java/com/hmsonline/st

Re: Svend's blog - several questions

2014-02-06 Thread P. Taylor Goetz
ot; % "hms-cassandra-rest" % "1.0.0" > "com.github.ptgoetz" % "storm-cassandra" % "0.1.2" > > And now > "com.hmsonline" % "storm-cassandra" % "0.4.0-rc4" > from > http://mvnrepository.com/artifact/com.

Re: http-client version conflict

2014-02-06 Thread P. Taylor Goetz
any > other issues as a result. > > > On Wed, Feb 5, 2014 at 10:31 AM, P. Taylor Goetz wrote: > Your best bet is probably to use the shade plugin to relocate the > http-client package so it doesn’t conflict with the version storm uses. > > Storm does this with the

Re: http-client version conflict

2014-02-06 Thread P. Taylor Goetz
> extension joda-time) really needed? Or just a small fraction of the > functionality in that library? I can probably pitch in some of the effort > required to do this, if this is the direction people want to go in. > > > > > On Thu, Feb 6, 2014 at 8:44 PM, P. Taylor Goetz wrote:

Re: Is there a way to add a custom jar (or directory of jars) to the Storm classpath without copying the jar(s) to Storm lib folder

2014-02-07 Thread P. Taylor Goetz
Bijoy, Out of curiosity, why can’t you bundle your dependencies with the topology jar? If you have to then you can put your topology dependencies in $STORM_HOME/lib, but you should be careful with replacing Storm’s core dependencies since it could lead to unpredictable results. - Taylor On Fe

Re: Storm 0.9.0.1 - topology halts processing after a while

2014-02-13 Thread P. Taylor Goetz
> tried to reopen about 9 months ago. > > Marc > > On Wed, Feb 12, 2014 at 03:50:52PM -0500, P. Taylor Goetz wrote: >> It looks like you have GC logging turned on, but you haven t specified an >> output file: >> >> "worker.chil

Re: Storm 0.9.0.1 and Zookeeper 3.4.5 hung issue.

2014-02-14 Thread P. Taylor Goetz
Look for project.clj in storm-core. -Taylor > On Feb 14, 2014, at 6:12 PM, "Saurabh Agarwal (BLOOMBERG/ 731 LEXIN)" > wrote: > > > After adding the changes, I am building storm 0.9.0.1 source code using > Leiningen . Can anyone point me out from where lein pick up the classpath? I > am tryi

Re: How to consume from last offset when topology restarts(STORM-KAFKA)

2014-02-16 Thread P. Taylor Goetz
If turn off forceOffsetTime, it should resume from the last offset stored in zookeeper. - Taylor On Feb 16, 2014, at 12:35 PM, Chitra Raveendran wrote: > Hi > > So according to this logic I should set the timestamp parameter to the value > when the topology was stopped ? > > But how do w

Re: storm version 0.9.0.1 and zeromq 2.2.0 compatibility

2014-02-19 Thread P. Taylor Goetz
Hi Andrew, There’s not a way that I know of to use multiple versions of zeromq with Storms zeromq transport. I think your best option is to use Storm’s Netty transport to avoid zeromq version conflicts. - Taylor On Feb 19, 2014, at 11:06 AM, Andrew Milkowski wrote: > thanks so much bijoy, w

[RELEASE] Apache Storm 0.9.1-incubating released

2014-02-22 Thread P. Taylor Goetz
The Storm team is pleased to announce the release of Apache Storm version 0.9.1-incubating. This is our first Apache release. Storm is a distributed, fault-tolerant, and high-performance realtime computation system that provides strong guarantees on the processing of data. You can read more abo

Re: Trident localOrShuffle

2014-02-24 Thread P. Taylor Goetz
Hi Rob, You are correct about needing a repartitioning operation in order for the spout to have a different parallelism. Can you file a JIRA to implement localOrShuffle()? Taylor > On Feb 24, 2014, at 5:31 PM, Robert Turner wrote: > > Thanks Tom for the quick reply. > > I need a single spo

[DISCUSS] Pulling "Contrib" Modules into Apache

2014-02-25 Thread P. Taylor Goetz
A while back I opened STORM-206 [1] to capture ideas for pulling in “contrib” modules to the Apache codebase. In the past, we had the storm-contrib github project [2] which subsequently got broken up into individual projects hosted on the stormprocessor github group [3] and elsewhere. The prob

Re: [DISCUSS] Pulling "Contrib" Modules into Apache

2014-02-25 Thread P. Taylor Goetz
trom community. I also like to contribute to > this process. > > Thanks > Milinda > > On Tue, Feb 25, 2014 at 5:28 PM, P. Taylor Goetz wrote: > > A while back I opened STORM-206 [1] to capture ideas for pulling in > > "contrib" modules to the Apache codebase. >

Re: [DISCUSS] Pulling "Contrib" Modules into Apache

2014-02-26 Thread P. Taylor Goetz
sure > many would be, am willing to offer support and time to working through how to > set this up and helping with the implementation if it is decided to pursue > some solution. > I hope these views are taken in the sprit they are made, to make this > incredible system even b

Re: [DISCUSS] Pulling "Contrib" Modules into Apache

2014-02-26 Thread P. Taylor Goetz
down a request to create juju charms > for storm. > > —Bobby > > From: "P. Taylor Goetz" mailto:ptgo...@gmail.com>> > Reply-To: > mailto:d...@storm.incubator.apache.org>> > Date: Wednesday, February 26, 2014 at 1:21 PM > To: mailto:d...@storm.i

Re: Spout missing Acks when a Bolt uses JRuby

2014-02-26 Thread P. Taylor Goetz
Hi Jonathan, I've used jruby fairly extensively with storm (though with the trident API), but it's been a while so I'm rusty. Initializing the jruby runtime is very expensive, so you should do that in the prepare() method of your bolt. That means you'll have to store it as an instance variable

Re: [DISCUSS] Pulling "Contrib" Modules into Apache

2014-03-04 Thread P. Taylor Goetz
matter the > actual outcome I'm sure the state of the project will be improved. > > Best, > Michael > > > > [1] https://github.com/elasticsearch/logstash > [2] http://mesosphere.io/learn/run-storm-on-mesos/#step-7 > [3] https://github.com/miguno/puppet-storm

Re: Running Storm on Local Mode does not utilize the multi-core capacity?

2014-03-04 Thread P. Taylor Goetz
Local mode is really only intended to make developing and testing of Storm components easier by approximating the behavior of a real cluster locally. And it is certainly not meant for production. You should see better resource utilization when you deploy to a real cluster. - Taylor On Mar 4,

Re: [DISCUSS] Pulling "Contrib" Modules into Apache

2014-03-05 Thread P. Taylor Goetz
Mattjis, That’s awesome that the NFI might be willing to contribute source code to Storm! I hope I didn’t come across as suggesting that pulling in HolmesNL/kafka-spout was not an option. That’s up to the Storm community to decide. The point I was trying to make was that contributions from corp

Re: Error when trying to use multilang in a project built from scratch (not storm-starter)

2014-03-10 Thread P. Taylor Goetz
I don't have access to a windows machine at the moment, but does this help? http://support.microsoft.com/kb/832434 > On Mar 10, 2014, at 4:51 PM, Chris James > wrote: > > Reposting since I posted this before at a poor time and got no response. > > I'm trying out a storm project built from scr

Re: Nimbus Fails After Uploading Topology - Reading Too Large of Frame Size

2014-03-11 Thread P. Taylor Goetz
Hi Robert, Can you provide additional details, like what storm version you are using, etc.? -Taylor > On Mar 11, 2014, at 6:57 PM, Robert Lee wrote: > > > > After submitting my topology via the storm jar command: > > > > 562 [main] INFO backtype.storm.StormSubmitter - Upload

Re: Nimbus Fails After Uploading Topology - Reading Too Large of Frame Size

2014-03-12 Thread P. Taylor Goetz
.submitTopology(name, config, > sentenceAggregationTopology.buildTopology()); > } else { > LocalDRPC drpc = new LocalDRPC(); > config.setNumWorkers(2); > config.setDebug(true); > config.setMaxTaskParallelism(2); >

Re: [DISCUSS] Pulling "Contrib" Modules into Apache

2014-03-12 Thread P. Taylor Goetz
27;m sure the state of the project will be improved. > > Best, > Michael > > > > [1] https://github.com/elasticsearch/logstash > [2] http://mesosphere.io/learn/run-storm-on-mesos/#step-7 > [3] https://github.com/miguno/puppet-storm > [4] https://github.com/apache/i

Re: [DISCUSS] Pulling "Contrib" Modules into Apache

2014-03-13 Thread P. Taylor Goetz
(storm-core, storm-netty, etc) and things like storm-starter and storm-kafka >> are something different. I don't like "connectors" because something like >> storm-starter is not a connector. Maybe we call them "extras"? >> >> I would say jus

Re: [404 Error] at storm.incubator.apache.org

2014-03-14 Thread P. Taylor Goetz
Thanks for pointing this out. The link has been fixed. - Taylor On Mar 13, 2014, at 1:39 AM, Chengwei Yang wrote: > Hi List, > > Hope some site admin are here. > > I found that there is a page not found error on > storm.incubator.apache.org. > > http://storm.incubator.apache.org/documentatio

Re: nimbus.thrift.max_buffer_size

2014-03-18 Thread P. Taylor Goetz
It uploads the file in small (1024*5 bytes) chunks. Does this happen every time (i.e. reproducible)? What is the size of your topology jar? Can you post the server side message (I want to see the length it output). - Taylor On Mar 18, 2014, at 3:40 PM, Adam Lewis wrote: > Upon upgrading from

Re: nimbus.thrift.max_buffer_size

2014-03-18 Thread P. Taylor Goetz
g I will break each DRPC spout into its own topology, which I've > needed to do anyway since right now they seem to block each other from > processing even though I don't need requests to separate DRPCs to be wholly > ordered in the way I imagine trident would try

Re: Is Storm a right tool for processing of thousands of small tasks?

2014-03-19 Thread P. Taylor Goetz
Have you considered using DRPC [1]? [1] http://storm.incubator.apache.org/documentation/Distributed-RPC.html On Mar 19, 2014, at 11:21 AM, Eugene Dzhurinsky wrote: > Hello! > > I'm evaluating Storm for the project, which involves processing of many > distinct small tasks in the following way:

Re: Topology is stuck

2014-04-09 Thread P. Taylor Goetz
Hey Jason, Do you know a way to reliably reproduce this? If so can you share the steps? -Taylor > On Apr 9, 2014, at 5:52 PM, Jason Jackson wrote: > > Fyi we're using Summingbird in production not Trident. However summingbird > does not give you exactly once semantics, it does give you a high

[ANNOUNCE] Storm Logo Contest Now Open

2014-04-10 Thread P. Taylor Goetz
The storm logo contest is now open. If you or someone you know has great design skills, then this is the chance to help establish Storm’s brand with an awesome new logo. The contest is open now, and will run through 4/30/2014. More details are available on the storm website: http://bit.ly/1jwM

Re: Topology submission exception caused by Class Not Found backtype.storm.daemon.nimbus$normalize_conf$get_merged_conf_val__3916$fn__3917

2014-04-22 Thread P. Taylor Goetz
I have seen this as well and thought I was going nuts. In my testing I could reliably reproduce it in local mode against 0.9.1-incubating. What I noticed in my testing: - It only occurred when a custom logback.xml was on the class path. - More specifically, it only happened when there was an “ap

Re: How resilient is Storm w/o supervision

2014-05-02 Thread P. Taylor Goetz
I don’t think you need root to run supervisord: http://supervisord.org/running.html If you’re just testing something out, and don’t mind your cluster going down, then running without supervision is okay. But I would NEVER suggest someone run Storm’s daemons without supervision in a production e

Re: Question about OpaqueTridentKafkaSpout

2014-05-07 Thread P. Taylor Goetz
It all depends on the nature of the spout. With a transactional spout, batches are always the same, even if replayed. With an opaque spout, batches can change. But you have the guarantee that a tuple will only ever be processed successfully in a single batch. If a tuple fails in one batch, it c

Re: storm trident question

2014-05-11 Thread P. Taylor Goetz
I would be suspicious of the results in that paper. The same with a similar study published by IBM comparing their streaming offering to storm. It's not that I think the studies were invalid. It's that I have no way to tell. Without publishing the code and configuration used in the study, there

Re: Weirdness running topology on multiple nodes

2014-05-16 Thread P. Taylor Goetz
Hi Justin, Can you share your storm.yaml config file? Do you have any firewall software running on any of the machines in your cluster? - Taylor On May 7, 2014, at 11:11 AM, Justin Workman wrote: > We have spent the better part of 2 weeks now trying to get a pretty basic > topology running

[VOTE] Storm Logo Contest - Round 1

2014-05-16 Thread P. Taylor Goetz
This is a call to vote on selecting the top 3 Storm logos from the 11 entries received. This is the first of two rounds of voting. In the first round the top 3 entries will be selected to move onto the second round where the winner will be selected. The entries can be viewed on the storm websit

Re: Sometimes topology crashed with internal exception

2014-05-21 Thread P. Taylor Goetz
Are you using the OutputCollector to emit in a separate thread (i.e. Outside of the execute() method.)? As the wiki states, this will cause the problem you are seeing? -Taylor > On May 21, 2014, at 7:24 AM, Irek Khasyanov wrote: > > Hello. > > I have strange problem with by topology, sometim

Re: Sample project using Storm and Kafka (using Scala 2.10)

2014-05-21 Thread P. Taylor Goetz
storm-kafka is now part of the Storm project. When 0.9.2 is released (shortly) it will include the Kafka spout (for Kafka 0.8.x). - Taylor On May 21, 2014, at 4:32 PM, Marco wrote: > Hi, > I'm having some troubles understanding how to boostrap a Kafka + Storm > project. > > Specifically I d

Re: Sample project using Storm and Kafka (using Scala 2.10)

2014-05-21 Thread P. Taylor Goetz
> (like kafka) to produce multiple version) > > János Háber > Fine Solution Ltd > > > > On Wed, May 21, 2014 at 11:28 PM, János Háber > wrote: > > On Wed, May 21, 2014 at 11:27 PM, P. Taylor Goetz wrote: > ill include the Kafka spout (for Kafka 0.8.x). > >

Re: Sample project using Storm and Kafka (using Scala 2.10)

2014-05-21 Thread P. Taylor Goetz
too, no > exception. > > b0c1 > > János Háber > Fine Solution Ltd > > > >> On Wed, May 21, 2014 at 11:48 PM, P. Taylor Goetz wrote: >> If you build yourself, you can do the following: >> >> mvn install -DscalaVersion=2.10.3 -DkafkaArtifact=kafka_2

Re: Which daemon tool for Storm nodes?

2014-05-21 Thread P. Taylor Goetz
Exactly what Michael said... Any good process monitoring tool will work. The only issue I've had with supervisord is not with the software, but rather the name conflict with the Storm supervisor process -- it can confuse the hell out of some people. -Taylor > On May 21, 2014, at 9:29 PM, Mich

Re: Sample project using Storm and Kafka (using Scala 2.10)

2014-05-22 Thread P. Taylor Goetz
(mixing > Clojure and Java for example). > > Also, I agree with all the points János made. > Having it on Maven Central would be THE perfect starting point for any > (Scala) developer willing to give it a spin. > > ps: Scala 2.11 is just out > > -Marco > Il Gio

[RESULT] [VOTE] Storm Logo Contest - Round 1

2014-05-27 Thread P. Taylor Goetz
, 2014, at 12:28 PM, P. Taylor Goetz wrote: > This is a call to vote on selecting the top 3 Storm logos from the 11 entries > received. This is the first of two rounds of voting. In the first round the > top 3 entries will be selected to move onto the second round where the winner

Re: Building Storm

2014-05-27 Thread P. Taylor Goetz
I'll do a couple tests, but for the most part it should just work on OSX, etc. (Storm releases are built on OSX). What version of maven are you using? Have you tried with the latest version? -Taylor > On May 27, 2014, at 5:54 PM, Przemek Grzędzielski > wrote: > > Hi guys, > > got exactly th

Re: Storm failover, is it possible to know the last succeeded bolt when fail method gets called in spout?

2014-05-27 Thread P. Taylor Goetz
In short, no. Not today. Storm's acking mechanism is largely stateless (one long can track an entire tuple tree),which is one of the reasons it is so efficient. But the acking mechanism is also based on Storms core primitives, so it is entirely possible. There is a JIRA for adding additional m

Re: Building Storm

2014-05-28 Thread P. Taylor Goetz
t what went wrong, you can usually find it by grep’ing for errors in the test reports: grep -ir error ./storm-core/target/test-reports/ - Taylor On May 28, 2014, at 11:01 AM, Justin Workman wrote: > On Ubuntu 12.04 I have tried with Maven 3.0.4 and now the latest 3.2.1. > > > O

Re: logging 'failed' tuples in mastercoord-bg0

2014-05-28 Thread P. Taylor Goetz
"Silent replays" are usually a sign of batches timing out. By default storm uses a timeout value of thirty seconds. Try upping that value and setting TOPOLOGY_SPOUT_MAX_PENDING to a very low value like 1. In trident that controls how many batches can be in-flight at a time. -Taylor > On May 2

Re: All tuples are going to same worker

2014-05-28 Thread P. Taylor Goetz
Fields grouping uses a mod hash function to determine which task to send a tuple. It sounds like there's not enough variety in the field values you are grouping such that they are all getting sent to the same task. Without seeing your code and data I can't tell for sure. -Taylor > On May 28,

Re: All tuples are going to same worker

2014-05-28 Thread P. Taylor Goetz
#"); > this.collector.emit(new Values(null,null,activity)); > } > -------- > > > HBSt

Re: Trident, ZooKeeper and Kafka

2014-05-29 Thread P. Taylor Goetz
Not really. It’s not really intuitive. By setting forceFromStart = true, you are saying that you want to start from a specific position, rather than the last offset stored in ZooKeeper. Then you specify the position with -1, -2, or a specific point in time (in milliseconds). - Taylor On May 29

Re: Workers constantly restarted due to session timeout

2014-05-29 Thread P. Taylor Goetz
Can you share your storm config and version? > On May 29, 2014, at 12:45 PM, Michael Dev wrote: > > Derek, > > We are currently running with -Xmx60G and only about 20-30G of that has been > observed to be used. I'm still observing workers restarted every 2 minutes. > > What timeout is relevan

New Committer/PPMC Member: Michael G. Noll

2014-05-29 Thread P. Taylor Goetz
The Podling Project Management Committee (PPMC) for Apache Storm has asked Michael G. Noll to become a committer/PPMC member and we are pleased to announce that he has accepted. Michael has contributed to Storm in many ways, including code patches, community support, and high quality documentat

[VOTE] Storm Logo Contest - Final Round

2014-06-09 Thread P. Taylor Goetz
This is a call to vote on selecting the winning Storm logo from the 3 finalists. The three candidates are: * [No. 6 - Alec Bartos](http://storm.incubator.apache.org/2014/04/23/logo-abartos.html) * [No. 9 - Jennifer Lee](http://storm.incubator.apache.org/2014/04/29/logo-jlee1.html) * [No. 1

Re: [VOTE] Storm Logo Contest - Final Round

2014-06-09 Thread P. Taylor Goetz
#10 - 5 pts. On Jun 9, 2014, at 2:38 PM, P. Taylor Goetz wrote: > This is a call to vote on selecting the winning Storm logo from the 3 > finalists. > > The three candidates are: > > * [No. 6 - Alec > Bartos](http://storm.incubator.apache.org/2014/04/23/logo-abar

Re: Apache Storm vs Apache Spark

2014-06-09 Thread P. Taylor Goetz
The way I usually describe the difference is that Spark is a batch processing framework that also does micro-batching (Spark Streaming), while Storm is a stream processing framework that also does micro-batching (Trident). So architecturally they are very different, but have some similarity on t

Re: JVM instances and Workers

2014-06-09 Thread P. Taylor Goetz
Separate. Supervisors spawn worker JVMs when topologies are deployed. Workers are also dedicated to one topology. -Taylor > On Jun 9, 2014, at 5:44 PM, Nima Movafaghrad > wrote: > > Good afternoon Storm community, > > I have a quick question. Do worker processes run in separate JVMs or do t

Re: standard DRPC server deployment

2014-06-09 Thread P. Taylor Goetz
I generally put one on the same node as nimbus, and then monitor it to make sure it doesn't get overloaded. If it start to see heavy load, then consider moving it to a dedicated node or multiple nodes + load balancing. Ultimately it comes down to your use case and usage. Monitor usage and adjust

Re: Storm/Hbase Bolt Performance

2014-06-09 Thread P. Taylor Goetz
ttled. >>>> >>>> You should also try attaching a profiler to your bolt, and see what's >>>> holding it up. Are you doing batched puts (or puts being committed on a >>>> separate thread)? That could also cause substantial improvements. >>&

Re: Storm 0.9.2-incubating release

2014-06-17 Thread P. Taylor Goetz
A release candidate has been cut, and is currently being voted on by the PPMC. If that vote passes, it will move on to vote by the IPMC. If both votes pass, it will be released. Each vote stays open for at least 72 hours. If approved, there is also a 72 hour waiting period to allow download mir

[RESULT] [VOTE] Storm Logo Contest - Final Round

2014-06-17 Thread P. Taylor Goetz
| |:-|-:|--:| |6 - Alec Bartos | 2| 41| |9 - Jennifer Lee | 7| 111 | |10 - Jennifer Lee | 26 | 123 | -Taylor On Jun 9, 2014, at 2:38 PM, P. Taylor Goetz wrote: > This is a call to vote

Re: Trident Binned Aggregation Design Help

2014-06-17 Thread P. Taylor Goetz
Andrew/Adam, Partitioning operations like groupBy() form the bolt boundaries in trident topologies, so the more you have the more bolts you will have and thus, potentially, more network transfer. What backing store are you using for persistence? If you are using something with counter support

Re: Topology Memory leaks

2014-06-17 Thread P. Taylor Goetz
I've not seen this, (un)fortunately. :) Are there any other relevant details you might be able to provide? Or better yet can you distill it down to a bare bones topology that reproduces it and share the code? -Taylor > On Jun 17, 2014, at 6:27 PM, Michael Rose wrote: > > I've run into simila

Re: Trident Binned Aggregation Design Help

2014-06-17 Thread P. Taylor Goetz
it. Are there any > example of using either to do that? > > Andrew > > >> On Tue, Jun 17, 2014 at 3:51 PM, P. Taylor Goetz wrote: >> Andrew/Adam, >> >> Partitioning operations like groupBy() form the bolt boundaries in trident >> topolo

Re: v0.9.2-incubating and .ser files

2014-06-19 Thread P. Taylor Goetz
Hi Andrew, Thanks for pointing this out. I agree with your point about bit rot. However, we had to remove the the 0mq transport due to license incompatibilities with Apache, so any kind of release test suite would have to be maintained outside of Apache since it would likely pull in LGPL-licens

Re: v0.9.2-incubating and .ser files

2014-06-19 Thread P. Taylor Goetz
> investigate more tomorrow. We think the issue only happens with more than one > supervisor and multiple workers. > >> On Jun 19, 2014 7:32 PM, "P. Taylor Goetz" wrote: >> Hi Andrew, >> >> Thanks for pointing this out. I agree with your point ab

Re: Storm 0.9.2-incubating not on maven central, KafkaSpout release?

2014-06-20 Thread P. Taylor Goetz
It has not been officially released yet. The PPMC voted to approve it, but it still needs to be approved by the IPMC. That vote is currently underway. Only when that succeeds can we release the artifacts from staging. -Taylor > On Jun 20, 2014, at 10:29 AM, Toni Menzel wrote: > > Hi there, >

  1   2   >