Re: Storm with Python

We are building a new Storm and Python interop option that is called streamparse: https://github.com/Parsely/streamparse It includes a heavily rewritten Storm interop library and a command line tool, sparse, for managing local and remote Storm clusters. The idea is to make Storm projects as easy

Re: Storm Message Timeout

Thats not true. The tuples are ack'd as soon as all the tuples in the tuple tree is ack'd. On Fri, May 30, 2014 at 6:58 AM, Phil Burress wrote: > Stupid question here perhaps... but I've noticed that the Spout in a > Topology doesn't get ack'd until *topology.message.timeout.secs* is > reached

Storm Message Timeout

Stupid question here perhaps... but I've noticed that the Spout in a Topology doesn't get ack'd until *topology.message.timeout.secs* is reached even though all the bolts in the Topo have finished processing/ack'd. Is that the correct behavior? Seems strange to me that it doesn't ack the Spout imm

Re: LocalCluster throwing ClassNotFoundException

I had thrown this onto Stack Overflow before a coworker suggested this mailing list, so if anyone has any advice but prefers to answer there: http://stackoverflow.com/questions/23923323/classnotfoundexception-when-trying-to-start-topology On Thu, May 29, 2014 at 2:40 PM, Stephen Armstrong < steph

Re: Position in Kafka Stream

Hi, On Wed, May 28, 2014 at 11:39 AM, Tyson Norris wrote: > Thanks Cody - > I tried the BrightTag fork and still have problems with > storm 0.9.1-incubating and kafka 0.8.1, I get an error with my trident > topology (haven’t tried non-trident yet): > (venv)tnorris-osx:stormkafkamon tnorris$ ./m

Re: Position in Kafka Stream

Thanks Tyson! This blog is super helpful. I've been able to get LoggingMetrics working to an extent, however if I try to create multiple CountMetrics in the same function, I only see one show up in my NimbusUI. Does anybody know why this is ? On Thu, May 29, 2014 at 8:57 AM, Tyson Norris wrote:

Re: Storm with Python

(the reason being is that we are still running Python 2.6 but Petrel is only compatible with 2.7) On May 29, 2014, at 2:48 PM, Ashu Goel mailto:a...@shopkick.com>> wrote: Awesome! I’m looking more into using the storm.thrift to define a non-JVM DSL… does anyone have any working examples of this

Re: Storm with Python

Awesome! I’m looking more into using the storm.thrift to define a non-JVM DSL… does anyone have any working examples of this? Python preferred but any example will do. the wiki is a bit confusing... On May 28, 2014, at 1:54 PM, FRANCISCO JESUS GOMEZ RODRIGUEZ mailto:franciscojesus.gomezrodrig...

Fwd: LocalCluster throwing ClassNotFoundException

Hello, I've been hitting the following ClassNotFoundException when running in local mode: java.lang.ClassNotFoundException: backtype.storm.daemon.nimbus$normalize_conf$get_merged_conf_val__3916$fn__3917 Based on the thread: https://groups.google.com/forum/#!searchin/storm-user/classnotfoundexce

Re: New Committer/PPMC Member: Michael G. Noll

Congrats! Michael Rose (@Xorlev ) Senior Platform Engineer, FullContact mich...@fullcontact.com On Thu, May 29, 2014 at 3:01 PM, Derek Dagit wrote: > Welcome Michael! > > -- > Derek > > > On 5/29/14, 15:58, P. Taylor Goetz wrote: > >> T

Re: New Committer/PPMC Member: Michael G. Noll

Welcome Michael! -- Derek On 5/29/14, 15:58, P. Taylor Goetz wrote: The Podling Project Management Committee (PPMC) for Apache Storm has asked Michael G. Noll to become a committer/PPMC member and we are pleased to announce that he has accepted. Michael has contributed to Storm in many ways,

Re: New Committer/PPMC Member: Michael G. Noll

Congratulation Michael! Your blog posts are awesome to storm! Thanks you! On 30 May 2014 00:58, P. Taylor Goetz wrote: > The Podling Project Management Committee (PPMC) for Apache Storm has asked > Michael G. Noll to become a committer/PPMC member and we are pleased to > announce that he has a

Re: New Committer/PPMC Member: Michael G. Noll

Congratulations! Well deserved, from what I can tell! :) Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Thu, May 29, 2014 at 4:58 PM, P. Taylor Goetz wrote: > The Podling Project Management Committee (PPMC) for Apache St

New Committer/PPMC Member: Michael G. Noll

The Podling Project Management Committee (PPMC) for Apache Storm has asked Michael G. Noll to become a committer/PPMC member and we are pleased to announce that he has accepted. Michael has contributed to Storm in many ways, including code patches, community support, and high quality documentat

Re: Workers constantly restarted due to session timeout

OK, so GC is probably not the issue. Specifically, this is a connection timeout to ZK from the worker, and it is resulting in nimbus removing it from the assignments for that node. In turn, the supervisor reads the schedule and shoots the worker because it is no longer scheduled to be runnin

Re: Workers constantly restarted due to session timeout

Can you share your storm config and version? > On May 29, 2014, at 12:45 PM, Michael Dev wrote: > > Derek, > > We are currently running with -Xmx60G and only about 20-30G of that has been > observed to be used. I'm still observing workers restarted every 2 minutes. > > What timeout is relevan

Re: Workers constantly restarted due to session timeout

Do you have GC logging turned on? With a 60GB heap I could pretty easily see stop-the-world GCs taking longer than the session timeout. Michael Rose (@Xorlev ) Senior Platform Engineer, FullContact mich...@fullcontact.com On Thu, May 29,

RE: Workers constantly restarted due to session timeout

Derek, We are currently running with -Xmx60G and only about 20-30G of that has been observed to be used. I'm still observing workers restarted every 2 minutes. What timeout is relevant to increase for the heartbeats in question? Is it be a config on the Zookeeper side we can increase to make

Re: Nimbus UI fields

what might cause a tuple to be 'Acked' vs jus 'Executed'? How should I interpret these values ? Thanks On Tue, May 20, 2014 at 9:20 PM, Cody A. Ray wrote: > The two bolts which emit/transfer 0 are likely your persistentAggregate > bolts. These are *sinks* so they don't logically emit/transfer

Re: Position in Kafka Stream

I found this blog helpful: http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to Best regards, Tyson On May 29, 2014, at 8:41 AM, Raphael Hsieh mailto:raffihs...@gmail.com>> wrote: Can someone explain to me what LoggingMetrics is ? I've heard of it and people have told me to use

Re: Trident, ZooKeeper and Kafka

Not really. It’s not really intuitive. By setting forceFromStart = true, you are saying that you want to start from a specific position, rather than the last offset stored in ZooKeeper. Then you specify the position with -1, -2, or a specific point in time (in milliseconds). - Taylor On May 29

Re: Position in Kafka Stream

Can someone explain to me what LoggingMetrics is ? I've heard of it and people have told me to use it, but I can't find any documentation on it or any resources on how to use it. Thanks On Thu, May 29, 2014 at 12:06 AM, Tyson Norris wrote: > Hi - > Thanks - it turns out that the JSON parsing

Re: Trident, ZooKeeper and Kafka

By setting forceFromStart to true, aren't I telling it to start from the beginning or earliest time then ? On Thu, May 29, 2014 at 12:59 AM, Danijel Schiavuzzi wrote: > You must set both forceFromStart to true and startOffsetTime to -1 or -2. > > > On Thu, May 29, 2014 at 12:23 AM, Raphael Hsie

Re: All tuples are going to same worker

I would recommend that when you subscribe using fields grouping, you subscribe a specific stream using this method: http://nathanmarz.github.io/storm/doc/backtype/storm/topology/InputDeclarer.html#fieldsGrouping(java.lang.String, java.lang.String, backtype.storm.tuple.Fields) And when you emit, us

Re: All tuples are going to same worker

Thanks Nathan for quick reply. Here no bolt is failing. But all the tuples are routed to same worker. Remaining workers are not doing anything. I guess something is wrong with grouping which we have implemented. Could you please help me to achieve below scenario? 1. Kafkaspout pulls one tuple fr

Re: Storm data structures in Zookeeper?

You can use something like https://github.com/radekg/node-zookeeper-explorerto poke through Zookeeper yourself. The most interesting things tend to be under /transactional root, IMHO. -Cody On Thu, May 29, 2014 at 6:51 AM, @Sanjiv Singh wrote: > Hi All, > > I am looking for Storm data structur

Re: All tuples are going to same worker

Which is the bolt that is failing? Your setup seems strange to me. You use fields grouping, and have three bolts that subscribe from a router bolt. But for the three fields that you set two to null. This means, I assume, that if you send a tweet, it will always get routed to the same retweet bolt,

Re: All tuples are going to same worker

Hi All, Please help me to solve this problem. Still not, server is going down due to high load. Thanks &Regards, Riyaz On Thu, May 29, 2014 at 2:05 AM, Shaikh Riyaz wrote: > Here is the configuration. > Config conf = new Config(); > conf.setNumWorkers(10); > conf.setMaxSpoutPending(8); >

Unable to read from beginning of the topic storm-kafka-0.8-plus

Dear all, I am using Storm 0.9.1-incubating, along with kafka 0.8.1.1 with storm-kafka-0.8-plus. Initially, the library worked fine and was able to fetch messages from a kafka topic. However, after running the topology few times, now I am unable to fetch any message from the kafka topic. I believe

Storm data structures in Zookeeper?

Hi All, I am looking for Storm data structure in Zookeeper. As Storm uses Zookeeper for storing its State/meta-data and for collaboration among all participating components. I wants to explore that state/meta-data stored in zookeeper to extract some internal details. If some knows about Storm d

Storm data structures in Zookeeper?

Hi All, I am looking for Storm data structure in Zookeeper. As Storm uses Zookeeper for storing its State/meta-data and for collaboration among all participating components. I wants to explore that state/meta-data stored in zookeeper to extract some internal details. If some knows about Storm d

CGLib generated Class not accessible in EsperBolt in clustermode

Hi, I am using cglib-node2.2 with storm 0.9.1 for generating a Java-Bean (com.cts.Syslog_vmstatsClass) at run-time which is used in EsperBolt. In local mode, topology works fine but in cluster mode, it gives java.lang.ClassNotFoundException exception. Here is the error stack : 2014-05-29 12:57:59

Re: Trident, ZooKeeper and Kafka

You must set both forceFromStart to true and startOffsetTime to -1 or -2. On Thu, May 29, 2014 at 12:23 AM, Raphael Hsieh wrote: > I'm doing both tridentKafkaConfig.forceFromStart = false; as well as > tridentKafkaConfig.startOffsetTime = -1; > > Neither are working for me. Looking at my nimbus

Facing exception on raising failure (java.lang.RuntimeException: backtype.storm.topology.ReportedFailedException: :1:0:null:null:null)

Hi All, We are facing this issue when we are raising the failure on bolt. Initially raise failure is not giving any issue. But now it is giving following exception. Can you please help me in resolving this issue? java.lang.RuntimeException: backtype.storm.topology.ReportedFailedException: :1:

Re: Position in Kafka Stream

Hi - Thanks - it turns out that the JSON parsing is actually fine with HEAD, although inaccurate without the required message format (comments mention expecting an “s” property with timestamp value). My problem was that I was not specifying the spout root properly, i.e. --spoutroot /transaction