NoNode exception on storm-kafka

Hi, I'm having troubles using KafkaSpout in storm-kafka. It can connect to zookeeper, but it throws a NoNode exception when trying to write the offset ZNode. Here's my setup conf: String topicName = "events"; BrokerHosts hosts = new ZkHosts("localhost:2181"); SpoutConfig conf = new SpoutConfig(h

Re: Exactly once semantics with Trident

Steps: 1. Implement IBackingMap interface for DynamoDB 2. Use OpaqueTridentKafkaSpout to read Kafka stream 3. Use OpaqueMap with your IBackingMap implementation within the topology 4. All your logic will now have exactly once semantics inside your Trident topology This is all on that wiki page. L

Re: Transferred and Acked is zeros !

Is there any help? On Wednesday, January 13, 2016, researcher cs wrote: > another thing may be help set of MaxSpoutPending is 5000 > > is there anyhitng in properties should i change it cause i'm using > defaults > > On Wed, Jan 13, 2016 at 9:05 PM, researcher cs > wrote: > >> i need to ask

Re: Storm process memory constantly increasing

+1 for Andrew, definitely agree profiling with jvisualvm or whatever is definitely something to do if you have not done already On Wed, Jan 13, 2016 at 3:30 PM, Andrew Xor wrote: > Hey, > > Care to give version of storm/jvm? Does this happen on cluster execution > only or when also running the

Re: Storm process memory constantly increasing

Hey, Care to give version of storm/jvm? Does this happen on cluster execution only or when also running the topology in local mode? Unfortunately, probably the best way to find what's really going on is to profile your topology... if you can run the topology locally this will make things quite a

Re: Storm process memory constantly increasing

Hi Nikolaos, Maybe try experimenting with max.spout.pending. You may have a buildup of tuples due to a high max.spout.pending. Would check capacity of each bolt, find which one(s) are ~ 1, add more executors for those, and see how things look then. --John On Wed, Jan 13, 2016 at 3:06 PM, Nikolao

Re: Kafka Spout failing despite all bolts acking

Hi Patrick, Depends upon whether your tuple is anchored going from KS -> Bolt 1...Bolt n. If you emit (incomingTuple,outgoingTuple), then the tuple is anchored and will not be fully acked until it has been acked in all Bolts it was sent. For example, if your topology it KS -> Bolt 1 -> Bolt 2 ->

Storm process memory constantly increasing

Hello, I am implementing a distributed algorithm for pagerank estimation using Storm. I have been having memory problems, so I decided to create a dummy implementation that does not explicitly save anything in memory, to determine whether the problem lies in my algorithm or my Storm structure. In

Re: Kafka Spout failing despite all bolts acking

Thanks for the quick response. I was assuming that once it got the ack from the first bolt, the spout was done. That’s not the case? It needs to be sure the whole topology handled the tuple? Patrick From: John Yost mailto:hokiege...@gmail.com>> Reply-To: "user@storm.apache.org

Re: Kafka Spout failing despite all bolts acking

Hi Patrick, This means the tuples emitted by the KafkaSpout are not making it through your topology within the tuple timeout. I recommend seeing which Bolt has capacity ~1, increase executors for that. Then start experimenting with max.spout.pending and timeouts. --John On Wed, Jan 13, 2016 at 1

Re: Transferred and Acked is zeros !

another thing may be help set of MaxSpoutPending is 5000 is there anyhitng in properties should i change it cause i'm using defaults On Wed, Jan 13, 2016 at 9:05 PM, researcher cs wrote: > i need to ask about those properties > > topology.executor.send.buffer.size 1024 > topology.execu

Re: Transferred and Acked is zeros !

i need to ask about those properties topology.executor.send.buffer.size 1024 topology.executor.receive.buffer.size 1024 should i change it ? i have one worker with 58 executors one spout ... 10 bolts On Wed, Jan 13, 2016 at 7:10 PM, researcher cs wrote: > @ john i tried it now but tr

Re: Logging Values in Storm UI

There's https://github.com/apache/storm/blob/master/docs/documentation/ui-rest-api.md but don't think it's available for version 0.9.6 or 0.10.0 You can also get topology information by STORM_UI_HOST:PORT/api/v1/topology/{$topology_name}-{$id}?sys=false. That API is also used by Storm UI when load

Kafka Spout failing despite all bolts acking

I’m seeing a lot of failures from a Kafka spout, despite no failures from any of the processing bolts showing any failures. I configure the spout like this: SpoutConfig spoutConfig = new SpoutConfig(new ZkHosts(zookeeper,"/brokers"), TOPIC,

Re: Exactly once semantics with Trident

I am new to Storm and not able to understand clearly how to use Trident state to achieve exactly once semantics with external database. Can you please elaborate? May be a use case with one external database as below: One of the sample use case could be as below: 1) Read an event from Kafka 2) Par

Re: Transferred and Acked is zeros !

@ john i tried it now but transeferred and acked still zeros , i noticed also this property topology.worker.childopts is null , should i set it too or not ? On Wed, Jan 13, 2016 at 6:52 PM, researcher cs wrote: > @ John Thanks for replying i set topology.stats.sample.rate to 1.0 > and for topol

Re: Transferred and Acked is zeros !

@ John Thanks for replying i set topology.stats.sample.rate to 1.0 and for topology.acker.executors: 1.0 but what if the best value should i set for both ? On Wed, Jan 13, 2016 at 6:20 PM, John Yost wrote: > I think you need to set this topology.stats.sample.rate to a higher rate > depending u

Re: Transferred and Acked is zeros !

I think you need to set this topology.stats.sample.rate to a higher rate depending upon throughput to capture values, and yes, should set acker.executors > 0. I think this combo will fix the issue. On Wed, Jan 13, 2016 at 9:45 AM, researcher cs wrote: > i noticed this properties in storm.yaml ,

Re: KafkaBolt question: number of executors

Sorry, I m not sure about the max or ideal cap for number of threads for Kafka bolt. But my guess is that, it will be more of a function of traffic than mere number of topics and partition. Moreover, in my set up of kafkabolt, each bolt handles traffic of one topic which then eventually handled by

Re: topology.stats.sample.rate

http://stackoverflow.com/questions/26276096/performace-impact-of-using-setstatssamplerate-topology-stats-sample-rate On Wed, Jan 13, 2016 at 9:51 PM, researcher cs wrote: > what is this properties used for topology.stats.sample.rate ? >

Re: Transferred and Acked is zeros !

i noticed this properties in storm.yaml , topology.acker.executors it's by default .. nothing , should i set it to value ? On Wed, Jan 13, 2016 at 5:00 AM, researcher cs wrote: > yes , here what i got > > ​and log file of drpc ended with this statement and didn't changed > [INFO] Starting D

topology.stats.sample.rate

what is this properties used for topology.stats.sample.rate ?

Re: Metrics,access and custom Log files

sorry i mean after submitted topology i found those files but empty ! how can i use it ? On Wed, Jan 13, 2016 at 4:45 PM, researcher cs wrote: > There are files in storm/logs like metrics , access and custom but > there are empty how can i use it ? >

Metrics,access and custom Log files

There are files in storm/logs like metrics , access and custom but there are empty how can i use it ?

Re: KafkaBolt question: number of executors

Hi Millind, I apologize, I need to ask my question more precisely. :) I know that I set the number of executors for storm, but the question is specifically in regards to the KafkaBolt. I am thinking that the max number of KafkaBolt executors should be no more than the number of topics times the n