Re: Maintaining message ordering using KafkaSpout/Bolt

int set when a KafkaSpout is added to a topology, > > need to match the number of partitions in a topic? > > No. > > On 06/05/2016 11:26 AM, Matthias J. Sax wrote: > > Hi Kanagha, > > > > For reading, KafkaSpout's internally used KafkaConsumer ensures that >

Re: Maintaining message ordering using KafkaSpout/Bolt

> Does the parallelism_hint set when a KafkaSpout is added to a topology, > need to match the number of partitions in a topic? No. On 06/05/2016 11:26 AM, Matthias J. Sax wrote: > Hi Kanagha, > > For reading, KafkaSpout's internally used KafkaConsumer ensures that > data is

Re: Maintaining message ordering using KafkaSpout/Bolt

n the spout and > use fieldsGrouping on partitionId for downstream bolts. I guess, > KafkaSpout does not support this out of the box -- you can either patch > KafakSpout itself, if inherit from it to build you own > "PartionKafkaSpout" to add the partitionId to the output tupl

Re: Maintaining message ordering using KafkaSpout/Bolt

by a single bolt, you need to extract the partitionId (ie, add it to the Storm tuple) in the spout and use fieldsGrouping on partitionId for downstream bolts. I guess, KafkaSpout does not support this out of the box -- you can either patch KafakSpout itself, if inherit from it to build you own

Maintaining message ordering using KafkaSpout/Bolt

Hi, I'm looking at the documentation for using KafkaSpout/KafkaBolt. https://github.com/apache/storm/tree/master/external/storm-kafka How is ordering guaranteed while reading messages from Kafka using KafkaSpout? Does the parallelism_hint set when a KafkaSpout is added to a topology, need

Re: KafkaSpout support for Kafka 0.9 version

com> wrote: > Hi, > > Does Apache Storm KafkaSpout support 0.9 kafka version? > > Thanks > Kanagha > > > > -- > Kanagha >

KafkaSpout support for Kafka 0.9 version

Hi, Does Apache Storm KafkaSpout support 0.9 kafka version? Thanks Kanagha -- Kanagha

Re: KafkaSpout forceFromStart Issue

What happened to this issue ? Any updates ? I seem to be facing the same issue. Regards, Raki

Re: KafkaSpout throwing OutOfMemoryError

You need to allocate extra memory for the topology to run. On Wed, Sep 2, 2015 at 11:36 PM, Khalasi, Vipul Kantibhai < vipul.kantibhai.khal...@citi.com> wrote: > Hi , > > > > I am using kafkaspout in my topology ann each kafka topic have 8 > partitions and all topic atle

KafkaSpout throwing OutOfMemoryError

Hi , I am using kafkaspout in my topology ann each kafka topic have 8 partitions and all topic atleast contains 1GB of data. java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57) at java.nio.ByteBuffer.allocate(ByteBuffer.java:335

Re: KafkaSpout forceFromStart Issue

partitions. ( Kafka 0.8.2, Storm 0.9.3 ) Recently, we had to set the KafkaSpout to read from the beginning, so we temporary configured our KafkaConfig this way: kafkaConfig.forceFromStart=true kafkaConfig.startOffsetTime = OffsetRequest.EarliestTime() It worked well, but afterward

Fwd: Help: KafkaSpout not getting data from Kafka

Hi folks, I am new to both Kafka and Storm and I have problem having KafkaSpout to get data from Kafka in our three-node environment with Kafka 0.8.1.1 and Storm 0.9.3. What is working: - I have a Kafka producer (a java application) to generate random string to a topic and I was able to run

Re: KafkaSpout

Are you running this using storm LocalCluster and you want to use external zookeeper for LocalCluster?. The latest code has those changes you can pass params to LocalCluster(localhost,2181). -Harsha On Mon, Sep 15, 2014, at 10:30 PM, siddharth ubale wrote: Hi harsha, Yes i did check with

Re: KafkaSpout

Hi harsha, Do you mean that o.9.2 incubating has those changes?? i am using the latest release of storm i.e 0.9.2 incubating... i am unable to pass on : public static LocalCluster lc = new LocalCluster(localhost,2181); please do let me know.. Thanks, Sidharth Ubale On Tue, Sep 16, 2014 at

KafkaSpout

Hi all, I am using a kafka spout for reading data from a producer into my storm topology. On the kafkaconfig : 1. When i set Kafka.forcefromstart = true .. it read from the beginning , however if i close my topology midway and run the topology again by setting Kafka.forcefromstart = false , it

Re: KafkaSpout

Siddharth, Did you try using SpoutConfig spoutConfig.forceFromStart = false spoutConfig.startOffsetTime = kafka.api.OffsetRequest.LatestTime() There are some good examples here on integrating storm and kafka https://github.com/miguno/kafka-storm-starter

Re: KafkaSpout

Hi harsha, Yes i did check with the above mentioned spoutconfig setup I think i need to bypass the in memory zookeeper which i am unable to do in local mode of storm. Any thoughts on that? Thanks, Siddharth ubale