Kafka 0.8.2 Consumer

In the following bit of code that I got from https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example /** * gives us the leader partition metadata for the given * host and port! there can be only one broker that is * the leader for the given partitionId */ val partitionM

How to retrieve the HighWaterMark

Hi all, I'm looking for a way to retrieve the HighWaterMark using the new API. Is that possible ? Thank you in advance -- Florian HUSSONNOIS

Re: How to use RPC mechanism in Kafka?

Several answers in this thread seem to indicate Kafka can not be used to simulate RPC. In Joe Stein's Real-Time Distributed and Reactive Systems presentation, RPC style calls look like they can be part of a system built using Kafka. Reference : https://youtu.be/wRzL-0VSpTY?t=5m19s The video

RE: WELCOME

drop attached pom.xml into >kafka/connect/runtime mvn compile mvn package you *should* package runtime-0.9.0.0.jar artifact in >kafka/connect/runtime/target folder Let me know if you have any problems Best Regards Martin Gainty __ > From: mgai...@

help to unsubscribe

I am sorry to bother anyone/everyone but I have been trying for some time to be unsubscribed from this list. I have tried several addresses including users-unsubscr...@kafka.apache.org but I remain unsuccessful. Can anyone point me in the right direction? Again, my apologies for the intrusion.

Re: Log message key content into LOG4j

Hi Robert, I'm not sure Kafka is a good usecase storing the logs for that long. However, it is a good candidate for an intermediate storage before you store your logs somewhere else (S3 or other blob storage). /J On Mon, Feb 8, 2016 at 11:53 AM, Robert Drygała wrote: > Hi > > Is it possibile t

Kafka 0.8.2 ConsumerGroup Example

I'm following the ConsumerGroup example, https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example How can I specify the batch size of the messages that I want to consume? I see that if I use the SimpleConsumer, I can specify a size that I want to read. How can I do it here with th

Re: Kafka 0.8.2 ConsumerGroup Example

How you do this in the high level consumer is slightly different. First, set the auto offset commit property to false. Then you can keep reading messages from kafka and when you're done filling up your buffer, commit offsets manually. The following link gives an example: http://ingest.tips/2014/10

Connection to kafka stalls

Hi, I'm having trouble with some recurring stalling connections to kafka. What I see as a symptom is that some consumers lag behind and most times restarting the consumer doesn't help. (occasionally when some other consumer tries to take the problematic partition it no longer fails, but mostly eve

Re: Connection to kafka stalls

Sorry, I'm using kafka 0.8.2 and a ConsumerGroup similar to what it is in the documentation. 2016-02-10 14:39 GMT-03:00 Maximiliano Patricio Méndez : > Hi, > > I'm having trouble with some recurring stalling connections to kafka. What > I see as a symptom is that some consumers lag behind and mos

Kafka Streams summary, overview, etc?

Hi all! I’m working on a project that would really benefit from Kafka Streams, so I’m avidly following its development and eagerly awaiting its release and I’m considering how to approach my project such that we could migrate to Kafka Streams as soon as it’s ready. In looking into this, I had a l

Re: Kafka Streams summary, overview, etc?

Avi, I have updated the corresponding wiki pages for the release plan of Kafka Streams. Hope it helps for you. Guozhang On Thu, Feb 11, 2016 at 4:08 AM, Avi Flax wrote: > Hi all! > > I’m working on a project that would really benefit from Kafka Streams, > so I’m avidly following its developmen

Kafka 0.8.2 Consumer Poll Mechanism

I tried to mimic the poll method that we have with the new consumer API in the 0.9.0.0 version. Here is what I have: def readFromKafka() = { val streams = consumerStreamsMap.get(consumerConfig.topic) @tailrec def poll(pollConfig: PollConfig, messages: Seq[String]): Seq[String] = { val i

Re: Kafka 0.8.2 Consumer Poll Mechanism

So basically what I'm doing is the following: 1. I'm checking if the time to read the stream has lapsed. If yes, then I come out of the recursion. 2. If not, I read the stream by getting the next element in the iterator What will be the offset that will be committed when I do consumer.commitOffs

Re: Kafka Streams summary, overview, etc?

On Wed, Feb 10, 2016 at 5:19 PM, Guozhang Wang wrote: > I have updated the corresponding wiki pages for the release plan of Kafka > Streams. Hope it helps for you. Thanks, that is helpful!

What is the best way to write Kafka data into HDFS?

Hello All, New Kafka user here. What is the best way to write Kafka data into HDFS? I have looked into following options and found that Flume is quickest and easiest to setup. 1. Flume 2. KaBoom 3. Kafka Hadoop Loader 4. Camus -> Gobblin Although Flume can result into small file problems when

Re: What is the best way to write Kafka data into HDFS?

R P, happy to walk you through https://github.com/DemandCube/Scribengin if your interested On Wed, Feb 10, 2016 at 5:09 PM, R P wrote: > Hello All, > New Kafka user here. What is the best way to write Kafka data into HDFS? > I have looked into following options and found that Flume is quickest

Re: What is the best way to write Kafka data into HDFS?

If you're looking for a lightweight solution with a friendly GUI (and fully open source) check out streamsets.com . It supports writing messages to a parameterized d

Compacted topic earliest offset request

Greetings, I have a question regarding the earliest offset request for compacted topic. I wanted to confirm what the expected behavior is. I am using Kafka 0.8.2. I have a topic where the compaction thread has run. When I request the earliest offset like so: bin/kafka-run-class.sh kafka.tools.G

Re: Re: [kafka-clients] java clients problem

Iman, The problem may or may not be with serialization -- it looks like it can't even find the SerializationException class, which is in the same jar as every other Kafka clients class. You probably just need to adjust your CLASSPATH to make sure the kafka-clients jar is available at runtime. -Ew

RE: Building the 0.9.0 branch

Thanks Ismael - I've got a good build installed now. John -Original Message- From: isma...@gmail.com [mailto:isma...@gmail.com] On Behalf Of Ismael Juma Sent: Tuesday, February 09, 2016 6:00 PM To: users@kafka.apache.org Subject: Re: Building the 0.9.0 branch Hi John, The core jar can b

Re: Kafka 0.8.2 Consumer Poll Mechanism

any takers for my question? I will anyways try it out today! On Wed, Feb 10, 2016 at 11:32 PM, Joe San wrote: > So basically what I'm doing is the following: > > 1. I'm checking if the time to read the stream has lapsed. If yes, then I > come out of the recursion. > 2. If not, I read the stream