Re: KafkaProducer Retries in .9.0.1

Is it an option to set up a cluster and kill the leader? That's the way we checked retries and at if we would not lose messages that way. The sending to Kafka goes in two parts, some serialization etc, before an attempt is made to really send the binary message, and the actual sending. I'm not sure

Re: KafkaProducer Retries in .9.0.1

Hi Thanks for the suggestion. I lowered the broker message.max.bytes to be smaller than my payload so that I now receive an org.apache.kafka.common.errors.RecordTooLargeException : I still don't see the retries happening, the default back off is 100ms, and my producer loops for a few seconds, long

Tell us how you're using Kafka and what you'd like to see in the future

Folks, We'd like to hear from community members about how you are using Kafka today, and what sort of features you'd like to see in the future. We put together this survey to gather your feedback, which should take only a few minutes to complet

Re: Low-level Consumer Example (Scala)

> I use https://github.com/akka/reactive-kafka > . This library has just been taken > over by the Akka team and will in incorporated into future version of Akka. Thanks Dave for the pointer. I downloaded the reactive-kakka .8 branch since we're using

Re: Kafka constant shrinking and expanding after deleting a topic

Guozhang, thanks for these links. Hi Alexis, as Guozhang said, yours seems different from our case. We deleted a topic but caused shrinking/expanding for other topics. Yifan On Tue, Apr 5, 2016 at 10:02 PM, Alexis Midon wrote: > I ran into the same issue today. In a production cluster, I notic

Re: Kafka constant shrinking and expanding after deleting a topic

Alexis, Hmm, yours seems a bug in Kafka brokers since your message relates to a topic that has been deleted months ago, indicating that the topic was not deleted cleanly. Could you file a JIRA with server logs for further investigation? Guozhang On Tue, Apr 5, 2016 at 10:02 PM, Alexis Midon < a

kafka issue

Hi, Seeing below exceptions on broker id 3 node, in my three node kafka cluster.Also looks like this broker id is long under replicated and out of sync. [2016-04-06 04:00:00,720] ERROR [Replica Manager on Broker 3]: Error processing fetch operation on partition [subscribed_product_logs,2] offset

Re: Kafka C/C++ client with consumer group support?

Hi Vince, librdkafka (master branch) supports >=0.9 balanced consumers groups: https://github.com/edenhill/librdkafka As long as you are using different consumer groups each group will receive the same set of messages. Example C++ consumer: https://github.com/edenhill/librdkafka/blob/master/exa

Re: Kafka C/C++ client with consumer group support?

Hi Vince, I think this might help you https://github.com/edenhill/librdkafka ( - High-level balanced KafkaConsumer: supported (requires broker >= 0.9)) Regards, Ben On Wed, Apr 6, 2016 at 3:16 PM, Vince Deters wrote: > Hello, we have a requirement to run redundant Kafka consumers that con

Kafka C/C++ client with consumer group support?

Hello, we have a requirement to run redundant Kafka consumers that consume an identical stream of messages in identical order. For this reason we plan to have these consumers consume from the same topic partition and identify themselves as unique consumer groups. Does any C/C++ Kafka client have

Does Kafka support federation of clusters?

Hi, For higher availability and more flexible operations (upgrading, rolling ops, etc.), we are considering to federate two or more kafka clusters, each with a separated zk cluster. For example, if "topic-T" has 6 total partitions, 3 on cluster A, and another 3 on cluster B, messages will be

Kafka Connect concept question

Hi folks. I'm pretty new to Kafka. I have spent a fair amount of time so far understanding the Kafka system in general and how producers and consumers work. I'm now trying to get a grasp on how Kafka Connect compares/contrasts to Producers/Consumers written via the Java API. When might someone wan

Kafka Consumer Problem

Hi, I am trying to consume message from kafka with using web service. For exapmle; when the user call that web service, the all messages from partitions will be read by consumers and web service will returns the kafka messages to user who called it. I used below code for that purpose but Consu

kafka issue

Hi, Seeing below exceptions on broker id 3 node, in my three node kafka cluster.Also looks like this broker id is long under replicated and out of sync. [2016-04-06 04:00:00,720] ERROR [Replica Manager on Broker 3]: Error processing fetch operation on partition [subscribed_product_logs,2] offs

Re: Low-level Consumer Example (Scala)

I use https://github.com/akka/reactive-kafka . This library has just been taken over by the Akka team and will in incorporated into future version of Akka. It allows for at-least-once semantics as well as at-most-once semantics. WARNING: The new API ( v

Re: Question on Java client consumer.seek()

Sounds reasonable to me. On 04/05/2016 05:56 PM, Mario Ricci wrote: > I found this > thread > and see that poll() must be called before seek(). > > This se