delay of producer and consumer in kafka 0.9 is too big to be accepted

2016-06-25 Thread Kafka
Hi all, my kafka cluster is composed of three brokers with each have 8core cpu and 8g memory and 1g network card. with java async client,I sent 100 messages with size of 1024 bytes per message ,the send gap between each sending is 20us,the consumer’s config is like this,fetch

Re: delay of producer and consumer in kafka 0.9 is too big to be accepted

2016-06-25 Thread Jay Kreps
Can you sanity check this with the end-to-end latency test that ships with Kafka in the tools package? https://apache.googlesource.com/kafka/+/1769642bb779921267bd57d3d338591dbdf33842/core/src/main/scala/kafka/tools/TestEndToEndLatency.scala On Saturday, June 25, 2016, Kafka wrote: > Hi all, >

Re: Kafka producer metadata issue

2016-06-25 Thread Shekar Tippur
Any updates on this issue please? As suggested, I tried adding a microsleep between consecutive producer.send () And random messages got into the topic.(not all messages got into the topic) I am blocked on this issue. Appreciate if someone could help me out on this. Sent from my iPhone > On J

Halting because log truncation is not allowed for topic __consumer_offsets

2016-06-25 Thread Morellato, Wanny
Hi all, My kafka brokers (0.9.0.1) are refusing to restart and they return the following error Halting because log truncation is not allowed for topic __consumer_offsets, Current leader 11's latest offset 79445540 is less than replica 13's latest offset 79445565 (kafka.server.ReplicaFetcherThr

Deploying new connector to existing Kafka cluster

2016-06-25 Thread Dean Arnold
I'm looking for a comprehensive example for deploying a new connector plugin into an existing Kafka cluster. Is there any standard solution for distributing a connector jar across nodes, and then starting the connector ? Or is it a manual copy process (e.g., via pdcp), and then run the Connect REST

Re: Kafka producer metadata issue

2016-06-25 Thread Shekar Tippur
Another observation .. The below code produces. Cant understand this randomness :/ 2 xyz 3 xyz 3 xyz 3 xyz 3 xyz 4 xyz 4 xyz 4 xyz 4 xyz for(int i = 0; i < 5; i++) { producer.send(new ProducerRecord("test", Integer.toString(i), "xyz")); producer.send(new ProducerRecord(