Did you check if the controller is active in the cluster? If the controller isn't active (there are known 0.8 bugs that can lead to this), then this could cause this kind of data loss issue. I recommend upgrading to 0.9 ASAP.
Thanks Tom Crayford Heroku Kafka On Friday, 17 June 2016, Gulia, Vikram <vikram.gu...@dish.com> wrote: > Hi Gerard, thanks for the reply. Few follow ups - > > 1. I can try setting acks = all but wouldn't it lead to performance hit (I > am using sync produce thus response time will be more). > 2. I will try unclean.leader.election.enable = false and update you if it > helps. > 3. Regarding your last point, I am confused. What I understood about kafka > is that the producer client always retrieve the topic metadata and already > knows who the leader for the topic is. And the producer client always > sends the message to the leader only (the replicas replicate the message > and send acknowledgements to the leader). Are you saying the producer > client can send message to any broker who is not leader or to two or more > brokers one of which may or may not be leader? > > Thank you, > Vikram Gulia > > > > > On 6/17/16, 12:29 AM, "Gerard Klijs" <gerard.kl...@dizzit.com > <javascript:;>> wrote: > > >You could try set the acks to -1, so you wait for the produce to be > >succesfull, until most other brokers also received the message. Another > >thing you could try is set the unclean.leader.election.enable to false > >(this is a setting on the broker). > >I think what's happening now is that the message in your example is send > >to > >two different brokers, because one of them is not sending the record to > >the > >actual leader. Since you have set your acks to one, you wont see any error > >in the producer, cause it succeeded in sending it to the broker. You most > >likely will see some error on the broker, because it is not the leader. > > > >On Fri, Jun 17, 2016 at 5:19 AM Gulia, Vikram <vikram.gu...@dish.com > <javascript:;>> > >wrote: > > > >> Hi Users, I am facing message loss while using kafka v 0.8.2.2. Please > >>see > >> details below and help me if you can. > >> > >> Issue: 2 messages produced to same partition one by one Kafka producer > >> returns same offset back which means message produced earlier is lost.< > >> > >> > http://stackoverflow.com/questions/37732088/2-messages-produced-to-same-p > >>artition-one-by-one-message-1-overridden-by-next > >> > > >> > >> Details: > >> I have a unique problem which is happening like 50-100 times a day with > >> message volume of more than 2 millions per day on the topic.I am using > >> Kafka producer API 0.8.2.2 and I have 12 brokers (v 0.8.2.2) running in > >> prod with replication of 4. I have a topic with 60 partitions and I am > >> calculating partition for all my messages and providing the value in the > >> ProducerRecord itself. Now, the issue - > >> > >> Application creates 'ProducerRecord' using - > >> > >> new ProducerRecord<String, String>(topic, 30, null, message1); > >> providing topic, value message1 and partition 30. Then application call > >> the send method and future is returned - > >> > >> // null is for callback > >> Future<RecordMetadata> future = producer.send(producerRecord. null); > >> Now, app prints the offset and partition value by calling get on Future > >> and then getting values from RecordMetadata - this is what i get - > >> > >> Kafka Response : partition 30, offset 3416092 > >> Now, the app produce the next message - message2 to same partition - > >> > >> new ProducerRecord<String, String>(topic, 30, null, message2); > >> and kafka response - > >> > >> Kafka Response : partition 30, offset 3416092 > >> I received the same offset again, and if I pull message from the offset > >>of > >> partition 30 using simple consumer, it ends up being the message2 which > >> essentially mean i lost the message1. > >> > >> Currently, the messages are produced using 10 threads each having its > >>own > >> instance of kafka producer (Earlier threads shared 1 Kafka producer but > >>it > >> was performing slow and we still had message loss). > >> I am using all default properties for producer except a few mentioned > >> below, the message (String payload) size can be a few kbs to a 500 kbs. > >>I > >> am using acks value of 1. > >> > >> value.serializer: org.apache.kafka.common.serialization.StringSerializer > >> key.serializer: org.apache.kafka.common.serialization.StringSerializer > >> bootstrap.servers: {SERVER VIP ENDPOINT} > >> acks: 1 > >> batch.size: 204800 > >> linger.ms: 10 > >> send.buffer.bytes: 1048576 > >> max.request.size: 10000000 > >> > >> What am i doing wrong here? Is there something I can look into or any > >> producer property or server property I can tweak to make sure i don't > >>lose > >> any messages. I need some help here soon as I am losing some critical > >> messages in production which is not good at all because as there is no > >> exception given by Kafka Producer its even hard to find out the message > >> lost unless downstream process reports it. > >> > >> Thank you, > >> Vikram Gulia > >> > >
