Guozhang, I'm testing on 0.8.1.1; just kill pid, no -9. Regards, Jiang From: users@kafka.apache.org At: Jul 15 2014 13:27:50 To: JIANG WU (PRICEHISTORY) (BLOOMBERG/ 731 LEX -), users@kafka.apache.org Subject: Re: message loss for sync producer, acks=2, topic replicas=3
Hello Jiang, Which version of Kafka are you using, and did you kill the broker with -9? Guozhang On Tue, Jul 15, 2014 at 9:23 AM, Jiang Wu (Pricehistory) (BLOOMBERG/ 731 LEX -) <jwu...@bloomberg.net> wrote: > Hi, > I observed some unexpected message loss in kafka fault tolerant test. In > the test, a topic with 3 replicas is created. A sync producer with acks=2 > publishes to the topic. A consumer consumes from the topic and tracks > message ids. During the test, the leader is killed. Both producer and > consumer continue to run for a while. After the producer stops, the > consumer reports if all messages are received. > > The test was repeated multiple rounds; message loss happened in about 10% > of the tests. A typical scenario is as follows: before the leader is > killed, all 3 replicas are in ISR. After the leader is killed, one follower > becomes the leader, and 2 replicas (including the new leader) are in ISR. > Both the producer and consumer pause for several seconds during that time, > and then continue. Message loss happens after the leader is killed. > > Because the new leader is in ISR before the old leader is killed, unclean > leader election doesn't explain the message loss. > > I'm wondering if anyone else also observed such message loss? Is there any > known issue that may cause the message loss in the above scenario? > > Thanks, > Jiang -- -- Guozhang
