Guozhang,My coworker came up with an explaination: at one moment the leader L,
and two followers F1, F2 are all in ISR. The producer sends a message m1 and
receives acks from L and F1. Before the messge is replicated to F2, L is down.
In the following leader election, F2, instead of F1, becomes the leader, and
loses m1 somehow.
Could that be the root cause?
Thanks,
Jiang
From: users@kafka.apache.org At: Jul 15 2014 15:05:25
To: users@kafka.apache.org
Subject: Re: message loss for sync producer, acks=2, topic replicas=3
Guozhang,
Please find the config below:
Producer:
props.put("producer.type", "sync");
props.put("request.required.acks", 2);
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("partitioner.class", "kafka.producer.DefaultPartitioner");
props.put("message.send.max.retries", "60");
props.put("retry.backoff.ms", "300");
Consumer:
props.put("zookeeper.session.timeout.ms", "400");
props.put("zookeeper.sync.time.ms", "200");
props.put("auto.commit.interval.ms", "1000");
Broker:
num.network.threads=2
num.io.threads=8
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600
num.partitions=2
log.retention.hours=168
log.retention.bytes=20000000
log.segment.bytes=536870912
log.retention.check.interval.ms=60000
log.cleaner.enable=false
zookeeper.connection.timeout.ms=1000000
Topic:
Topic:p1r3 PartitionCount:1 ReplicationFactor:3
Configs:retention.bytes=10000000000
Thanks,
Jiang
From: users@kafka.apache.org At: Jul 15 2014 13:59:03
To: JIANG WU (PRICEHISTORY) (BLOOMBERG/ 731 LEX -), users@kafka.apache.org
Subject: Re: message loss for sync producer, acks=2, topic replicas=3
What config property values did you use on producer/consumer/broker?
Guozhang
On Tue, Jul 15, 2014 at 10:32 AM, Jiang Wu (Pricehistory) (BLOOMBERG/ 731
LEX -) <jwu...@bloomberg.net> wrote:
> Guozhang,
> I'm testing on 0.8.1.1; just kill pid, no -9.
> Regards,
> Jiang
>
> From: users@kafka.apache.org At: Jul 15 2014 13:27:50
> To: JIANG WU (PRICEHISTORY) (BLOOMBERG/ 731 LEX -), users@kafka.apache.org
> Subject: Re: message loss for sync producer, acks=2, topic replicas=3
>
> Hello Jiang,
>
> Which version of Kafka are you using, and did you kill the broker with -9?
>
> Guozhang
>
>
> On Tue, Jul 15, 2014 at 9:23 AM, Jiang Wu (Pricehistory) (BLOOMBERG/ 731
> LEX -) <jwu...@bloomberg.net> wrote:
>
> > Hi,
> > I observed some unexpected message loss in kafka fault tolerant test. In
> > the test, a topic with 3 replicas is created. A sync producer with acks=2
> > publishes to the topic. A consumer consumes from the topic and tracks
> > message ids. During the test, the leader is killed. Both producer and
> > consumer continue to run for a while. After the producer stops, the
> > consumer reports if all messages are received.
> >
> > The test was repeated multiple rounds; message loss happened in about 10%
> > of the tests. A typical scenario is as follows: before the leader is
> > killed, all 3 replicas are in ISR. After the leader is killed, one
> follower
> > becomes the leader, and 2 replicas (including the new leader) are in ISR.
> > Both the producer and consumer pause for several seconds during that
> time,
> > and then continue. Message loss happens after the leader is killed.
> >
> > Because the new leader is in ISR before the old leader is killed, unclean
> > leader election doesn't explain the message loss.
> >
> > I'm wondering if anyone else also observed such message loss? Is there
> any
> > known issue that may cause the message loss in the above scenario?
> >
> > Thanks,
> > Jiang
>
>
> --
> -- Guozhang
>
>
>
--
-- Guozhang
