ack = 2 *will* throw an exception when there's only one node in ISR. The problem with ack=2 is that if you have 3 replicas and you got acks from 2 of them, the one replica which did not get the message can still be in ISR and get elected as leader, leading for a loss of the message. If you specify ack=3, you can't tolerate the failure of a single replica. Not amazing either.
To makes things even worse, when specifying the number of acks you want, you don't always know how many replicas the topic should have, so its difficult to pick the correct number. acks = -1 solves that problem (since all messages need to get acked by all replicas), but introduces the new problem of not getting an exception if ISR shrank to 1 replica. Thats why the min.isr configuration was added. I hope this clarifies things :) I'm planning to add this to the docs in a day or two, so let me know if there are any additional explanations or scenarios you think we need to include. Gwen On Tue, Oct 14, 2014 at 12:27 PM, Scott Reynolds <sreyno...@twilio.com> wrote: > A question about 0.8.1.1 and acks. I was under the impression that setting > acks to 2 will not throw an exception when there is only one node in ISR. > Am I incorrect ? Thus the need for min_isr. > > On Tue, Oct 14, 2014 at 11:50 AM, Kyle Banker <kyleban...@gmail.com> wrote: > >> It's quite difficult to infer from the docs the exact techniques required >> to ensure consistency and durability in Kafka. I propose that we add a doc >> section detailing these techniques. I would be happy to help with this. >> >> The basic question is this: assuming that I can afford to temporarily halt >> production to Kafka, how do I ensure that no message written to Kafka is >> ever lost under typical failure scenarios (i.e., the loss of a single >> broker)? >> >> Here's my understanding of this for Kafka v0.8.1.1: >> >> 1. Create a topic with a replication factor of 3. >> 2. Use a sync producer and set acks to 2. (Setting acks to -1 may >> successfully write even in a case where the data is written only to a >> single node). >> >> Even with these two precautions, there's always the possibility of an >> "unclean leader election." Can data loss still occur in this scenario? Is >> it possible to achieve this level of durability on v0.8.1.1? >> >> In Kafka v0.8.2, in addition to the above: >> >> 3. Ensure that the triple-replicated topic also disallows unclean leader >> election (https://issues.apache.org/jira/browse/KAFKA-1028). >> >> 4. Set the min.isr value of the producer to 2 and acks to -1 ( >> https://issues.apache.org/jira/browse/KAFKA-1555). The producer will then >> throw an exception if data can't be written to 2 out of 3 nodes. >> >> In addition to producer configuration and usage, there are also monitoring >> and operations considerations for achieving durability and consistency. As >> those are rather nuanced, it'd probably be easiest to just start iterating >> on a document to flesh those out. >> >> If anyone has any advice on how to better specify this, or how to get >> started on improving the docs, I'm happy to help out. >>
