Hi Becket, Thanks for your comments. I added a 'Background' section to the KIP which defines the terms we are using. It also has a note on replication and delivery guarantees. Please have a look:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-185%3A+Make+exactly+once+in+order+delivery+per+partition+the+default+producer+setting If you accept those definitions, it would be helpful to clarify the precise scenarios where min.isr=2 and replication-factor=3 are necessary conditions for stronger guarantees like at-least once or exactly-once delivery. I do accept that replication-factor >=2 is a necessary condition for at least once and exactly once delivery, and I explain why in that wiki. However, I am not proposing that we change the default replication factor in this KIP. Regarding the max.in.flight discussion, I am sure there are many ways to optimize for a very high number of max.in.flight and yet retain idempotence and ordering. Are you saying that we should enable practically infinite max.in.flight with idempotence and ordering if we are to make idempotence the default setting? If not, I think discussions around optimizing this aspect of the system should be part of a separate thread. Note that before the KIP-98 and KAFKA-5494 changes, you got no ordering guarantees with max.in.flight > 1. So the new proposed cap of max.in.flight=5 with ordering and idempotence is already a major improvement on the status quo. Thanks, Apurva On Mon, Aug 14, 2017 at 7:09 PM, Becket Qin <becket....@gmail.com> wrote: > Hi Apurva, > > Regarding acks. I think acks=all is not an isolated independent > configuration for durability. The reason we want to enable acks=all is to > tolerate broker failures. But If that is the case, not setting min.isr to > >=2 seems defeating that purpose. If we set min.isr=2, setting replication > factor to 3 would be required to provide availability in case of one broker > failure. > > Regarding max.in.flight.requests.per.connection. I am thinking that one > way > to avoid excessive buffering on the broker side is to let the > ProduceRequest include a per partition last received acked sequence. This > way the broker just need to keep the sequence/offset/timestamp whose > sequence is above the received last acked sequence and less than or equals > to the last appended sequence. This should limit the amount of the sequence > we have to keep on the broker side. And I agree with Guozhang that we > should limit the total memory of the sequence buffered on the broker. In > the worst case, falling back to do a disk search may not be that bad > assuming people are not doing insane things. > > Thanks, > > Jiangjie (Becket) Qin > > > > On Mon, Aug 14, 2017 at 1:36 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > > Just want to clarify that regarding 1), I'm fine with changing it to > `all` > > but just wanted to argue it is not necessarily correlate with the > > exactly-once semantics, but rather on persistence v.s. availability > > trade-offs, so I'd like to discuss them separately. > > > > Regarding 2), one minor concern I had is that the enforcement is on the > > client side while the parts it affects is on the broker side. I.e. the > > broker code would assume at most 5 in.flight when idempotent is turned > on, > > but this is not enforced at the broker but relying at the client side's > > sanity. So other implementations of the client that may not obey this may > > likely break the broker code. If we do enforce this we'd better enforce > it > > at the broker side. Also, I'm wondering if we have considered the > approach > > for brokers to read the logs in order to get the starting offset when it > > does not about it in its snapshot, that whether it is worthwhile if we > > assume that such issues are very rare to happen? > > > > > > Guozhang > > > > > > > > On Mon, Aug 14, 2017 at 11:01 AM, Apurva Mehta <apu...@confluent.io> > > wrote: > > > > > Hello, > > > > > > I just want to summarize where we are in this discussion > > > > > > There are two major points of contention: should we have acks=1 or > > acsk=all > > > by default? and how to cap max.in.flight.requests.per.connection? > > > > > > 1) acks=1 vs acks=all1 > > > > > > Here are the tradeoffs of each: > > > > > > If you have replication-factor=N, your data is resilient N-1 to disk > > > failures. For N>1, here is the tradeoff between acks=1 and acks=all. > > > > > > With proposed defaults and acks=all, the stock Kafka producer and the > > > default broker settings would guarantee that ack'd messages would be in > > the > > > log exactly once. > > > > > > With the proposed defaults and acks=1, the stock Kafka producer and the > > > default broker settings would guarantee that 'retained ack'd messages > > would > > > be in the log exactly once. But all ack'd messages may not be > retained'. > > > > > > If you leave replication-factor=1, acks=1 and acks=all have identical > > > semantics and performance, but you are resilient to 0 disk failures. > > > > > > I think the measured cost (again the performance details are in the > wiki) > > > of acks=all is well worth the much clearer semantics. What does the > rest > > of > > > the community think? > > > > > > 2) capping max.in.flight at 5 when idempotence is enabled. > > > > > > We need to limit the max.in.flight for the broker to de-duplicate > > messages > > > properly. The limitation would only apply when idempotence is enabled. > > The > > > shared numbers show that when the client-broker latency is low, there > is > > no > > > performance gain for max.inflight > 2. > > > > > > Further, it is highly debatable that max.in.flight=500 is significantly > > > better than max.in.flight=5 for a really high latency client-broker > > link, > > > and so far there are no hard numbers one way or another. However, > > assuming > > > that max.in.flight=500 is significantly better than max.inflight=5 in > > some > > > niche use case, the user would have to sacrifice idempotence for > > > throughput. In this extreme corner case, I think it is an acceptable > > > tradeoff. > > > > > > What does the community think? > > > > > > Thanks, > > > Apurva > > > > > > > > > > > -- > > -- Guozhang > > >
