Good concept. I have a question though. Say there are two producers A and B. Both producers are producing to same partition. - A sends a message with expected offset, x1 - Broker accepts is and sends an Ack - B sends a message with expected offset, x1 - Broker rejects it, sends nack - B sends message again with expected offset, x1+1 - Broker accepts it and sends Ack I guess this is what this KIP suggests, right? If yes, then how does this ensure that same message will not be written twice when two producers are producing to same partition? Producer on receiving a nack will try again with next offset and will keep doing so till the message is accepted. Am I missing something?
Also, you have mentioned on KIP, "it imposes little to no runtime cost in memory or time", I think that is not true for time. With this approach producers' performance will reduce proportionally to number of producers writing to same partition. Please correct me if I am missing out something. On Fri, Jul 17, 2015 at 11:32 AM, Mayuresh Gharat < gharatmayures...@gmail.com> wrote: > If we have 2 producers producing to a partition, they can be out of order, > then how does one producer know what offset to expect as it does not > interact with other producer? > > Can you give an example flow that explains how it works with single > producer and with multiple producers? > > > Thanks, > > Mayuresh > > On Fri, Jul 17, 2015 at 10:28 AM, Flavio Junqueira < > fpjunque...@yahoo.com.invalid> wrote: > > > I like this feature, it reminds me of conditional updates in zookeeper. > > I'm not sure if it'd be best to have some mechanism for fencing rather > than > > a conditional write like you're proposing. The reason I'm saying this is > > that the conditional write applies to requests individually, while it > > sounds like you want to make sure that there is a single client writing > so > > over multiple requests. > > > > -Flavio > > > > > On 17 Jul 2015, at 07:30, Ben Kirwin <b...@kirw.in> wrote: > > > > > > Hi there, > > > > > > I just added a KIP for a 'conditional publish' operation: a simple > > > CAS-like mechanism for the Kafka producer. The wiki page is here: > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-27+-+Conditional+Publish > > > > > > And there's some previous discussion on the ticket and the users list: > > > > > > https://issues.apache.org/jira/browse/KAFKA-2260 > > > > > > > > > https://mail-archives.apache.org/mod_mbox/kafka-users/201506.mbox/%3CCAAeOB6ccyAA13YNPqVQv2o-mT5r=c9v7a+55sf2wp93qg7+...@mail.gmail.com%3E > > > > > > As always, comments and suggestions are very welcome. > > > > > > Thanks, > > > Ben > > > > > > > -- > -Regards, > Mayuresh R. Gharat > (862) 250-7125 > -- Regards, Ashish
