Hey Edo & Mickael, > The flag is needed to distinguish a batch with a desired base offset of 0, from a regular batch for which offsets need to be generated. If the producer can provide offsets, why not provide a base offset of 0?
> (I am reading your post thinking about partitions rather than topics). Yes, I meant partitions. Sorry about that. Thanks for answering my questions :) Best, Stanislav On Thu, Nov 22, 2018 at 5:28 PM Edoardo Comar <eco...@uk.ibm.com> wrote: > Hi Stanislav, > > you're right we envision the replicator use case to have a single producer > with offsets per partition (I am reading your post thinking about > partitions rather than topics). > > If a regular producer was to send its own records at the same time, it's > very likely that the one sending with an offset will fail because of > invalid offsets. > Same if two producers were sending with offsets, likely both would then > fail. > > > Does it make sense to *lock* the topic from other producers while there > is > > one that uses offsets? > > You could do that with ACL permissions if you wanted, I don't think it > needs to be mandated by changing the broker logic. > > > > Since we are tying the produce-with-offset request to the ACL, do we > need > > the `use_offset` field in the produce request? Maybe we make it > mandatory > > for produce requests with that ACL to have offsets. > > The flag is needed to distinguish a batch with a desired base offset of 0, > from a regular batch for which offsets need to be generated. > I would not restrict a principal to only send-with-offsets (by making that > mandatory via the ACL). > > Thanks > Edo & Mickael > > -------------------------------------------------- > > Edoardo Comar > > IBM Event Streams > IBM UK Ltd, Hursley Park, SO21 2JN > > > Stanislav Kozlovski <stanis...@confluent.io> wrote on 22/11/2018 16:17:11: > > > From: Stanislav Kozlovski <stanis...@confluent.io> > > To: dev@kafka.apache.org > > Date: 22/11/2018 16:17 > > Subject: Re: [DISCUSS] KIP-391: Allow Producing with Offsets for > > Cluster Replication > > > > Hey Edurdo, thanks for the KIP! > > > > I have some questions, apologies if they are naive: > > Is this intended to work for a single producer use case only? > > How would it work if two producers were producing to the same topic with > > offsets? > > How would it work if two producers, one with offsets and one without > were > > producing to a topic? > > Does it make sense to *lock* the topic from other producers while there > is > > one that uses offsets? > > > > Since we are tying the produce-with-offset request to the ACL, do we > need > > the `use_offset` field in the produce request? Maybe we make it > mandatory > > for produce requests with that ACL to have offsets. > > > > Best, > > Stanislav > > > > On Wed, Nov 21, 2018 at 5:14 PM Edoardo Comar <eco...@uk.ibm.com> wrote: > > > > > Hi, > > > we've opened a KIP to improve data replication between Kafka clusters > : > > > > > > > > > INVALID URI REMOVED > > > > u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D391-253A-2BAllow-2BProducing-2Bwith-2BOffsets-2Bfor-2BCluster-2BReplication&d=DwIBaQ&c=jf_iaSHvJObTbx- > > > siA1ZOg&r=EzRhmSah4IHsUZVekRUIINhltZK7U0OaeRo7hgW4_tQ&m=uUj9C3BdbYz0dDNA- > > > E6iXreg1M5hWiWgG6ClS86VIPI&s=Vav8_-N7_OpfYEW33yGOf_or8ESMUJ4S45t2g-EUWKg&e= > > > > > > We'd like to start a discussion, please post your feedback in this > thread. > > > > > > Thank you > > > Edo and Mickael > > > > > > > > > -------------------------------------------------- > > > > > > Edoardo Comar > > > > > > IBM Event Streams > > > IBM UK Ltd, Hursley Park, SO21 2JN > > > > > > Unless stated otherwise above: > > > IBM United Kingdom Limited - Registered in England and Wales with > number > > > 741598. > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 > 3AU > > > > > > > > > -- > > Best, > > Stanislav > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > -- Best, Stanislav
