Hi David. I'm wondering if it might be a good idea to have the broker send information about the last block it successfully received when it requests a new block. As the RPC stands right now it can't be idempotent -- it just tells the controller "provide me a new block, please". One case where it might be useful for the RPC to be idempotent is if the broker never receives the response from the controller such that it asks again. That would result in the burning of the block that the controller provided but that the broker never received. Now, granted, the ID space is 64 bits, so we would have to make ~2^54 requests to burn the entire space, and that isn't going to happen. So whether this is actually needed is questionable. And it might not be worth it to write the controller side code to make it act idempotently even if we added the request field to make it possible. But I figured this is worth mentioning even if we explicitly decide to reject it.
Ron On Thu, Apr 8, 2021 at 3:16 PM Ron Dagostino <rndg...@gmail.com> wrote: > > Oh, I see. Yes, my mistake -- I read it wrong. You are right that > all we need in the metadata log is the latest value allocated. > > Ron > > On Thu, Apr 8, 2021 at 11:21 AM David Arthur <mum...@gmail.com> wrote: > > > > Ron -- I considered making the RPC response and record use the same (or > > very similar) fields, but the use case is slightly different. A broker > > handling the RPC needs to know the bounds of the block since it has no idea > > what the block size is. Also, the brokers will normally see non-contiguous > > blocks. > > > > For the metadata log, we can just keep track of the latest producer Id that > > was allocated. It's kind of like a high watermark for producer IDs. This > > actually saves us from needing an extra field in the record (the KIP has > > just ProducerIdEnd => int64 in the record). > > > > Does that make sense? > > > > On Wed, Apr 7, 2021 at 8:44 AM Ron Dagostino <rndg...@gmail.com> wrote: > > > > > Thanks for the KIP, David. > > > > > > With the RPC returning a start and length, should the record in the > > > metadata log do the same thing for consistency and to save the byte > > > per record? > > > > > > Ron > > > > > > > > > On Tue, Apr 6, 2021 at 11:06 PM Ismael Juma <ism...@juma.me.uk> wrote: > > > > > > > > Great, thanks. Instead of calling it "bridge release", can we say 3.0? > > > > > > > > Ismael > > > > > > > > On Tue, Apr 6, 2021 at 7:48 PM David Arthur <mum...@gmail.com> wrote: > > > > > > > > > Thanks for the feedback, Ismael. Renaming the RPC and using start+len > > > > > instead of start+end sounds fine. > > > > > > > > > > And yes, the controller will allocate the IDs in ZK mode for the > > > > > bridge > > > > > release. > > > > > > > > > > I'll update the KIP to reflect these points. > > > > > > > > > > Thanks! > > > > > > > > > > On Tue, Apr 6, 2021 at 7:30 PM Ismael Juma <ism...@juma.me.uk> wrote: > > > > > > > > > > > Sorry, one more question: the allocation of ids will be done by the > > > > > > controller even in ZK mode, right? > > > > > > > > > > > > Ismael > > > > > > > > > > > > On Tue, Apr 6, 2021 at 4:26 PM Ismael Juma <ism...@juma.me.uk> > > > wrote: > > > > > > > > > > > > > One additional comment: if you return the number of ids instead of > > > the > > > > > > end > > > > > > > range, you can use an int32. > > > > > > > > > > > > > > Ismael > > > > > > > > > > > > > > On Tue, Apr 6, 2021 at 4:25 PM Ismael Juma <ism...@juma.me.uk> > > > wrote: > > > > > > > > > > > > > >> Thanks for the KIP, David. Any reason not to rename > > > > > > >> AllocateProducerIdBlockRequest to AllocateProducerIdsRequest? > > > > > > >> > > > > > > >> Ismael > > > > > > >> > > > > > > >> On Tue, Apr 6, 2021 at 3:51 PM David Arthur <mum...@gmail.com> > > > wrote: > > > > > > >> > > > > > > >>> Hello everyone, > > > > > > >>> > > > > > > >>> I'd like to start the discussion for KIP-730 > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-730%3A+Producer+ID+generation+in+KRaft+mode > > > > > > >>> > > > > > > >>> This KIP proposes a new RPC for generating blocks of IDs for > > > > > > >>> transactional > > > > > > >>> and idempotent producers. > > > > > > >>> > > > > > > >>> Cheers, > > > > > > >>> David Arthur > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > -- > > > > > David Arthur > > > > > > > > > > > > > > -- > > David Arthur
