On Tue, Nov 12, 2019, at 17:42, Guozhang Wang wrote: > Sounds good to me, let's continue our voting process here. > > Guozhang
Hi Brian & Guozhang, I'm looking forward to getting this in. But is the KIP complete yet? Brian's earlier response said " I'm going to leave out some specifics in my response, since I'll go back to the KIP, revise it, and then post an update..." It sounds like we should wait for those updates (if those updates have already happened, then let us know so we can recheck the wiki page.) > > On Tue, Nov 12, 2019 at 12:10 PM Ismael Juma <isma...@gmail.com> wrote: > > > This is not a bug fix, in my opinion. The existing behavior may be > > confusing, but it was documented, so I assume it was intended. > > Yeah, I agree. I think it will be nice to have this behavior documented in a KIP. And there are user-visible behavior changes here, like removing blocking (a very welcome change!) best, Colin > > Ismael > > > > On Mon, Nov 11, 2019, 2:47 PM Guozhang Wang <wangg...@gmail.com> wrote: > > > > > Thanks for the update Brian, I think I agree with Colin that we should > > > clarify on whether / how the blocking behavior due to metadata fetch > > would > > > be affected or not. > > > > > > About whether this needs to be voted as a KIP: to me the behavior change > > is > > > really a bug fix rather than a public contract change, since we are not > > > incorrectly blocking a non-blocking declared function, but if people feel > > > that we should be more careful about such changes I'm also fine with > > voting > > > it as an official KIP too. > > > > > > > > > Guozhang > > > > > > On Sat, Nov 9, 2019 at 12:50 AM Brian Byrne <bby...@confluent.io> wrote: > > > > > > > Hi Guozhang, > > > > > > > > Regarding metadata expiry, no access times other than the initial > > > lookup[1] > > > > are used for determining when to expire producer metadata. Therefore, > > > > frequently used topics' metadata will be aged out and subsequently > > > > refreshed (in a blocking manner) every five minutes, and infrequently > > > used > > > > topics will be retained for a minimum of five minutes and currently > > > > refetched on every metadata update during that time period. The > > sentence > > > is > > > > suggesting that we could reduce the expiry time to improve the latter > > > > without affecting (rather slightly improving) the former. > > > > > > > > Keep in mind that in most all cases, I wouldn't anticipate much of a > > > > difference with producer behavior, and the extra logic can be > > implemented > > > > to have insignificant cost. It's the large/dynamic topic corner cases > > > that > > > > we're trying to improve. > > > > > > > > It'd be convenient if the KIP is no longer necessary. You're right in > > > that > > > > there's no public API changes and the behavioral changes are entirely > > > > internal. I'd be happy to continue the discussion around the KIP, but > > > > unless otherwise objected, it can be retired. > > > > > > > > [1] Not entirely accurate, it's actually the first time when the client > > > > calculates whether to retain the topic in its metadata. > > > > > > > > Thanks, > > > > Brian > > > > > > > > On Thu, Nov 7, 2019 at 4:48 PM Guozhang Wang <wangg...@gmail.com> > > wrote: > > > > > > > > > Hello Brian, > > > > > > > > > > Could you elaborate a bit more on this sentence: "This logic can be > > > made > > > > > more intelligent by managing the expiry from when the topic was last > > > > used, > > > > > enabling the expiry duration to be reduced to improve cases where a > > > large > > > > > number of topics are touched intermittently." Not sure I fully > > > understand > > > > > the proposal. > > > > > > > > > > Also since now this KIP did not make any public API changes and the > > > > > behavioral changes are not considered a public API contract (i.e. how > > > we > > > > > maintain the topic metadata in producer cache is never committed to > > > > users), > > > > > I wonder if we still need a KIP for the proposed change any more? > > > > > > > > > > > > > > > Guozhang > > > > > > > > > > On Thu, Nov 7, 2019 at 12:43 PM Brian Byrne <bby...@confluent.io> > > > wrote: > > > > > > > > > > > Hello all, > > > > > > > > > > > > I'd like to propose a vote for a producer change to improve > > producer > > > > > > behavior when dealing with a large number of topics, in part by > > > > reducing > > > > > > the amount of metadata fetching performed. > > > > > > > > > > > > The full KIP is provided here: > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-526%3A+Reduce+Producer+Metadata+Lookups+for+Large+Number+of+Topics > > > > > > > > > > > > And the discussion thread: > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/b2f8f830ef04587144cf0840c7d4811bbf0a14f3c459723dbc5acf9e@%3Cdev.kafka.apache.org%3E > > > > > > > > > > > > Thanks, > > > > > > Brian > > > > > > > > > > > > > > > > > > > > > -- > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > -- > -- Guozhang >
