Hi Brian, This looks great, and should help reduce blocking and high metadata request volumes when the producer is sending to large numbers of topics, especially at low volumes. I think the approach to make metadata fetching asynchronous and batch metadata requests together will help significantly.
The only other approach I can think of is to allow users to supply the producer with the expected topics upfront, allowing the producer to perform a single initial metadata request before any sends occur. I see no real advantages to this approach compared to the async method you’ve proposed, but maybe we could add it to the rejected alternatives section? Thanks, Lucas On Fri, 20 Sep 2019 at 11:46, Brian Byrne <bby...@confluent.io> wrote: > I've updated the 'Proposed Changes' to include two new producer > configuration variables: topic.expiry.ms and topic.refresh.ms. Please take > a look. > > Thanks, > Brian > > On Tue, Sep 17, 2019 at 12:59 PM Brian Byrne <bby...@confluent.io> wrote: > > > Dev team, > > > > Requesting discussion for improvement to the producer when dealing with a > > large number of topics. > > > > KIP: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-526%3A+Reduce+Producer+Metadata+Lookups+for+Large+Number+of+Topics > > > > JIRA: https://issues.apache.org/jira/browse/KAFKA-8904 > > > > Thoughts and feedback would be appreciated. > > > > Thanks, > > Brian > > >
