Currently metadata.broker.list is a separate list, unrelated to the cluster, which is used for all metadata requests.
We are actually just now discussing whether this makes sense or not with respect to the new producer and consumer we are working on. I actually misunderstood how the existing producer worked so in the producer rewrite I made it work the way you describe. We are currently trying to figure out which of these is better. -Jay On Mon, Mar 17, 2014 at 1:29 PM, Ryan Berdeen <rberd...@hubspot.com> wrote: > I recently moved my 0.8.0 cluster to a set of entirely new brokers, and was > surprised to find that the producers did not update their list of brokers > to remove the brokers that were no longer in the cluster. That is, I had > brokers 1,2,3,4,5, added brokers 6,7,8,910, waited a day, and stopped > brokers 1,2,3,4,5. After stopping the original brokers, the producers > continued trying to fetch metadata from them, even though they no longer > appeared the topic metadata. It looks like it was basically the same issue > described in > > http://mail-archives.apache.org/mod_mbox/kafka-users/201402.mbox/%3ccadpdzrksmjcpr0-l9txxoegthbjtptamh_fupq3ranwytkp...@mail.gmail.com%3E > . > > The documentation for "metadata.broker.list" says "This is for > bootstrapping and the producer will only use it for getting metadata". I, > and apparently others, have interpreted the "this is for bootstrapping" > part to mean that it is only used for the initial metadata fetch, and not > as the sole list of brokers to use to fetch metadata. > > Does it make sense to change the documentation to something like "The list > of brokers used to fetch metadata. These brokers must always be available"? > > Am I understanding metadata.broker.list correctly? Is it necessary to load > balance these brokers or otherwise make sure it does not refer directly to > any brokers that could be removed? > > Thanks, > > Ryan >
