It looks like this problem is caused by this bug in Kafka 8, which was fixed in Kafka 9:
https://issues.apache.org/jira/browse/KAFKA-972 On Thu, Sep 8, 2016 at 3:55 PM, Jeff Widman <j...@jeffwidman.com> wrote: > How do I permanently remove a broker from a Kafka cluster? > > Scenario: > > I have a stable cluster of 3 brokers. I temporarily added a fourth broker > that successfully joined the cluster. The controller returned metadata > indicating this broker was part of the cluster. > > However, I never rebalanced partitions onto this broker, so this broker #4 > was never actually used, didn't join any replica lists, ISRs, etc. > > I later decided to remove this unused broker from the cluster. I shutdown > the broker successfully and Zookeeper /broker/ids no longer lists broker #4. > > However, when my application code connects to any Kafka broker and fetches > metadata, I get a broker list that includes this deleted broker. > > How do I indicate to the cluster that this broker has been permanently > removed from the cluster and not just a transient downtime? > > Additionally, what's happening under the covers that causes this? > > I'm guessing that when I connect to a broker and ask for metadata, the > broker checks its local cache for the controller ID, contacts the broker > and asks it for the list of all brokers. Then the controller checks it's > cached list of brokers and returns the list of all brokers known to have > belonged to the cluster at any point in time. > > I'm guessing this happens because it's not certain if the dead broker is > permanently removed or just transient downtime. So I'm thinking I just need > to indicate to the controller that it needs to reset it's list of known > cluster brokers to the known live brokers in zookeeper. But would not be > surprised if something in my mental model is incorrect. > > This is for Kafka 0.8.2. I am planning to upgrade to 0.10 in the > not-to-distant future, so if 0.10 handles this differently, I'm also > curious about that. > > > -- > > *Jeff Widman* > jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265) > <>< > -- *Jeff Widman* jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265) <><
