[ https://issues.apache.org/jira/browse/KAFKA-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088200#comment-15088200 ]
Ewen Cheslack-Postava commented on KAFKA-3068: ---------------------------------------------- [~junrao] It's definitely a real issue -- see https://issues.apache.org/jira/browse/KAFKA-1843?focusedCommentId=14517659&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14517659 where you can get into unrecoverable situations if a Kafka node is decomissioned or in the case of a failure. This doesn't necessarily seem like it would be that rare to me -- sure, it requires there to be failures, but any app that produces to a single topic could encounter a case like this since it'll have a very limited set of brokers to work with. Multiple people reported encountering this issue, although as far as I've seen it may only have been in testing environments: http://mail-archives.apache.org/mod_mbox/kafka-users/201505.mbox/%3ccaj7fv7qbfqzrd0eodjmtrq725q+ziirnctwjhyhqldm+5cw...@mail.gmail.com%3E It sounds like KAFKA-1843 is messy enough that it needs a more complete discussion to resolve and maybe a KIP. Just waiting on the old list doesn't help in the situation described above if there's an actual hardware failure that takes the node out entirely -- eventually you need to try something else. If we're going to change anything now, I'd suggest either making the entries in nodesEverSeen expire but at a longer interval than the metadata refresh or falling back to bootstrap nodes. I think the former is probably the better solution, though the latter may be easier for people to reason about until we work out a more complete solution. The former seems nicer because it decouples the expiration from other failures. Currently a broker failure that triggers a metadata refresh can force that broker out of the set of nodes to try immediately, which is why you can get into situations like the one linked above (and in much less time than the metadata refresh interval since these events force a metadata refresh). Ultimately, the problem boils down to the fact that we're not getting the information we want from these metadata updates -- we get the info we need about topic leaders, but we're also using this metadata for basic connectivity to the cluster. That breaks down if you're only working with one or a small number of topics and they end up with too few or crashed replicas. I think caching old entries is not the ideal solution here, but it's a way to work around the current situation. It would probably be better if metadata responses could include a random subset of the current set of brokers so that rather than relying on the original bootstrap servers we could get new, definitely valid bootstrap brokers with each metadata refresh. (It might even be possible to implement this just by having the broker return extra brokers in the metadata response even if there aren't entries in the topic metadata section of the response referencing those brokers; but I'm not sure if that'd break anything, either in our clients or in third party clients. It also doesn't fully fix the problem since the linked issue can still occur, but should only ever happen in the case of their tiny test case, not in practice.) > NetworkClient may connect to a different Kafka cluster than originally > configured > --------------------------------------------------------------------------------- > > Key: KAFKA-3068 > URL: https://issues.apache.org/jira/browse/KAFKA-3068 > Project: Kafka > Issue Type: Bug > Components: clients > Affects Versions: 0.9.0.0 > Reporter: Jun Rao > Assignee: Eno Thereska > > In https://github.com/apache/kafka/pull/290, we added the logic to cache all > brokers (id and ip) that the client has ever seen. If we can't find an > available broker from the current Metadata, we will pick a broker that we > have ever seen (in NetworkClient.leastLoadedNode()). > One potential problem this logic can introduce is the following. Suppose that > we have a broker with id 1 in a Kafka cluster. A producer client remembers > this broker in nodesEverSeen. At some point, we bring down this broker and > use the host in a different Kafka cluster. Then, the producer client uses > this broker from nodesEverSeen to refresh metadata. It will find the metadata > in a different Kafka cluster and start producing data there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)