[ https://issues.apache.org/jira/browse/KAFKA-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junyang Liu updated KAFKA-14189: -------------------------------- Summary: connection count limit of coordinator and leader in KafkaConsumer (was: reuse connection of coordinator and leader in KafkaConsumer) > connection count limit of coordinator and leader in KafkaConsumer > ----------------------------------------------------------------- > > Key: KAFKA-14189 > URL: https://issues.apache.org/jira/browse/KAFKA-14189 > Project: Kafka > Issue Type: Improvement > Components: clients > Affects Versions: 0.9.0.0 > Reporter: Junyang Liu > Priority: Major > > The connection id of connection with coordinator in KafkaConsumer is > Integer.MAX_VALUE - coordinator id, which is different with connection id of > partition leader. So the connection cannot be reused when coordinator and > leader are in the same broker, which means we need two seperated connections > with the same broker. Suppose such case, a consumer has connected to the > coordinator and finished Join and Sync, and wants to send FETCH to leader in > the same broker. But the connection count has reached limit, so the consumer > with be in the group but cannot consume messages > > The root cause of this issue is that the process of consuming is a > combination of multiple connections(connections with coordinator and leader > in same broker), not atomic, which may leads to "half connected". I think we > can do some improvement: > # reuse the connection with coordinator and leader in the same broker > # make the connection limit more flexible, such as allowing extra related > connections of a consumer when the connection count limit has reached if it > has connected to broker -- This message was sent by Atlassian Jira (v8.20.10#820010)