Junyang Liu created KAFKA-14189:
-----------------------------------
Summary: reuse connection of coordinator and leader in
KafkaConsumer
Key: KAFKA-14189
URL: https://issues.apache.org/jira/browse/KAFKA-14189
Project: Kafka
Issue Type: Improvement
Components: clients
Affects Versions: 0.9.0.0
Reporter: Junyang Liu
The connection id of connection with coordinator in KafkaConsumer is
Integer.MAX_VALUE - coordinator id, which is different with connection id of
partition leader. So the connection cannot be reused when coordinator and
leader are in the same broker, which means we need two seperated connections
with the same broker. Suppose such case, a consumer has connected to the
coordinator and finished Join and Sync, and wants to send FETCH to leader in
the same broker. But the connection count has reached limit, so the consumer
with be in the group but cannot consume messages
The root cause of this issue is that the process of consuming is a combination
of multiple connections(connections with coordinator and leader in same
broker), not atomic, which may leads to "half connected". I think we can do
some improvement:
# reuse the connection with coordinator and leader in the same broker
# make the connection limit more flexible, such as allowing extra related
connections of a consumer when the connection count limit has reached if it has
connected to broker
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
