[
https://issues.apache.org/jira/browse/KAFKA-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junyang Liu updated KAFKA-14189:
--------------------------------
Summary: improve connection limit and reuse of coordinator and leader in
KafkaConsumer (was: connection count limit of coordinator and leader in
KafkaConsumer)
> improve connection limit and reuse of coordinator and leader in KafkaConsumer
> -----------------------------------------------------------------------------
>
> Key: KAFKA-14189
> URL: https://issues.apache.org/jira/browse/KAFKA-14189
> Project: Kafka
> Issue Type: Improvement
> Components: clients
> Affects Versions: 0.9.0.0
> Reporter: Junyang Liu
> Priority: Major
>
> The connection id of connection with coordinator in KafkaConsumer is
> Integer.MAX_VALUE - coordinator id, which is different with connection id of
> partition leader. So the connection cannot be reused when coordinator and
> leader are in the same broker, which means we need two seperated connections
> with the same broker. Suppose such case, a consumer has connected to the
> coordinator and finished Join and Sync, and wants to send FETCH to leader in
> the same broker. But the connection count has reached limit, so the consumer
> with be in the group but cannot consume messages
>
> The root cause of this issue is that the process of consuming is a
> combination of multiple connections(connections with coordinator and leader
> in same broker), not atomic, which may leads to "half connected". I think we
> can do some improvement:
> # reuse the connection with coordinator and leader in the same broker
> # make the connection limit more flexible, such as allowing extra related
> connections of a consumer when the connection count limit has reached if it
> has connected to broker
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
