Sudarshan Pathalam created KAFKA-13450:
------------------------------------------
Summary: Kafka broker to return alternating sets of listeners to
clients for multipathing
Key: KAFKA-13450
URL: https://issues.apache.org/jira/browse/KAFKA-13450
Project: Kafka
Issue Type: Improvement
Components: network
Reporter: Sudarshan Pathalam
It is not uncommon to deploy Kafka for high throughput use cases. In this
context operators could have multiple network paths for client to broker
traffic.
The use of multiple network paths is not only to cover for failure conditions
but also to balance load between the available paths. Common link speeds
(currently) in data centers/cloud are 10Gbps/40Gbps and when there is a need to
scale past the link speed multiple paths (multipathing) to the target (Kafka)
is desired. The option of bundling multiple links in to a single channel is not
commonly offered as a user facing feature in the Cloud.
However when using DNS to load balance between the available network paths, TTL
caching creates traffic skew between the network paths causing either
deterioration or failure.
It would be ideal for the broker to return alternating sets of listener names
to the metadata requests made by the client for even distribution of traffic
between the available network paths. KIP 302 provides the ability for the
clients to attempt to use all IP address for a name but does not cover
situations when there is a need for multiple names each pointing to a separate
network path.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
