Sudarshan Pathalam created KAFKA-13450: ------------------------------------------
Summary: Kafka broker to return alternating sets of listeners to clients for multipathing Key: KAFKA-13450 URL: https://issues.apache.org/jira/browse/KAFKA-13450 Project: Kafka Issue Type: Improvement Components: network Reporter: Sudarshan Pathalam It is not uncommon to deploy Kafka for high throughput use cases. In this context operators could have multiple network paths for client to broker traffic. The use of multiple network paths is not only to cover for failure conditions but also to balance load between the available paths. Common link speeds (currently) in data centers/cloud are 10Gbps/40Gbps and when there is a need to scale past the link speed multiple paths (multipathing) to the target (Kafka) is desired. The option of bundling multiple links in to a single channel is not commonly offered as a user facing feature in the Cloud. However when using DNS to load balance between the available network paths, TTL caching creates traffic skew between the network paths causing either deterioration or failure. It would be ideal for the broker to return alternating sets of listener names to the metadata requests made by the client for even distribution of traffic between the available network paths. KIP 302 provides the ability for the clients to attempt to use all IP address for a name but does not cover situations when there is a need for multiple names each pointing to a separate network path. -- This message was sent by Atlassian Jira (v8.20.1#820001)