[
https://issues.apache.org/jira/browse/KAFKA-13427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ekaterina Chernousova updated KAFKA-13427:
------------------------------------------
Description:
UPDATED 05/11/2021
According to the documentation, new connections should be blocked when the max
connections limit is reached except it's an inter-broker listener
[http://kafka.apache.org/documentation/#brokerconfigs_max.connections]
Shouldn't connections to external listeners be blocked/dropped when
max.connections limit is reached instead of waiting for an available connection
slot? Thanks!
The following tests have been executed:
*Test 1: Broker-wide limit*
Comment: connections to external listeners are not being blocked/dropped the
same way as it happens when per.ip broker-wide limit is reached, instead the
system waits for an available connection slot.
configuration
{code:java}
max.connection.creation.rate = 2147483647
max.connections = 10
max.connections.per.ip = 2147483647
max.connections.per.ip.overrides =
{code}
client logs when limit is reached (bin/kafka-console-consumer.sh)
{code:java}
[2021-11-05 05:56:57,714] WARN [Consumer
clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553]
Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 05:57:30,915] WARN [Consumer
clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553]
Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 05:58:01,022] WARN [Consumer
clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553]
Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 05:58:10,899] WARN [Consumer
clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553]
Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
{code}
broker
{code:java}
2021-11-05 05:51:10,798 INFO [GroupCoordinator 0]: Assignment received from
leader
__strimzi-topic-operator-kstreams-45caeee4-1322-411d-a7f2-0bdcb9e38863-StreamThread-1-consumer-b6659f01-7981-4090-a9d3-89b9b80c340f
for group __strimzi-topic-operator-kstreams for generation 1. The group has 1
members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-7]
2021-11-05 05:56:09,169 INFO [Controller id=0] Processing automatic preferred
replica leader election (kafka.controller.KafkaController)
[controller-event-thread]
2021-11-05 05:56:09,169 TRACE [Controller id=0] Checking need to trigger auto
leader balancing (kafka.controller.KafkaController) [controller-event-thread]
2021-11-05 05:56:09,173 DEBUG [Controller id=0] Topics not in preferred replica
for broker 0 HashMap() (kafka.controller.KafkaController)
[controller-event-thread]
2021-11-05 05:56:09,176 TRACE [Controller id=0] Leader imbalance ratio for
broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
{code}
*Test 2: per.ip broker-wide limit*
Comment: works as expected, however, it also blocks inter-broker connections
configuration
{code:java}
max.connection.creation.rate = 2147483647
max.connections = 100
max.connections.per.ip = 10
max.connections.per.ip.overrides =
{code}
client logs when limit is reached (bin/kafka-console-consumer.sh)
{code:java}
[2021-11-05 06:43:06,832] WARN [Consumer
clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:43:06,937] WARN [Consumer
clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:43:07,042] WARN [Consumer
clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:43:07,144] WARN [Consumer
clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:43:07,246] WARN [Consumer
clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
{code}
broker
{code:java}
2021-11-05 06:43:06,728 INFO Rejected connection from /172.17.0.1, address
already has the configured maximum of 10 connections. (kafka.network.Acceptor)
[data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
2021-11-05 06:43:06,831 INFO Rejected connection from /172.17.0.1, address
already has the configured maximum of 10 connections. (kafka.network.Acceptor)
[data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
2021-11-05 06:43:06,934 INFO Rejected connection from /172.17.0.1, address
already has the configured maximum of 10 connections. (kafka.network.Acceptor)
[data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
2021-11-05 06:43:07,039 INFO Rejected connection from /172.17.0.1, address
already has the configured maximum of 10 connections. (kafka.network.Acceptor)
[data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
2021-11-05 06:43:07,142 INFO Rejected connection from /172.17.0.1, address
already has the configured maximum of 10 connections. (kafka.network.Acceptor)
[data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
2021-11-05 06:43:07,246 INFO Rejected connection from /172.17.0.1, address
already has the configured maximum of 10 connections. (kafka.network.Acceptor)
[data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
2021-11-05 06:43:07,349 INFO Rejected connection from /172.17.0.1, address
already has the configured maximum of 10 connections. (kafka.network.Acceptor)
[data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
{code}
*Test 3: listener-level limit*
Comment: I assume connections should be blocked/dropped instead of waiting for
an available connection slot
configuration
{code:java}
max.connection.creation.rate = 2147483647
max.connections = 2147483647
max.connections.per.ip = 2147483647
max.connections.per.ip.overrides =
listener.name.example-9095.max.connections=10
{code}
client logs when limit is reached (bin/kafka-console-consumer.sh)
{code:java}
[2021-11-05 06:38:05,084] WARN [Consumer
clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043]
Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:38:38,175] WARN [Consumer
clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043]
Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:39:08,278] WARN [Consumer
clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043]
Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:39:17,228] WARN [Consumer
clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043]
Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
{code}
broker
{code:java}
2021-11-05 06:36:22,266 INFO [Controller id=0] Processing automatic preferred
replica leader election (kafka.controller.KafkaController)
[controller-event-thread]
2021-11-05 06:36:22,267 TRACE [Controller id=0] Checking need to trigger auto
leader balancing (kafka.controller.KafkaController) [controller-event-thread]
2021-11-05 06:36:22,274 DEBUG [Controller id=0] Topics not in preferred replica
for broker 0 HashMap() (kafka.controller.KafkaController)
[controller-event-thread]
2021-11-05 06:36:22,274 TRACE [Controller id=0] Leader imbalance ratio for
broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
{code}
*Test 4: Broker-wide limit with inter-broker connection*
Comment: works as expected
configuration
{code:java}
max.connection.creation.rate = 2147483647
max.connections = 10
max.connections.per.ip = 2147483647
max.connections.per.ip.overrides =
{code}
client that connects to an internal listener logs when limit is reached
(bin/kafka-console-consumer.sh)
{code:java}
[2021-11-05 06:56:51,602] WARN [Consumer
clientId=consumer-console-consumer-4479-1, groupId=console-consumer-4479]
Bootstrap broker my-cluster-kafka-bootstrap:9092 (id: -1 rack: null)
disconnected (org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:57:24,697] WARN [Consumer
clientId=consumer-console-consumer-4479-1, groupId=console-consumer-4479]
Bootstrap broker my-cluster-kafka-bootstrap:9092 (id: -1 rack: null)
disconnected (org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:57:57,797] WARN [Consumer
clientId=consumer-console-consumer-4479-1, groupId=console-consumer-4479]
Bootstrap broker my-cluster-kafka-bootstrap:9092 (id: -1 rack: null)
disconnected (org.apache.kafka.clients.NetworkClient)
{code}
broker
{code:java}
2021-11-05 06:52:42,869 INFO [GroupCoordinator 0]: Assignment received from
leader
__strimzi-topic-operator-kstreams-ece4c6be-1043-467c-8785-872910af313f-StreamThread-1-consumer-1ed22ca1-b161-40ab-921f-994cdabe5f82
for group __strimzi-topic-operator-kstreams for generation 1. The group has 1
members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-4]
2021-11-05 06:57:40,509 INFO [Controller id=0] Processing automatic preferred
replica leader election (kafka.controller.KafkaController)
[controller-event-thread]
2021-11-05 06:57:40,509 TRACE [Controller id=0] Checking need to trigger auto
leader balancing (kafka.controller.KafkaController) [controller-event-thread]
2021-11-05 06:57:40,518 DEBUG [Controller id=0] Topics not in preferred replica
for broker 0 HashMap() (kafka.controller.KafkaController)
[controller-event-thread]
2021-11-05 06:57:40,518 TRACE [Controller id=0] Leader imbalance ratio for
broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
{code}
was:
UPDATED 05/11/2021
According to the documentation, new connections should be blocked when the max
connections limit is reached except it's an inter-broker listener, however, it
seems an external listener is treated the same way as an inter-broker listener.
[http://kafka.apache.org/documentation/#brokerconfigs_max.connections]
Shouldn't connections to external listeners be blocked/dropped when
max.connections limit is reached instead of waiting for an available connection
slot? Thanks!
The following tests have been executed:
*Test 1: Broker-wide limit*
Comment: connections to external listeners are not being blocked/dropped the
same way as it happens when per.ip broker-wide limit is reached, instead the
system waits for an available connection slot.
configuration
{code:java}
max.connection.creation.rate = 2147483647
max.connections = 10
max.connections.per.ip = 2147483647
max.connections.per.ip.overrides =
{code}
client logs when limit is reached (bin/kafka-console-consumer.sh)
{code:java}
[2021-11-05 05:56:57,714] WARN [Consumer
clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553]
Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 05:57:30,915] WARN [Consumer
clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553]
Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 05:58:01,022] WARN [Consumer
clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553]
Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 05:58:10,899] WARN [Consumer
clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553]
Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
{code}
broker
{code:java}
2021-11-05 05:51:10,798 INFO [GroupCoordinator 0]: Assignment received from
leader
__strimzi-topic-operator-kstreams-45caeee4-1322-411d-a7f2-0bdcb9e38863-StreamThread-1-consumer-b6659f01-7981-4090-a9d3-89b9b80c340f
for group __strimzi-topic-operator-kstreams for generation 1. The group has 1
members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-7]
2021-11-05 05:56:09,169 INFO [Controller id=0] Processing automatic preferred
replica leader election (kafka.controller.KafkaController)
[controller-event-thread]
2021-11-05 05:56:09,169 TRACE [Controller id=0] Checking need to trigger auto
leader balancing (kafka.controller.KafkaController) [controller-event-thread]
2021-11-05 05:56:09,173 DEBUG [Controller id=0] Topics not in preferred replica
for broker 0 HashMap() (kafka.controller.KafkaController)
[controller-event-thread]
2021-11-05 05:56:09,176 TRACE [Controller id=0] Leader imbalance ratio for
broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
{code}
*Test 2: per.ip broker-wide limit*
Comment: works as expected, however, it also blocks inter-broker connections
configuration
{code:java}
max.connection.creation.rate = 2147483647
max.connections = 100
max.connections.per.ip = 10
max.connections.per.ip.overrides =
{code}
client logs when limit is reached (bin/kafka-console-consumer.sh)
{code:java}
[2021-11-05 06:43:06,832] WARN [Consumer
clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:43:06,937] WARN [Consumer
clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:43:07,042] WARN [Consumer
clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:43:07,144] WARN [Consumer
clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:43:07,246] WARN [Consumer
clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
{code}
broker
{code:java}
2021-11-05 06:43:06,728 INFO Rejected connection from /172.17.0.1, address
already has the configured maximum of 10 connections. (kafka.network.Acceptor)
[data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
2021-11-05 06:43:06,831 INFO Rejected connection from /172.17.0.1, address
already has the configured maximum of 10 connections. (kafka.network.Acceptor)
[data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
2021-11-05 06:43:06,934 INFO Rejected connection from /172.17.0.1, address
already has the configured maximum of 10 connections. (kafka.network.Acceptor)
[data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
2021-11-05 06:43:07,039 INFO Rejected connection from /172.17.0.1, address
already has the configured maximum of 10 connections. (kafka.network.Acceptor)
[data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
2021-11-05 06:43:07,142 INFO Rejected connection from /172.17.0.1, address
already has the configured maximum of 10 connections. (kafka.network.Acceptor)
[data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
2021-11-05 06:43:07,246 INFO Rejected connection from /172.17.0.1, address
already has the configured maximum of 10 connections. (kafka.network.Acceptor)
[data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
2021-11-05 06:43:07,349 INFO Rejected connection from /172.17.0.1, address
already has the configured maximum of 10 connections. (kafka.network.Acceptor)
[data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
{code}
*Test 3: listener-level limit*
Comment: I assume connections should be blocked/dropped instead of waiting for
an available connection slot
configuration
{code:java}
max.connection.creation.rate = 2147483647
max.connections = 2147483647
max.connections.per.ip = 2147483647
max.connections.per.ip.overrides =
listener.name.example-9095.max.connections=10
{code}
client logs when limit is reached (bin/kafka-console-consumer.sh)
{code:java}
[2021-11-05 06:38:05,084] WARN [Consumer
clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043]
Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:38:38,175] WARN [Consumer
clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043]
Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:39:08,278] WARN [Consumer
clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043]
Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:39:17,228] WARN [Consumer
clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043]
Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
(org.apache.kafka.clients.NetworkClient)
{code}
broker
{code:java}
2021-11-05 06:36:22,266 INFO [Controller id=0] Processing automatic preferred
replica leader election (kafka.controller.KafkaController)
[controller-event-thread]
2021-11-05 06:36:22,267 TRACE [Controller id=0] Checking need to trigger auto
leader balancing (kafka.controller.KafkaController) [controller-event-thread]
2021-11-05 06:36:22,274 DEBUG [Controller id=0] Topics not in preferred replica
for broker 0 HashMap() (kafka.controller.KafkaController)
[controller-event-thread]
2021-11-05 06:36:22,274 TRACE [Controller id=0] Leader imbalance ratio for
broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
{code}
*Test 4: Broker-wide limit with inter-broker connection*
Comment: works as expected
configuration
{code:java}
max.connection.creation.rate = 2147483647
max.connections = 10
max.connections.per.ip = 2147483647
max.connections.per.ip.overrides =
{code}
client that connects to an internal listener logs when limit is reached
(bin/kafka-console-consumer.sh)
{code:java}
[2021-11-05 06:56:51,602] WARN [Consumer
clientId=consumer-console-consumer-4479-1, groupId=console-consumer-4479]
Bootstrap broker my-cluster-kafka-bootstrap:9092 (id: -1 rack: null)
disconnected (org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:57:24,697] WARN [Consumer
clientId=consumer-console-consumer-4479-1, groupId=console-consumer-4479]
Bootstrap broker my-cluster-kafka-bootstrap:9092 (id: -1 rack: null)
disconnected (org.apache.kafka.clients.NetworkClient)
[2021-11-05 06:57:57,797] WARN [Consumer
clientId=consumer-console-consumer-4479-1, groupId=console-consumer-4479]
Bootstrap broker my-cluster-kafka-bootstrap:9092 (id: -1 rack: null)
disconnected (org.apache.kafka.clients.NetworkClient)
{code}
broker
{code:java}
2021-11-05 06:52:42,869 INFO [GroupCoordinator 0]: Assignment received from
leader
__strimzi-topic-operator-kstreams-ece4c6be-1043-467c-8785-872910af313f-StreamThread-1-consumer-1ed22ca1-b161-40ab-921f-994cdabe5f82
for group __strimzi-topic-operator-kstreams for generation 1. The group has 1
members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-4]
2021-11-05 06:57:40,509 INFO [Controller id=0] Processing automatic preferred
replica leader election (kafka.controller.KafkaController)
[controller-event-thread]
2021-11-05 06:57:40,509 TRACE [Controller id=0] Checking need to trigger auto
leader balancing (kafka.controller.KafkaController) [controller-event-thread]
2021-11-05 06:57:40,518 DEBUG [Controller id=0] Topics not in preferred replica
for broker 0 HashMap() (kafka.controller.KafkaController)
[controller-event-thread]
2021-11-05 06:57:40,518 TRACE [Controller id=0] Leader imbalance ratio for
broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
{code}
> Max connections are not blocked
> -------------------------------
>
> Key: KAFKA-13427
> URL: https://issues.apache.org/jira/browse/KAFKA-13427
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 3.0.0
> Reporter: Ekaterina Chernousova
> Priority: Minor
>
> UPDATED 05/11/2021
> According to the documentation, new connections should be blocked when the
> max connections limit is reached except it's an inter-broker listener
> [http://kafka.apache.org/documentation/#brokerconfigs_max.connections]
> Shouldn't connections to external listeners be blocked/dropped when
> max.connections limit is reached instead of waiting for an available
> connection slot? Thanks!
>
> The following tests have been executed:
> *Test 1: Broker-wide limit*
> Comment: connections to external listeners are not being blocked/dropped the
> same way as it happens when per.ip broker-wide limit is reached, instead the
> system waits for an available connection slot.
> configuration
> {code:java}
> max.connection.creation.rate = 2147483647
> max.connections = 10
> max.connections.per.ip = 2147483647
> max.connections.per.ip.overrides =
> {code}
> client logs when limit is reached (bin/kafka-console-consumer.sh)
> {code:java}
> [2021-11-05 05:56:57,714] WARN [Consumer
> clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553]
> Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 05:57:30,915] WARN [Consumer
> clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553]
> Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 05:58:01,022] WARN [Consumer
> clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553]
> Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 05:58:10,899] WARN [Consumer
> clientId=consumer-console-consumer-25553-1, groupId=console-consumer-25553]
> Bootstrap broker 192.168.49.2:31107 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> {code}
> broker
> {code:java}
> 2021-11-05 05:51:10,798 INFO [GroupCoordinator 0]: Assignment received from
> leader
> __strimzi-topic-operator-kstreams-45caeee4-1322-411d-a7f2-0bdcb9e38863-StreamThread-1-consumer-b6659f01-7981-4090-a9d3-89b9b80c340f
> for group __strimzi-topic-operator-kstreams for generation 1. The group has
> 1 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator)
> [data-plane-kafka-request-handler-7]
> 2021-11-05 05:56:09,169 INFO [Controller id=0] Processing automatic preferred
> replica leader election (kafka.controller.KafkaController)
> [controller-event-thread]
> 2021-11-05 05:56:09,169 TRACE [Controller id=0] Checking need to trigger auto
> leader balancing (kafka.controller.KafkaController) [controller-event-thread]
> 2021-11-05 05:56:09,173 DEBUG [Controller id=0] Topics not in preferred
> replica for broker 0 HashMap() (kafka.controller.KafkaController)
> [controller-event-thread]
> 2021-11-05 05:56:09,176 TRACE [Controller id=0] Leader imbalance ratio for
> broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
> {code}
> *Test 2: per.ip broker-wide limit*
> Comment: works as expected, however, it also blocks inter-broker connections
> configuration
> {code:java}
> max.connection.creation.rate = 2147483647
> max.connections = 100
> max.connections.per.ip = 10
> max.connections.per.ip.overrides =
> {code}
> client logs when limit is reached (bin/kafka-console-consumer.sh)
> {code:java}
> [2021-11-05 06:43:06,832] WARN [Consumer
> clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
> Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 06:43:06,937] WARN [Consumer
> clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
> Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 06:43:07,042] WARN [Consumer
> clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
> Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 06:43:07,144] WARN [Consumer
> clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
> Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 06:43:07,246] WARN [Consumer
> clientId=consumer-console-consumer-81841-1, groupId=console-consumer-81841]
> Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> {code}
> broker
> {code:java}
> 2021-11-05 06:43:06,728 INFO Rejected connection from /172.17.0.1, address
> already has the configured maximum of 10 connections.
> (kafka.network.Acceptor)
> [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
> 2021-11-05 06:43:06,831 INFO Rejected connection from /172.17.0.1, address
> already has the configured maximum of 10 connections.
> (kafka.network.Acceptor)
> [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
> 2021-11-05 06:43:06,934 INFO Rejected connection from /172.17.0.1, address
> already has the configured maximum of 10 connections.
> (kafka.network.Acceptor)
> [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
> 2021-11-05 06:43:07,039 INFO Rejected connection from /172.17.0.1, address
> already has the configured maximum of 10 connections.
> (kafka.network.Acceptor)
> [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
> 2021-11-05 06:43:07,142 INFO Rejected connection from /172.17.0.1, address
> already has the configured maximum of 10 connections.
> (kafka.network.Acceptor)
> [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
> 2021-11-05 06:43:07,246 INFO Rejected connection from /172.17.0.1, address
> already has the configured maximum of 10 connections.
> (kafka.network.Acceptor)
> [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
> 2021-11-05 06:43:07,349 INFO Rejected connection from /172.17.0.1, address
> already has the configured maximum of 10 connections.
> (kafka.network.Acceptor)
> [data-plane-kafka-socket-acceptor-ListenerName(EXAMPLE-9095)-PLAINTEXT-9095]
> {code}
> *Test 3: listener-level limit*
> Comment: I assume connections should be blocked/dropped instead of waiting
> for an available connection slot
> configuration
> {code:java}
> max.connection.creation.rate = 2147483647
> max.connections = 2147483647
> max.connections.per.ip = 2147483647
> max.connections.per.ip.overrides =
> listener.name.example-9095.max.connections=10
> {code}
> client logs when limit is reached (bin/kafka-console-consumer.sh)
> {code:java}
> [2021-11-05 06:38:05,084] WARN [Consumer
> clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043]
> Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 06:38:38,175] WARN [Consumer
> clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043]
> Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 06:39:08,278] WARN [Consumer
> clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043]
> Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 06:39:17,228] WARN [Consumer
> clientId=consumer-console-consumer-81043-1, groupId=console-consumer-81043]
> Bootstrap broker 192.168.49.2:30226 (id: -1 rack: null) disconnected
> (org.apache.kafka.clients.NetworkClient)
> {code}
> broker
> {code:java}
> 2021-11-05 06:36:22,266 INFO [Controller id=0] Processing automatic preferred
> replica leader election (kafka.controller.KafkaController)
> [controller-event-thread]
> 2021-11-05 06:36:22,267 TRACE [Controller id=0] Checking need to trigger auto
> leader balancing (kafka.controller.KafkaController) [controller-event-thread]
> 2021-11-05 06:36:22,274 DEBUG [Controller id=0] Topics not in preferred
> replica for broker 0 HashMap() (kafka.controller.KafkaController)
> [controller-event-thread]
> 2021-11-05 06:36:22,274 TRACE [Controller id=0] Leader imbalance ratio for
> broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
> {code}
> *Test 4: Broker-wide limit with inter-broker connection*
> Comment: works as expected
> configuration
> {code:java}
> max.connection.creation.rate = 2147483647
> max.connections = 10
> max.connections.per.ip = 2147483647
> max.connections.per.ip.overrides =
> {code}
> client that connects to an internal listener logs when limit is reached
> (bin/kafka-console-consumer.sh)
> {code:java}
> [2021-11-05 06:56:51,602] WARN [Consumer
> clientId=consumer-console-consumer-4479-1, groupId=console-consumer-4479]
> Bootstrap broker my-cluster-kafka-bootstrap:9092 (id: -1 rack: null)
> disconnected (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 06:57:24,697] WARN [Consumer
> clientId=consumer-console-consumer-4479-1, groupId=console-consumer-4479]
> Bootstrap broker my-cluster-kafka-bootstrap:9092 (id: -1 rack: null)
> disconnected (org.apache.kafka.clients.NetworkClient)
> [2021-11-05 06:57:57,797] WARN [Consumer
> clientId=consumer-console-consumer-4479-1, groupId=console-consumer-4479]
> Bootstrap broker my-cluster-kafka-bootstrap:9092 (id: -1 rack: null)
> disconnected (org.apache.kafka.clients.NetworkClient)
> {code}
> broker
> {code:java}
> 2021-11-05 06:52:42,869 INFO [GroupCoordinator 0]: Assignment received from
> leader
> __strimzi-topic-operator-kstreams-ece4c6be-1043-467c-8785-872910af313f-StreamThread-1-consumer-1ed22ca1-b161-40ab-921f-994cdabe5f82
> for group __strimzi-topic-operator-kstreams for generation 1. The group has
> 1 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator)
> [data-plane-kafka-request-handler-4]
> 2021-11-05 06:57:40,509 INFO [Controller id=0] Processing automatic preferred
> replica leader election (kafka.controller.KafkaController)
> [controller-event-thread]
> 2021-11-05 06:57:40,509 TRACE [Controller id=0] Checking need to trigger auto
> leader balancing (kafka.controller.KafkaController) [controller-event-thread]
> 2021-11-05 06:57:40,518 DEBUG [Controller id=0] Topics not in preferred
> replica for broker 0 HashMap() (kafka.controller.KafkaController)
> [controller-event-thread]
> 2021-11-05 06:57:40,518 TRACE [Controller id=0] Leader imbalance ratio for
> broker 0 is 0.0 (kafka.controller.KafkaController) [controller-event-thread]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
