Joost van de Wijgerd created KAFKA-9953: -------------------------------------------
Summary: support multiple consumerGroupCoordinators in TransactionManager Key: KAFKA-9953 URL: https://issues.apache.org/jira/browse/KAFKA-9953 Project: Kafka Issue Type: Improvement Components: clients Affects Versions: 2.5.0 Reporter: Joost van de Wijgerd We are using kafka with a transactional producer and have the following use case: 3 KafkaConsumers (each with their own ConsumerGroup) polled by the same thread and 1 transactional kafka producer. When we add the offsets to the transaction we run into the following problem: TransactionManager only keeps track of 1 consumerGroupCoordinator, however it can be that some consumerGroupCoordinators are on another node, now we constantly see the TransactionManager switching between nodes, this has overhead of 1 failing _TxnOffsetCommitRequest_ and 1 unnecessary _FindCoordinatorRequest_. Also with _retry.backoff.ms_ set to 100 by default this is causing a pause of 100ms for every other transaction (depending on what KafkaConsumer triggered the transaction of course) If the TransactionManager could keep track of coordinator nodes per consumerGroupId this problem would be solved. I have already a patch for this but still need to test it. Will add it to the ticket when that is done -- This message was sent by Atlassian Jira (v8.3.4#803005)