Joost van de Wijgerd created KAFKA-9953:
-------------------------------------------
Summary: support multiple consumerGroupCoordinators in
TransactionManager
Key: KAFKA-9953
URL: https://issues.apache.org/jira/browse/KAFKA-9953
Project: Kafka
Issue Type: Improvement
Components: clients
Affects Versions: 2.5.0
Reporter: Joost van de Wijgerd
We are using kafka with a transactional producer and have the following use
case:
3 KafkaConsumers (each with their own ConsumerGroup) polled by the same thread
and 1 transactional kafka producer. When we add the offsets to the transaction
we run into the following problem:
TransactionManager only keeps track of 1 consumerGroupCoordinator, however it
can be that some consumerGroupCoordinators are on another node, now we
constantly see the TransactionManager switching between nodes, this has
overhead of 1 failing _TxnOffsetCommitRequest_ and 1 unnecessary
_FindCoordinatorRequest_.
Also with _retry.backoff.ms_ set to 100 by default this is causing a pause of
100ms for every other transaction (depending on what KafkaConsumer triggered
the transaction of course)
If the TransactionManager could keep track of coordinator nodes per
consumerGroupId this problem would be solved.
I have already a patch for this but still need to test it. Will add it to the
ticket when that is done
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
