[
https://issues.apache.org/jira/browse/KAFKA-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17104893#comment-17104893
]
Guozhang Wang commented on KAFKA-9953:
--------------------------------------
[~jwijgerd] I'm curious to learn your scenario, more specifically, what makes
you needing three consumers instead of one consumer mapped to one producer? Is
there any throughput limits or it is due to other reasons?
> support multiple consumerGroupCoordinators in TransactionManager
> ----------------------------------------------------------------
>
> Key: KAFKA-9953
> URL: https://issues.apache.org/jira/browse/KAFKA-9953
> Project: Kafka
> Issue Type: Improvement
> Components: clients
> Affects Versions: 2.5.0
> Reporter: Joost van de Wijgerd
> Priority: Major
> Attachments: KAFKA-9953.patch
>
>
> We are using kafka with a transactional producer and have the following use
> case:
> 3 KafkaConsumers (each with their own ConsumerGroup) polled by the same
> thread and 1 transactional kafka producer. When we add the offsets to the
> transaction we run into the following problem:
> TransactionManager only keeps track of 1 consumerGroupCoordinator, however it
> can be that some consumerGroupCoordinators are on another node, now we
> constantly see the TransactionManager switching between nodes, this has
> overhead of 1 failing _TxnOffsetCommitRequest_ and 1 unnecessary
> _FindCoordinatorRequest_.
> Also with _retry.backoff.ms_ set to 100 by default this is causing a pause
> of 100ms for every other transaction (depending on what KafkaConsumer
> triggered the transaction of course)
> If the TransactionManager could keep track of coordinator nodes per
> consumerGroupId this problem would be solved.
> I have already a patch for this but still need to test it. Will add it to the
> ticket when that is done
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
