[
https://issues.apache.org/jira/browse/KAFKA-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135011#comment-17135011
]
Boyang Chen commented on KAFKA-9953:
------------------------------------
Thanks for reporting the use case [~jwijgerd] . At a minimum, the EOS model has
a recommended usage pattern that supports one -> one mapping from Consumer to
the Producer. I have concern that by extending the logic, what's the impact to
the new users when they approach writing EOS apps, should we also extend the
example with many -> one mapping? Isn't the API giving the user a harder time
to make it right, such as mixing the group.id or group metadata for different
consumer groups unnecessarily. Also the consumer groups are rebalancing at
separate time slots, which make the scenario even more complicated.
As far my perception is that our discussed scenario here is not common, so
would it be possible for you to just starting more producers to match
consumers? How much impact would it be to your production system, in terms of
stability and cost?
> support multiple consumerGroupCoordinators in TransactionManager
> ----------------------------------------------------------------
>
> Key: KAFKA-9953
> URL: https://issues.apache.org/jira/browse/KAFKA-9953
> Project: Kafka
> Issue Type: Improvement
> Components: clients
> Affects Versions: 2.5.0
> Reporter: Joost van de Wijgerd
> Priority: Major
> Attachments: KAFKA-9953.patch
>
>
> We are using kafka with a transactional producer and have the following use
> case:
> 3 KafkaConsumers (each with their own ConsumerGroup) polled by the same
> thread and 1 transactional kafka producer. When we add the offsets to the
> transaction we run into the following problem:
> TransactionManager only keeps track of 1 consumerGroupCoordinator, however it
> can be that some consumerGroupCoordinators are on another node, now we
> constantly see the TransactionManager switching between nodes, this has
> overhead of 1 failing _TxnOffsetCommitRequest_ and 1 unnecessary
> _FindCoordinatorRequest_.
> Also with _retry.backoff.ms_ set to 100 by default this is causing a pause
> of 100ms for every other transaction (depending on what KafkaConsumer
> triggered the transaction of course)
> If the TransactionManager could keep track of coordinator nodes per
> consumerGroupId this problem would be solved.
> I have already a patch for this but still need to test it. Will add it to the
> ticket when that is done
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
