[ https://issues.apache.org/jira/browse/KAFKA-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106190#comment-17106190 ]
Joost van de Wijgerd commented on KAFKA-9953: --------------------------------------------- Hi [~guozhang] thanks for the information, Indeed I see that the SerDes pass in the topic so this can indeed be used to select the proper deserializer in that case. I guess the typing on the Consumer and ConsumerRecords interface would have to be Object though as I am now mixing diffent java class types. The other situation (b) is a bit more problematic for us. We have a constant stream of DomainEvents which are processed exactly once. If we would like to process an event from that stream (one that was already there but we weren't interested before) then rewinding the existing consumer would be hard to do (not impossible of course) and it would block processing of newer events until we have caught up again. In this scenario, adding a new consumer(group) would be much simpler to implement (at the expense of network traffic of course, but for us this is not an issue). Arguably this could all be solved with a single consumer -> producer mapping but unfortunately this is not the route we took. Since we are processing financial transactions with this system I am very wary of rewriting our consumer logic as we would have to drop our current consumer groups and create a new one (for the whole service) and if we are not careful we might miss transactions . Another solution would be to pair a producer to each consumer but that will add extra overhead as well. Maybe our use case / implementation is a bit out of the ordinary but fact is that the kafka-clients library allows this construct and it also works. Except for the fact that a latency of around 105ms is introduced with the standards settings because the TransactionManager has to keep refreshing the groupCoordinator Node. So best practices notwithstanding I think it makes sense for the kafka-clients library to support many-to-one consumer -> producer mappings in a performant way. If you decide not to, then maybe add at least a warn log when this situation is detected inside the TransactionManager to alert developers they are doing something wrong that will hurt the performance of their app. Obviously my preference is to have this fix in the kafka-clients library ;) > support multiple consumerGroupCoordinators in TransactionManager > ---------------------------------------------------------------- > > Key: KAFKA-9953 > URL: https://issues.apache.org/jira/browse/KAFKA-9953 > Project: Kafka > Issue Type: Improvement > Components: clients > Affects Versions: 2.5.0 > Reporter: Joost van de Wijgerd > Priority: Major > Attachments: KAFKA-9953.patch > > > We are using kafka with a transactional producer and have the following use > case: > 3 KafkaConsumers (each with their own ConsumerGroup) polled by the same > thread and 1 transactional kafka producer. When we add the offsets to the > transaction we run into the following problem: > TransactionManager only keeps track of 1 consumerGroupCoordinator, however it > can be that some consumerGroupCoordinators are on another node, now we > constantly see the TransactionManager switching between nodes, this has > overhead of 1 failing _TxnOffsetCommitRequest_ and 1 unnecessary > _FindCoordinatorRequest_. > Also with _retry.backoff.ms_ set to 100 by default this is causing a pause > of 100ms for every other transaction (depending on what KafkaConsumer > triggered the transaction of course) > If the TransactionManager could keep track of coordinator nodes per > consumerGroupId this problem would be solved. > I have already a patch for this but still need to test it. Will add it to the > ticket when that is done -- This message was sent by Atlassian Jira (v8.3.4#803005)