We have a use case where in we want to assign partitions manually for a set of topics to allow fine grained control of the records we are fetching.
Basically what we are trying to achieve is that a group of messages which logically belong to a particular entity should be sent to the same partition . So, Topic A and topic B both have a certain field which are unique and common( say some id = 101) so we want to send both of those records to partition 1 ( 101 % 10 is our simple partition assignment strategy written in both producer and Kafka Connect interceptor. Fro what I understood, if I want to have a consumer which listens to partition 1 for both Topic A and Topic B, then we need to use the assign method. I have been reading up a bit and what has been mentioned everywhere is that we won't have any rebalancing triggered. Also, I tried a simple use case where in I didn't poll for more than the value configured in group.max.session.timeout.ms but it didn't die. Is it because its not part of a consumer group per se? So just wanted to understand what points should we declare a consumer is dead so that we can spin up a new consumer for the same topic partition. We are using AWS ECS for running our consumers so target group would spin up a new consumer based upon health checks. Any examples + guidelines around this would be highly appreciated. Thanks! Sagar.
