Thank you Luke, Here is the logs from client side (rebalance event listener)
2021-12-08 16:30:47,301 INFO com.***.KafkaListener Partition removed from listener 2021-12-08 16:30:47,302 INFO com.***.KafkaListener Topic: **InternalLoggingMetadataInfo, partition: 1 2021-12-08 16:30:50,066 INFO com.***.KafkaListener Partition assigned to listener 2021-12-08 16:30:50,066 INFO com.***.KafkaListener Topic: **InternalLoggingMetadataInfo, partition: 2 2021-12-08 16:30:50,066 INFO com.***.KafkaListener Topic: **InternalLoggingMetadataInfo, partition: 0 2021-12-08 16:31:20,047 INFO com.***.KafkaListener Partition removed from listener 2021-12-08 16:31:20,047 INFO com.***.KafkaListener Topic: **InternalLoggingMetadataInfo, partition: 2 2021-12-08 16:31:20,047 INFO com.***.KafkaListener Topic: **InternalLoggingMetadataInfo, partition: 0 2021-12-08 16:31:20,070 INFO com.***.KafkaListener Partition assigned to listener 2021-12-08 16:31:20,070 INFO com.***.KafkaListener Topic: **InternalLoggingMetadataInfo, partition: 0 There is no relevant rebalance event logged on the consumer in trouble (app008) I am not sure how to attach the image in this thread as I am using gmail. It basically shows the members in a consumer group, with one of the consumers (app008) missing even though the consumer is still alive and consuming events. Here is the duplicated Event Processing log app008 [ consumer not in consumer group however still consuming events ] 2021-12-09 04:39:39 Received message from topic **InternalLoggingMetadataInfo, partition: 0, offset: 375, RequestId: b9740fdd-bc1c-4ef3-bb60-7cb8ab623af3 app009 [ Consumer taking over partition 0 from app08, since rebalance event above ] 2021-12-09 04:39:39 Received message from topic ***InternalLoggingMetadataInfo, partition: 0, offset: 375, RequestId: b9740fdd-bc1c-4ef3-bb60-7cb8ab623af3 I don't have access to the broker side, so I am sorry, I don't have logs from the broker. Thanks! On 2021/12/11 09:01:05 Luke Chen wrote: > Hi Tao, > > I think we need the logs (client and broker side) to know clearly what > happened there. > Besides, I can't see the image you attached under "Consumer Group State > (Please note that app008 is kicked out)" section. > > One thing to note is that, we've fixed some bugs related to rebalance and > partition assignments in newer releases, maybe you can try to upgrade to > newer version to see if it happens again (if possible :) ) > > Thank you. > Luke > > On Fri, Dec 10, 2021 at 11:13 PM Tao Huang <sa...@gmail.com> wrote: > > > Hi there, > > > > We have experienced issues on Topic Partition assignment and lead to > > duplicate process of event. What happened is that, somehow one listener > > (consumer) was taken out by the Broker (lost heartbeat??), and the assigned > > partition was taken over by another consumer in the same consumer group. > > However, the listener (consumer) kicked out is not really dead, it > > continues pulling and consuming the event and has no idea about the > > partition revocation. > > > > *# Consumer Group State (Please note that app008 is kicked out)* > > > > [image: image.png] > > > > *# When we identified the issue and restarted the listener on app008, it > > shows partition 0 is removed (means, it still binds to partition 0)* > > > > 2021-12-09 22:05:23 INFO Shutting down listener on ****LoggingMetadataInfo > > 2021-12-09 22:05:23 INFO Partition removed from listener: partition: 0 > > 2021-12-09 22:05:23 INFO ****LoggingMetadataInfo listener destroyed > > > > *Version: *2.6.1 > > *PARTITION_ASSIGNMENT_STRATEGY:* RoundRobinAssignor > > > > Wondering if anyone has experienced a similar case? Any suggestions on how > > to mitigate such risk? > > > > Thanks! > > > > Tao > > >
