Hi Tao, Sorry, I can't see anything wrong from these logs. But one thing I'd like to confirm: are you using Kafka directly? Or you're using Kafka via 3rd party library? The logs you provided don't look like the logs from Kafka itself. (or it's because you modified the logs for privacy?) It'll be better to have the Kafka logs to do troubleshooting.
But I think it is still a good idea to upgrade your Kafka to newer versions than 2.6.1, ex: v3.0.0, 2.8.1 Thank you. Luke On Sun, Dec 12, 2021 at 7:49 AM Tao Huang <sandy.huang...@gmail.com> wrote: > Thank you Luke, > > Here is the logs from client side (rebalance event listener) > > 2021-12-08 16:30:47,301 INFO com.***.KafkaListener Partition removed from > listener > 2021-12-08 16:30:47,302 INFO com.***.KafkaListener Topic: > **InternalLoggingMetadataInfo, partition: 1 > 2021-12-08 16:30:50,066 INFO com.***.KafkaListener Partition assigned to > listener > 2021-12-08 16:30:50,066 INFO com.***.KafkaListener Topic: > **InternalLoggingMetadataInfo, partition: 2 > 2021-12-08 16:30:50,066 INFO com.***.KafkaListener Topic: > **InternalLoggingMetadataInfo, partition: 0 > 2021-12-08 16:31:20,047 INFO com.***.KafkaListener Partition removed from > listener > 2021-12-08 16:31:20,047 INFO com.***.KafkaListener Topic: > **InternalLoggingMetadataInfo, partition: 2 > 2021-12-08 16:31:20,047 INFO com.***.KafkaListener Topic: > **InternalLoggingMetadataInfo, partition: 0 > 2021-12-08 16:31:20,070 INFO com.***.KafkaListener Partition assigned to > listener > 2021-12-08 16:31:20,070 INFO com.***.KafkaListener Topic: > **InternalLoggingMetadataInfo, partition: 0 > > There is no relevant rebalance event logged on the consumer in trouble > (app008) > > I am not sure how to attach the image in this thread as I am using gmail. > It basically shows the members in a consumer group, with one of the > consumers (app008) missing even though the consumer is still alive and > consuming events. > > Here is the duplicated Event Processing log > > app008 [ consumer not in consumer group however still consuming events ] > 2021-12-09 04:39:39 Received message from topic > **InternalLoggingMetadataInfo, partition: 0, offset: 375, RequestId: > b9740fdd-bc1c-4ef3-bb60-7cb8ab623af3 > > app009 [ Consumer taking over partition 0 from app08, since rebalance event > above ] > 2021-12-09 04:39:39 Received message from topic > ***InternalLoggingMetadataInfo, partition: 0, offset: 375, RequestId: > b9740fdd-bc1c-4ef3-bb60-7cb8ab623af3 > > I don't have access to the broker side, so I am sorry, I don't have logs > from the broker. > > Thanks! > > On 2021/12/11 09:01:05 Luke Chen wrote: > > Hi Tao, > > > > I think we need the logs (client and broker side) to know clearly what > > happened there. > > Besides, I can't see the image you attached under "Consumer Group State > > (Please note that app008 is kicked out)" section. > > > > One thing to note is that, we've fixed some bugs related to rebalance and > > partition assignments in newer releases, maybe you can try to upgrade to > > newer version to see if it happens again (if possible :) ) > > > > Thank you. > > Luke > > > > On Fri, Dec 10, 2021 at 11:13 PM Tao Huang <sa...@gmail.com> wrote: > > > > > Hi there, > > > > > > We have experienced issues on Topic Partition assignment and lead to > > > duplicate process of event. What happened is that, somehow one listener > > > (consumer) was taken out by the Broker (lost heartbeat??), and the > assigned > > > partition was taken over by another consumer in the same consumer > group. > > > However, the listener (consumer) kicked out is not really dead, it > > > continues pulling and consuming the event and has no idea about the > > > partition revocation. > > > > > > *# Consumer Group State (Please note that app008 is kicked out)* > > > > > > [image: image.png] > > > > > > *# When we identified the issue and restarted the listener on app008, > it > > > shows partition 0 is removed (means, it still binds to partition 0)* > > > > > > 2021-12-09 22:05:23 INFO Shutting down listener on > ****LoggingMetadataInfo > > > 2021-12-09 22:05:23 INFO Partition removed from listener: partition: 0 > > > 2021-12-09 22:05:23 INFO ****LoggingMetadataInfo listener destroyed > > > > > > *Version: *2.6.1 > > > *PARTITION_ASSIGNMENT_STRATEGY:* RoundRobinAssignor > > > > > > Wondering if anyone has experienced a similar case? Any suggestions on > how > > > to mitigate such risk? > > > > > > Thanks! > > > > > > Tao > > > > > >
