Ravi, It is not a bug. Broker assumes that your consumer faced live-lock. You need to tune property max.poll.interval.ms to increase expected interval between poll() on consumer side.
-- Lukasz sob., 21 mar 2020 o 02:19 Ravi Kanth <ravikanth....@gmail.com> napisał(a): > Hi All, > > I have a Kafka Consumer that polls the data and gets *paused* for 15-20 > mins for the post-processing of the polled records. However, during the > time of pause, the broker assumes that the consumer group is dead(check > below log) and rebalances the consumer group. > > *Application Log:* > k8s-worker403: 2020-03-20 21:41:14.147 [kafka-coordinator-heartbeat-thread > | <consumer_group>] INFO o.a.k.c.c.i.AbstractCoordinator - [Consumer > clientId=consumer-1, groupId=<consumer_group>] Marking the coordinator > <broker>:9093 (id: 2147483480 rack: null) dead > > *Broker Log:* > Member consumer-1-4eb939a6-c37d-4853-96dc-4255ec8663ff in group <consumer > group> has failed, removing it from the group > > When rebalancing happens, consumer is automatically *resumed* and starts > polling for more records though I am expecting it to be paused. This is > causing unexpected behavior. > > When reading through Kafka docs, it looks like tuning up > request.timeout.ms/session.timeout.ms will help avoid running into this > problem. Please correct if my understanding is incorrect. However, I am not > willing to tweak the configurations, instead handle the problem at the code > level. > > *Question:* > 1. Is this a bug in Kafka Java API letting the consumer be resumed after > rebalancing even when it is initially paused? If so, details of the bug > would be helpful. > 2. Is there anyother way to avoid running into this problem without > tweaking configurations? > > *Kafka Version Using:* > *1.0.1* > > Any suggestions would be of great help. > > Thanks, > Ravi >
