May i know what log level did you configured on consumer and broker? you say it will resume when rebalance happen, so consumer is alive, can you see any heartbeat information in consumer log?
Best, Lisheng sampath kumar <sampath...@gmail.com> 于2019年8月21日周三 下午5:23写道： > Hi, > > Using Broker 5.3.0, new consumers(Consumers managed by brokers). Brokers > are deployed in a Kubernetes environment > > Number of brokers : 3, Number of 3 Zookeeper setup > > One of the Topic "inventory.request" we have 3 replication, with insync > replicas configured as 2 and partition count is 1024 > > We have 20 instances of microservice subscribe to the above topic, each > instance will have 48 consumers registered as a group "agent.group.inv" > > Issue : > > Here some times a couple of the consumers suddenly stopped receiving the > request, and lag seems to keep increasing. Only option to recover it > restart the consumers and invokes rebalancing > > ``` agent.group.inv inventory.request 543 17423 17612 > 189 agent19.inv.35-6e6eb252-8d26-489b-8d7f-53b25f182f30 > /10.200.187.103 agent19.inv.35 ``` > > we checked the thread dump of the consumer, the consumer keeps performing > polling and assigned with partitions, However not receiving the any > messages > > ``` "inventory.request-agent19.inv.35" #499 prio=1 os_prio=4 > tid=0x00007f88a855b000 nid=0x389 runnable [0x00007f87e8be6000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000000aa502730> (a sun.nio.ch.Util$3) > - locked <0x00000000aa5026b0> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00000000aa502668> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > org.apache.kafka.common.network.Selector.select(Selector.java:794) > at org.apache.kafka.common.network.Selector.poll(Selector.java:467) > at > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:539) > at > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262) > at > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233) > at > > org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1281) > at > > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1225) > at > > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1159) > ``` > > > No errors are observed consumer client, brokers and also resource issue not > seen. > > Can you please help us in identifying the root cause for this consumer > client behavior? > > Please let me know if any other details required? > > -- > Regards, > Sampath >
