In kafka streams consumer seems to hang while retrieving the offsets

[Sachin Mittal]

Hi,
In my streams applications cluster in one or more instances I see some
threads always waiting with the following stack.

Every time I check on jstack I see the following trace.

Is this some kind of new deadlock that we have failed to identify.

Thanks
Sachin

here is the stack trace:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
"StreamThread-4" #20 prio=5 os_prio=0 tid=0x00007fb814be3000 nid=0x19bf
runnable [0x00007fb7cb4f6000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x0000000701c50c98> (a sun.nio.ch.Util$3)
        - locked <0x0000000701c50c88> (a java.util.Collections$
UnmodifiableSet)
        - locked <0x0000000701c4f6a8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.kafka.common.network.Selector.select(
Selector.java:489)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:298)
        at org.apache.kafka.clients.NetworkClient.poll(
NetworkClient.java:349)
        at org.apache.kafka.clients.consumer.internals.
ConsumerNetworkClient.poll(ConsumerNetworkClient.java:226)
        - locked <0x0000000701c5da48> (a org.apache.kafka.clients.
consumer.internals.ConsumerNetworkClient)
        at org.apache.kafka.clients.consumer.internals.
ConsumerNetworkClient.poll(ConsumerNetworkClient.java:203)
        at org.apache.kafka.clients.consumer.internals.
ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:138)
        at org.apache.kafka.clients.consumer.internals.Fetcher.
retrieveOffsetsByTimes(Fetcher.java:422)
        at org.apache.kafka.clients.consumer.internals.Fetcher.
resetOffset(Fetcher.java:370)
        at org.apache.kafka.clients.consumer.internals.Fetcher.
resetOffsetsIfNeeded(Fetcher.java:227)
        at org.apache.kafka.clients.consumer.KafkaConsumer.
updateFetchPositions(KafkaConsumer.java:1592)
        at org.apache.kafka.clients.consumer.KafkaConsumer.
position(KafkaConsumer.java:1265)
        at org.apache.kafka.streams.processor.internals.
ProcessorStateManager.restoreActiveState(ProcessorStateManager.java:213)