Hi Paul, We tried patching out kafka connect as well referencing the PR which you had shared. But no improvement in our case. Have updated more details in the blocker issue which I have created for the same. https://issues.apache.org/jira/browse/KAFKA-8485
But the issue seems to be coming from the internal producers/consumers getting into hung state after the broker restarts. Any other lead/inputs would be useful. This issue is always reproducible even with/without data streaming in progress. Thanks, Kaushik -----Original Message----- From: Paul Whalen <pgwha...@gmail.com> Sent: Wednesday, June 05, 2019 5:17 PM To: dev@kafka.apache.org Cc: Basil Brito, Aldan (Nokia - IN/Bangalore) <aldan.basil_br...@nokia.com> Subject: Re: kafka connect stops consuming data when kafka broker goes down It’s not totally clear, but this may be https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-7941 For which there is a fix that is very nearly approved: https://github.com/apache/kafka/pull/6283 Paul > On Jun 5, 2019, at 1:26 AM, Srinivas, Kaushik (Nokia - IN/Bangalore) > <kaushik.srini...@nokia.com> wrote: > > Hello, > Anyone has any information on this issue. > Created a critical ticket for the same, since this is a major stability issue > for connect framework. > https://issues.apache.org/jira/browse/KAFKA-8485?filter=-2 > > Thanks. > Kaushik, > NOKIA > > From: Srinivas, Kaushik (Nokia - IN/Bangalore) > Sent: Monday, June 03, 2019 5:22 PM > To: dev@kafka.apache.org > Cc: Basil Brito, Aldan (Nokia - IN/Bangalore) > <aldan.basil_br...@nokia.com> > Subject: kafka connect stops consuming data when kafka broker goes > down > > Hello kafka dev, > > We are encountering an issue when kafka connect is running hdfs sink > connector pulling data from kafka and writing to hdfs location. > In between when the data is flowing in the pipeline from producer -> > kafka topic -> kafka connect hdfs sink connector -> hdfs, If even one of the > kafka broker goes down, the connect framework stops responding. Stops > consuming records and REST API also becomes not interactive. > > Until the kafka connect framework is restarted, it would not pull the data > from kafka and REST api remains inactive. Nothing is coming in the logs as > well. > Checked the topics in kafka used by connect, everything has been reassigned > to another broker and has the leader available. > > Has anyone encountered this issue ? what would be the expected behavior ? > > Thanks in advance > Kaushik
