According to the Kafka docs [1] the property is "session.timeout.ms".
Any user defined properties in the Kafka processors are passed along to the Kafka client as configuration. You create a user defined property from the properties screen using the + icon in the top right when the processor is stopped. Add a new property with name as session.timeout.ms and a value in milliseconds. The docs say the default value is 30000 so I'm concerned that something else is going on because I find it hard to believe that the broker hasn't heard from the consumer in NiFi for over 30 seconds when your processor is schedule for every 2 seconds, and there is no back-pressure. -Bryan [1] https://kafka.apache.org/0100/documentation.html#newconsumerconfigs On Tue, Mar 21, 2017 at 12:19 PM, Anshuman Ghosh <anshuman.ghosh2...@gmail.com> wrote: > Hello Bryan, > > One more thing, I have just checked the config/consumer.properties where the > consumer timeout is not set > Does it have to do anything with this? > > #consumer timeout > #consumer.timeout.ms=5000 > > # timeout in ms for connecting to zookeeper > zookeeper.connection.timeout.ms=6000 > > > Thank you! > ______________________ > > Kind Regards, > Anshuman Ghosh > Contact - +49 179 9090964 > > > On Tue, Mar 21, 2017 at 5:12 PM, Anshuman Ghosh > <anshuman.ghosh2...@gmail.com> wrote: >> >> Hello Bryan, >> >> Thank you so much for your reply! >> >> Just to answer your questions - >> >> Back Pressure is set to the next processor but that isn't triggered >> anytime (threshold limit is 1 GB) >> No other custom properties are set apart from the one I have shared with >> you. >> >> Also it suggests to increase the session timeout value - Where do we do >> that? >> Settings for my Kafka cluster is zookeeper.connection.timeout.ms=6000 >> >> >> Thank you! >> ______________________ >> >> Kind Regards, >> Anshuman Ghosh >> Contact - +49 179 9090964 >> >> >> On Tue, Mar 21, 2017 at 3:36 PM, Bryan Bende <bbe...@gmail.com> wrote: >>> >>> Anshuman, >>> >>> The error message is saying that NiFi tried to use a consumer where >>> the session already timed-out. So for example, if the session timeout >>> was 10 seconds, and NiFi had a consumer that hadn't been used in 12 >>> seconds and then tried to use it, you would get this error because the >>> session for that consumer already exceeded the 10 seconds and expired. >>> >>> The typical reasons you would see that error message are the following... >>> >>> - The processor run schedule exceeds the duration of the >>> session.timeout (i.e. processor scheduled to run every 30 seconds, >>> session.timeout is 10 seconds) >>> - The processor wasn't scheduled to run for some period of time due to >>> back-pressure, so even though the run schedule might be every 1 >>> second, it actually didn't execute for longer than the session.timeout >>> - The amount of time to perform a single poll() from Kafka takes >>> longer than the session.timeout, this is unlikely for NiFi but >>> theoretically possible for other consumers depending how they are >>> implemented >>> >>> From your screenshots it looks like ConsumeKafka_0_10 is scheduled to >>> run every 2 seconds... >>> >>> Do you ever have back-pressure between ConsumeKafka and the next >>> processors? >>> Do you have any other custom properties set in ConsumeKafka? I can >>> only see down to Max Uncommitted Time >>> >>> The good news is that newer versions of the Kafka 0.10.x client have >>> some built in heart-beating such that even if a consumer hasn't been >>> used in a while, it will continue to keep the session alive. >>> >>> Koji actually has a PR up to address this: >>> https://github.com/apache/nifi/pull/1527 >>> >>> Thanks, >>> >>> Bryan >>> >>> >>> On Tue, Mar 21, 2017 at 9:33 AM, Anshuman Ghosh >>> <anshuman.ghosh2...@gmail.com> wrote: >>> > Hello, >>> > >>> > Can anyone please help me with this error message "ConsumeKafka_0_10" >>> > processor. >>> > Earlier I was not getting these messages, now I am seeing this Error >>> > and >>> > Waring message (attached screenshot) and sometimes getting duplicates >>> > :( >>> > Processor is running though! >>> > >>> > I have attached the configuration parameters screenshots for both the >>> > Publisher and Consumer processor. >>> > >>> > Thank >>> > ing >>> > you >>> > in advance >>> > ! >>> > >>> > ______________________ >>> > >>> > Kind Regards, >>> > Anshuman Ghosh >>> > Contact - +49 179 9090964 >>> > >> >> >
