Re: Queue wasn't emptying

[Christopher Hamm]

I will look, but how do I just kill it with fire. What can I delete?
On Nov 12, 2015 7:42 AM, "Mark Payne"  wrote:

> Chris / Elli,
>
> What version of NiFi are you running? I am wondering if you may have been
> bitten by
> this bug [1]. This should affect only the 'master' branch, not any
> formally released
> versions of NiFi. If you built the application from source code, rather
> than using an
> already-released version, this may well be what is happening.
>
> Thanks
> -Mark
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-1155
>
>
> On Nov 12, 2015, at 2:32 AM, Christopher Hamm 
> wrote:
>
> where does it dump? I cant get mine to expire fast enough. It is dev test
> I want to blow away but cant delete.
>
> On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt  wrote:
>
>> Elli,
>>
>> Can you share your flow configuration that we could possibly use to
>> replicate this?  Perhaps turn this into a JIRA and attach that.
>>
>> Also anytime you see something that appears 'stuck' please try to get
>> a stack dump (bin/nifi.sh dump)  As if there is truly a stuck thread
>> we'll see it there and what it is blocked on.
>>
>> Thanks
>> Joe
>>
>> On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz 
>> wrote:
>> > I had a queue that built up overnight with several thousand flowfiles.
>> The
>> > queue was pointing to a RouteOnAttribute processor, which was running.
>> For
>> > some reason, the RouteOnAttribute processor wasn't emptying the queue
>> so the
>> > queue just built up. Stopping and starting the processor didn't help.
>> > However, simply restarting Nifi got the flow moving again. I can't
>> figure
>> > out why the queue was stuck.
>> >
>> > What could cause a queue to build up like that? How would restarting
>> Nifi
>> > get it going again? There weren't any errors in the log.  I did have
>> > backpressure on the relationship to the processor after the
>> RouteOnAttribute
>> > set to 1, but that queue was empty. There was no backpressure on the
>> queue
>> > heading in to the RouteOnAttribute processor. The RouteOnAttribute
>> processor
>> > has a Penalty Duration and a Yield of 1 sec. The relationships all used
>> the
>> > PriorityAttributeAnalyzer as the prioritizer.
>> >
>> > The interesting thing is I had the exact same problem on two different
>> nifis
>> > (one was version 0.2.0 and one 0.3.0). The same type of flow files were
>> > going from one to the other using site-to-site (I was doing this for
>> load
>> > balancing). On the other Nifi, instead of restarting, I added another
>> > RouteOnAttribute processor and rerouted the stuck queue to the new one
>> > instead, and that also got things moving again. (I have a backpressure
>> since
>> > the processor after the RouteOnAttribute is a ControlRate processor,
>> and I
>> > only want to allow 1 flowfile through per second so as not to overwhelm
>> the
>> > system downstream. The backpressure is to assist with load balancing,
>> so if
>> > one pathway fills up, I have the other pathway which uses the
>> site-to-site
>> > to route the flow to the other nifi).
>> >
>> > Is there some combination of backpressure, penalty or yield along with
>> > prioritization that could cause a kind of deadlock-like situation? Any
>> ideas
>> > as to how I can prevent this from occurring?
>> >
>> > Thanks!
>> > -Elli
>> >
>>
>
>
>
> --
> Sincerely,
> Chris Hamm
> (E) ceham...@gmail.com
> (Twitter) http://twitter.com/webhamm
> (Linkedin) http://www.linkedin.com/in/chrishamm
>
>
>

Re: Queue wasn't emptying

[Mark Payne]

Chris / Elli,

What version of NiFi are you running? I am wondering if you may have been 
bitten by
this bug [1]. This should affect only the 'master' branch, not any formally 
released
versions of NiFi. If you built the application from source code, rather than 
using an
already-released version, this may well be what is happening.

Thanks
-Mark


[1] https://issues.apache.org/jira/browse/NIFI-1155 



> On Nov 12, 2015, at 2:32 AM, Christopher Hamm  
> wrote:
> 
> where does it dump? I cant get mine to expire fast enough. It is dev test I 
> want to blow away but cant delete.
> 
> On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt  > wrote:
> Elli,
> 
> Can you share your flow configuration that we could possibly use to
> replicate this?  Perhaps turn this into a JIRA and attach that.
> 
> Also anytime you see something that appears 'stuck' please try to get
> a stack dump (bin/nifi.sh dump)  As if there is truly a stuck thread
> we'll see it there and what it is blocked on.
> 
> Thanks
> Joe
> 
> On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz  > wrote:
> > I had a queue that built up overnight with several thousand flowfiles. The
> > queue was pointing to a RouteOnAttribute processor, which was running. For
> > some reason, the RouteOnAttribute processor wasn't emptying the queue so the
> > queue just built up. Stopping and starting the processor didn't help.
> > However, simply restarting Nifi got the flow moving again. I can't figure
> > out why the queue was stuck.
> >
> > What could cause a queue to build up like that? How would restarting Nifi
> > get it going again? There weren't any errors in the log.  I did have
> > backpressure on the relationship to the processor after the RouteOnAttribute
> > set to 1, but that queue was empty. There was no backpressure on the queue
> > heading in to the RouteOnAttribute processor. The RouteOnAttribute processor
> > has a Penalty Duration and a Yield of 1 sec. The relationships all used the
> > PriorityAttributeAnalyzer as the prioritizer.
> >
> > The interesting thing is I had the exact same problem on two different nifis
> > (one was version 0.2.0 and one 0.3.0). The same type of flow files were
> > going from one to the other using site-to-site (I was doing this for load
> > balancing). On the other Nifi, instead of restarting, I added another
> > RouteOnAttribute processor and rerouted the stuck queue to the new one
> > instead, and that also got things moving again. (I have a backpressure since
> > the processor after the RouteOnAttribute is a ControlRate processor, and I
> > only want to allow 1 flowfile through per second so as not to overwhelm the
> > system downstream. The backpressure is to assist with load balancing, so if
> > one pathway fills up, I have the other pathway which uses the site-to-site
> > to route the flow to the other nifi).
> >
> > Is there some combination of backpressure, penalty or yield along with
> > prioritization that could cause a kind of deadlock-like situation? Any ideas
> > as to how I can prevent this from occurring?
> >
> > Thanks!
> > -Elli
> >
> 
> 
> 
> -- 
> Sincerely,
> Chris Hamm
> (E) ceham...@gmail.com 
> (Twitter) http://twitter.com/webhamm 
> (Linkedin) http://www.linkedin.com/in/chrishamm 
> 

Re: Queue wasn't emptying

[Christopher Hamm]

that isn't clearing them out really. it gradually decreases but doesn't
clear out 500k items.
On Nov 12, 2015 8:22 AM, "Juan Sequeiros" 
wrote:

> To answer Christopher,
>
> You can clear out flowfiles on a queue by setting expiration on
> relationship line.
>
> On Thu, Nov 12, 2015 at 7:53 AM, Christopher Hamm 
> wrote:
>
>> I will look, but how do I just kill it with fire. What can I delete?
>> On Nov 12, 2015 7:42 AM, "Mark Payne"  wrote:
>>
>>> Chris / Elli,
>>>
>>> What version of NiFi are you running? I am wondering if you may have
>>> been bitten by
>>> this bug [1]. This should affect only the 'master' branch, not any
>>> formally released
>>> versions of NiFi. If you built the application from source code, rather
>>> than using an
>>> already-released version, this may well be what is happening.
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/NIFI-1155
>>>
>>>
>>> On Nov 12, 2015, at 2:32 AM, Christopher Hamm 
>>> wrote:
>>>
>>> where does it dump? I cant get mine to expire fast enough. It is dev
>>> test I want to blow away but cant delete.
>>>
>>> On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt  wrote:
>>>
 Elli,

 Can you share your flow configuration that we could possibly use to
 replicate this?  Perhaps turn this into a JIRA and attach that.

 Also anytime you see something that appears 'stuck' please try to get
 a stack dump (bin/nifi.sh dump)  As if there is truly a stuck thread
 we'll see it there and what it is blocked on.

 Thanks
 Joe

 On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz <
 eliezer_schw...@yahoo.com> wrote:
 > I had a queue that built up overnight with several thousand
 flowfiles. The
 > queue was pointing to a RouteOnAttribute processor, which was
 running. For
 > some reason, the RouteOnAttribute processor wasn't emptying the queue
 so the
 > queue just built up. Stopping and starting the processor didn't help.
 > However, simply restarting Nifi got the flow moving again. I can't
 figure
 > out why the queue was stuck.
 >
 > What could cause a queue to build up like that? How would restarting
 Nifi
 > get it going again? There weren't any errors in the log.  I did have
 > backpressure on the relationship to the processor after the
 RouteOnAttribute
 > set to 1, but that queue was empty. There was no backpressure on the
 queue
 > heading in to the RouteOnAttribute processor. The RouteOnAttribute
 processor
 > has a Penalty Duration and a Yield of 1 sec. The relationships all
 used the
 > PriorityAttributeAnalyzer as the prioritizer.
 >
 > The interesting thing is I had the exact same problem on two
 different nifis
 > (one was version 0.2.0 and one 0.3.0). The same type of flow files
 were
 > going from one to the other using site-to-site (I was doing this for
 load
 > balancing). On the other Nifi, instead of restarting, I added another
 > RouteOnAttribute processor and rerouted the stuck queue to the new one
 > instead, and that also got things moving again. (I have a
 backpressure since
 > the processor after the RouteOnAttribute is a ControlRate processor,
 and I
 > only want to allow 1 flowfile through per second so as not to
 overwhelm the
 > system downstream. The backpressure is to assist with load balancing,
 so if
 > one pathway fills up, I have the other pathway which uses the
 site-to-site
 > to route the flow to the other nifi).
 >
 > Is there some combination of backpressure, penalty or yield along with
 > prioritization that could cause a kind of deadlock-like situation?
 Any ideas
 > as to how I can prevent this from occurring?
 >
 > Thanks!
 > -Elli
 >

>>>
>>>
>>>
>>> --
>>> Sincerely,
>>> Chris Hamm
>>> (E) ceham...@gmail.com
>>> (Twitter) http://twitter.com/webhamm
>>> (Linkedin) http://www.linkedin.com/in/chrishamm
>>>
>>>
>>>
>

Re: Queue wasn't emptying

[Christopher Hamm]

yes and thank you
On Nov 12, 2015 8:30 AM, "Mark Payne"  wrote:

> Chris,
>
> If this is a test system (which I believe you said it is) and you're okay
> losing all data in the system, you can shutdown
> and delete your content and flowfile repositories (by default they are
> located at
> ./flowfile_repository and ./content_repository) and then restart.
>
> Thanks
> -Mark
>
>
> On Nov 12, 2015, at 8:25 AM, Christopher Hamm  wrote:
>
> that isn't clearing them out really. it gradually decreases but doesn't
> clear out 500k items.
> On Nov 12, 2015 8:22 AM, "Juan Sequeiros" 
> wrote:
>
>> To answer Christopher,
>>
>> You can clear out flowfiles on a queue by setting expiration on
>> relationship line.
>>
>> On Thu, Nov 12, 2015 at 7:53 AM, Christopher Hamm 
>> wrote:
>>
>>> I will look, but how do I just kill it with fire. What can I delete?
>>> On Nov 12, 2015 7:42 AM, "Mark Payne"  wrote:
>>>
 Chris / Elli,

 What version of NiFi are you running? I am wondering if you may have
 been bitten by
 this bug [1]. This should affect only the 'master' branch, not any
 formally released
 versions of NiFi. If you built the application from source code, rather
 than using an
 already-released version, this may well be what is happening.

 Thanks
 -Mark


 [1] https://issues.apache.org/jira/browse/NIFI-1155


 On Nov 12, 2015, at 2:32 AM, Christopher Hamm <
 em...@christopherhamm.com> wrote:

 where does it dump? I cant get mine to expire fast enough. It is dev
 test I want to blow away but cant delete.

 On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt  wrote:

> Elli,
>
> Can you share your flow configuration that we could possibly use to
> replicate this?  Perhaps turn this into a JIRA and attach that.
>
> Also anytime you see something that appears 'stuck' please try to get
> a stack dump (bin/nifi.sh dump)  As if there is truly a stuck thread
> we'll see it there and what it is blocked on.
>
> Thanks
> Joe
>
> On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz <
> eliezer_schw...@yahoo.com> wrote:
> > I had a queue that built up overnight with several thousand
> flowfiles. The
> > queue was pointing to a RouteOnAttribute processor, which was
> running. For
> > some reason, the RouteOnAttribute processor wasn't emptying the
> queue so the
> > queue just built up. Stopping and starting the processor didn't help.
> > However, simply restarting Nifi got the flow moving again. I can't
> figure
> > out why the queue was stuck.
> >
> > What could cause a queue to build up like that? How would restarting
> Nifi
> > get it going again? There weren't any errors in the log.  I did have
> > backpressure on the relationship to the processor after the
> RouteOnAttribute
> > set to 1, but that queue was empty. There was no backpressure on the
> queue
> > heading in to the RouteOnAttribute processor. The RouteOnAttribute
> processor
> > has a Penalty Duration and a Yield of 1 sec. The relationships all
> used the
> > PriorityAttributeAnalyzer as the prioritizer.
> >
> > The interesting thing is I had the exact same problem on two
> different nifis
> > (one was version 0.2.0 and one 0.3.0). The same type of flow files
> were
> > going from one to the other using site-to-site (I was doing this for
> load
> > balancing). On the other Nifi, instead of restarting, I added another
> > RouteOnAttribute processor and rerouted the stuck queue to the new
> one
> > instead, and that also got things moving again. (I have a
> backpressure since
> > the processor after the RouteOnAttribute is a ControlRate processor,
> and I
> > only want to allow 1 flowfile through per second so as not to
> overwhelm the
> > system downstream. The backpressure is to assist with load
> balancing, so if
> > one pathway fills up, I have the other pathway which uses the
> site-to-site
> > to route the flow to the other nifi).
> >
> > Is there some combination of backpressure, penalty or yield along
> with
> > prioritization that could cause a kind of deadlock-like situation?
> Any ideas
> > as to how I can prevent this from occurring?
> >
> > Thanks!
> > -Elli
> >
>



 --
 Sincerely,
 Chris Hamm
 (E) ceham...@gmail.com
 (Twitter) http://twitter.com/webhamm
 (Linkedin) http://www.linkedin.com/in/chrishamm



>>
>

Re: Queue wasn't emptying

[Mark Payne]

Chris,

If this is a test system (which I believe you said it is) and you're okay 
losing all data in the system, you can shutdown
and delete your content and flowfile repositories (by default they are located 
at
./flowfile_repository and ./content_repository) and then restart.

Thanks
-Mark


> On Nov 12, 2015, at 8:25 AM, Christopher Hamm  wrote:
> 
> that isn't clearing them out really. it gradually decreases but doesn't clear 
> out 500k items.
> 
> On Nov 12, 2015 8:22 AM, "Juan Sequeiros"  > wrote:
> To answer Christopher,
> You can clear out flowfiles on a queue by setting expiration on relationship 
> line.
> 
> 
> On Thu, Nov 12, 2015 at 7:53 AM, Christopher Hamm  > wrote:
> I will look, but how do I just kill it with fire. What can I delete?
> 
> On Nov 12, 2015 7:42 AM, "Mark Payne"  > wrote:
> Chris / Elli,
> 
> What version of NiFi are you running? I am wondering if you may have been 
> bitten by
> this bug [1]. This should affect only the 'master' branch, not any formally 
> released
> versions of NiFi. If you built the application from source code, rather than 
> using an
> already-released version, this may well be what is happening.
> 
> Thanks
> -Mark
> 
> 
> [1] https://issues.apache.org/jira/browse/NIFI-1155 
> 
> 
> 
>> On Nov 12, 2015, at 2:32 AM, Christopher Hamm > > wrote:
>> 
>> where does it dump? I cant get mine to expire fast enough. It is dev test I 
>> want to blow away but cant delete.
>> 
>> On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt > > wrote:
>> Elli,
>> 
>> Can you share your flow configuration that we could possibly use to
>> replicate this?  Perhaps turn this into a JIRA and attach that.
>> 
>> Also anytime you see something that appears 'stuck' please try to get
>> a stack dump (bin/nifi.sh dump)  As if there is truly a stuck thread
>> we'll see it there and what it is blocked on.
>> 
>> Thanks
>> Joe
>> 
>> On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz > > wrote:
>> > I had a queue that built up overnight with several thousand flowfiles. The
>> > queue was pointing to a RouteOnAttribute processor, which was running. For
>> > some reason, the RouteOnAttribute processor wasn't emptying the queue so 
>> > the
>> > queue just built up. Stopping and starting the processor didn't help.
>> > However, simply restarting Nifi got the flow moving again. I can't figure
>> > out why the queue was stuck.
>> >
>> > What could cause a queue to build up like that? How would restarting Nifi
>> > get it going again? There weren't any errors in the log.  I did have
>> > backpressure on the relationship to the processor after the 
>> > RouteOnAttribute
>> > set to 1, but that queue was empty. There was no backpressure on the queue
>> > heading in to the RouteOnAttribute processor. The RouteOnAttribute 
>> > processor
>> > has a Penalty Duration and a Yield of 1 sec. The relationships all used the
>> > PriorityAttributeAnalyzer as the prioritizer.
>> >
>> > The interesting thing is I had the exact same problem on two different 
>> > nifis
>> > (one was version 0.2.0 and one 0.3.0). The same type of flow files were
>> > going from one to the other using site-to-site (I was doing this for load
>> > balancing). On the other Nifi, instead of restarting, I added another
>> > RouteOnAttribute processor and rerouted the stuck queue to the new one
>> > instead, and that also got things moving again. (I have a backpressure 
>> > since
>> > the processor after the RouteOnAttribute is a ControlRate processor, and I
>> > only want to allow 1 flowfile through per second so as not to overwhelm the
>> > system downstream. The backpressure is to assist with load balancing, so if
>> > one pathway fills up, I have the other pathway which uses the site-to-site
>> > to route the flow to the other nifi).
>> >
>> > Is there some combination of backpressure, penalty or yield along with
>> > prioritization that could cause a kind of deadlock-like situation? Any 
>> > ideas
>> > as to how I can prevent this from occurring?
>> >
>> > Thanks!
>> > -Elli
>> >
>> 
>> 
>> 
>> -- 
>> Sincerely,
>> Chris Hamm
>> (E) ceham...@gmail.com 
>> (Twitter) http://twitter.com/webhamm 
>> (Linkedin) http://www.linkedin.com/in/chrishamm 
>> 
> 

Re: Queue wasn't emptying

[Elli Schwarz]

I had the issue with 0.2.0 and 0.3.0, and I did not build from source. It 
happens on rare occasions, and I can't figure out what causes it. I do have 
backpressure but no expiration on the route that RouteOnAttribute feeds. I 
haven't had a chance to try to dump. 

It seems to happen after a long period of time of the queue being full and 
backpressure being applied due to a large amount of flowfiles being added 
constantly.
 


 On Thursday, November 12, 2015 7:42 AM, Mark Payne  
wrote:
   
 

 Chris / Elli,
What version of NiFi are you running? I am wondering if you may have been 
bitten bythis bug [1]. This should affect only the 'master' branch, not any 
formally releasedversions of NiFi. If you built the application from source 
code, rather than using analready-released version, this may well be what is 
happening.
Thanks-Mark

[1] https://issues.apache.org/jira/browse/NIFI-1155


On Nov 12, 2015, at 2:32 AM, Christopher Hamm  wrote:
where does it dump? I cant get mine to expire fast enough. It is dev test I 
want to blow away but cant delete.
On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt  wrote:

Elli,

Can you share your flow configuration that we could possibly use to
replicate this?  Perhaps turn this into a JIRA and attach that.

Also anytime you see something that appears 'stuck' please try to get
a stack dump (bin/nifi.sh dump)  As if there is truly a stuck thread
we'll see it there and what it is blocked on.

Thanks
Joe

On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz  wrote:
> I had a queue that built up overnight with several thousand flowfiles. The
> queue was pointing to a RouteOnAttribute processor, which was running. For
> some reason, the RouteOnAttribute processor wasn't emptying the queue so the
> queue just built up. Stopping and starting the processor didn't help.
> However, simply restarting Nifi got the flow moving again. I can't figure
> out why the queue was stuck.
>
> What could cause a queue to build up like that? How would restarting Nifi
> get it going again? There weren't any errors in the log.  I did have
> backpressure on the relationship to the processor after the RouteOnAttribute
> set to 1, but that queue was empty. There was no backpressure on the queue
> heading in to the RouteOnAttribute processor. The RouteOnAttribute processor
> has a Penalty Duration and a Yield of 1 sec. The relationships all used the
> PriorityAttributeAnalyzer as the prioritizer.
>
> The interesting thing is I had the exact same problem on two different nifis
> (one was version 0.2.0 and one 0.3.0). The same type of flow files were
> going from one to the other using site-to-site (I was doing this for load
> balancing). On the other Nifi, instead of restarting, I added another
> RouteOnAttribute processor and rerouted the stuck queue to the new one
> instead, and that also got things moving again. (I have a backpressure since
> the processor after the RouteOnAttribute is a ControlRate processor, and I
> only want to allow 1 flowfile through per second so as not to overwhelm the
> system downstream. The backpressure is to assist with load balancing, so if
> one pathway fills up, I have the other pathway which uses the site-to-site
> to route the flow to the other nifi).
>
> Is there some combination of backpressure, penalty or yield along with
> prioritization that could cause a kind of deadlock-like situation? Any ideas
> as to how I can prevent this from occurring?
>
> Thanks!
> -Elli
>




-- 
Sincerely,
Chris Hamm
(E) ceham...@gmail.com
(Twitter) http://twitter.com/webhamm
(Linkedin) http://www.linkedin.com/in/chrishamm



 
  

Re: Queue wasn't emptying

[Joe Witt]

Elli,

What you may be seeing could be a different thing.  In 0.4.0 we've
improved who large queue handling works significantly.

Thanks
Joe

On Thu, Nov 12, 2015 at 8:56 AM, Elli Schwarz  wrote:
> I had the issue with 0.2.0 and 0.3.0, and I did not build from source. It
> happens on rare occasions, and I can't figure out what causes it. I do have
> backpressure but no expiration on the route that RouteOnAttribute feeds. I
> haven't had a chance to try to dump.
>
> It seems to happen after a long period of time of the queue being full and
> backpressure being applied due to a large amount of flowfiles being added
> constantly.
>
>
>
> On Thursday, November 12, 2015 7:42 AM, Mark Payne 
> wrote:
>
>
>
> Chris / Elli,
>
> What version of NiFi are you running? I am wondering if you may have been
> bitten by
> this bug [1]. This should affect only the 'master' branch, not any formally
> released
> versions of NiFi. If you built the application from source code, rather than
> using an
> already-released version, this may well be what is happening.
>
> Thanks
> -Mark
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-1155
>
>
> On Nov 12, 2015, at 2:32 AM, Christopher Hamm 
> wrote:
>
> where does it dump? I cant get mine to expire fast enough. It is dev test I
> want to blow away but cant delete.
>
> On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt  wrote:
>
> Elli,
>
> Can you share your flow configuration that we could possibly use to
> replicate this?  Perhaps turn this into a JIRA and attach that.
>
> Also anytime you see something that appears 'stuck' please try to get
> a stack dump (bin/nifi.sh dump)  As if there is truly a stuck thread
> we'll see it there and what it is blocked on.
>
> Thanks
> Joe
>
> On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz 
> wrote:
>> I had a queue that built up overnight with several thousand flowfiles. The
>> queue was pointing to a RouteOnAttribute processor, which was running. For
>> some reason, the RouteOnAttribute processor wasn't emptying the queue so
>> the
>> queue just built up. Stopping and starting the processor didn't help.
>> However, simply restarting Nifi got the flow moving again. I can't figure
>> out why the queue was stuck.
>>
>> What could cause a queue to build up like that? How would restarting Nifi
>> get it going again? There weren't any errors in the log.  I did have
>> backpressure on the relationship to the processor after the
>> RouteOnAttribute
>> set to 1, but that queue was empty. There was no backpressure on the queue
>> heading in to the RouteOnAttribute processor. The RouteOnAttribute
>> processor
>> has a Penalty Duration and a Yield of 1 sec. The relationships all used
>> the
>> PriorityAttributeAnalyzer as the prioritizer.
>>
>> The interesting thing is I had the exact same problem on two different
>> nifis
>> (one was version 0.2.0 and one 0.3.0). The same type of flow files were
>> going from one to the other using site-to-site (I was doing this for load
>> balancing). On the other Nifi, instead of restarting, I added another
>> RouteOnAttribute processor and rerouted the stuck queue to the new one
>> instead, and that also got things moving again. (I have a backpressure
>> since
>> the processor after the RouteOnAttribute is a ControlRate processor, and I
>> only want to allow 1 flowfile through per second so as not to overwhelm
>> the
>> system downstream. The backpressure is to assist with load balancing, so
>> if
>> one pathway fills up, I have the other pathway which uses the site-to-site
>> to route the flow to the other nifi).
>>
>> Is there some combination of backpressure, penalty or yield along with
>> prioritization that could cause a kind of deadlock-like situation? Any
>> ideas
>> as to how I can prevent this from occurring?
>>
>> Thanks!
>> -Elli
>>
>
>
>
>
> --
> Sincerely,
> Chris Hamm
> (E) ceham...@gmail.com
> (Twitter) http://twitter.com/webhamm
> (Linkedin) http://www.linkedin.com/in/chrishamm
>
>
>
>

Re: Queue wasn't emptying

[Joe Witt]

Elli,

Can you share your flow configuration that we could possibly use to
replicate this?  Perhaps turn this into a JIRA and attach that.

Also anytime you see something that appears 'stuck' please try to get
a stack dump (bin/nifi.sh dump)  As if there is truly a stuck thread
we'll see it there and what it is blocked on.

Thanks
Joe

On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz  wrote:
> I had a queue that built up overnight with several thousand flowfiles. The
> queue was pointing to a RouteOnAttribute processor, which was running. For
> some reason, the RouteOnAttribute processor wasn't emptying the queue so the
> queue just built up. Stopping and starting the processor didn't help.
> However, simply restarting Nifi got the flow moving again. I can't figure
> out why the queue was stuck.
>
> What could cause a queue to build up like that? How would restarting Nifi
> get it going again? There weren't any errors in the log.  I did have
> backpressure on the relationship to the processor after the RouteOnAttribute
> set to 1, but that queue was empty. There was no backpressure on the queue
> heading in to the RouteOnAttribute processor. The RouteOnAttribute processor
> has a Penalty Duration and a Yield of 1 sec. The relationships all used the
> PriorityAttributeAnalyzer as the prioritizer.
>
> The interesting thing is I had the exact same problem on two different nifis
> (one was version 0.2.0 and one 0.3.0). The same type of flow files were
> going from one to the other using site-to-site (I was doing this for load
> balancing). On the other Nifi, instead of restarting, I added another
> RouteOnAttribute processor and rerouted the stuck queue to the new one
> instead, and that also got things moving again. (I have a backpressure since
> the processor after the RouteOnAttribute is a ControlRate processor, and I
> only want to allow 1 flowfile through per second so as not to overwhelm the
> system downstream. The backpressure is to assist with load balancing, so if
> one pathway fills up, I have the other pathway which uses the site-to-site
> to route the flow to the other nifi).
>
> Is there some combination of backpressure, penalty or yield along with
> prioritization that could cause a kind of deadlock-like situation? Any ideas
> as to how I can prevent this from occurring?
>
> Thanks!
> -Elli
>