Re: Empty Queue - but UI shows messages

[Mark Payne]

Joe,

The problem is that NiFi is not able to load balance the data. What errors do 
you see in the logs around load balancing?

Thanks
-Mark

On Nov 22, 2022, at 9:15 AM, Joe Obernberger 
mailto:joseph.obernber...@gmail.com>> wrote:


Hi Joe - this is happening only in queues that do round robin load balancing:



When I list queue on that queue it shows no flow files.  If I stop the 
processors, and change the queue to no longer load balance, then I can not only 
list the queue, but the 22k messages are then processed.

I did the dump as suggested - and it nifi-bootstrap.log file is attached.

-Joe


On 11/9/2022 5:09 PM, Joe Witt wrote:
Please get back to the bad state and then do a thread dump.  Please share

./bin/nifi.sh dump

Only if LB appears stuck.
Thanks

On Wed, Nov 9, 2022 at 3:05 PM Joe Obernberger 
mailto:joseph.obernber...@gmail.com>> wrote:

If I stop the RouteOnContent and InvokeHTTP processors, and then change the 
queue to no longer load balance, then the messages appear and the processor 
start to run.  I can also then list the messages in the queue.

-Joe

On 11/9/2022 12:04 PM, Joe Obernberger wrote:

The processor consuming from the queue runs on all nodes and is a InokeHTTP 
processor.  That processor is currently idle.

I can stop/start InvokeHTTP, but the queue size remains.  I can't list what is 
in the queue.  This seems like a bug?  What I'm confused about is - do I have 
data to process or not?  I can empty the queue, and something is removed, but I 
can't see what...so I don't know if I lost messages or not.

<6aueDc0O2l8gSHvp.png>

-Joe

On 11/9/2022 11:50 AM, Joe Witt wrote:
Hello

This likely means the processor consuming from this queue has the flowfiles 
held being processed.  MergeContent is a common processor that would do this 
but others certainly can.  What processor do you have there?

If you stop the target processor then delete it should always work though you 
generally should not need to do so but normally deleting queue content is a 
debug thing so you can.

Thanks

On Wed, Nov 9, 2022 at 9:47 AM Joe Obernberger 
mailto:joseph.obernber...@gmail.com>> wrote:

Hi - I'm using NiFi 1.18.0 in a three node cluster using internal zookeeper.  
Occasionally, I will see a queue showing queued messages, but when I list the 
queue, the UI says that there are no flow files.  Also the consumer of this 
queue seems to be idle.  I can then empty the queue and it reports that n 
messages were deleted.

Any idea what is happening here?






-Joe


[https://s-install.avcdn.net/ipm/preview/icons/icon-envelope-tick-green-avg-v1.png]
 
Virus-free.www.avg.com

Re: Empty Queue - but UI shows messages

[Mark Payne]

Joe,

Given that the icon shows the data is still being load balanced, I am guessing 
that the system is having trouble load balancing the data. Can you check the 
logs to see if there are errors related to load balancing?

Thanks
-Mark


On Nov 9, 2022, at 12:04 PM, Joe Obernberger 
mailto:joseph.obernber...@gmail.com>> wrote:


The processor consuming from the queue runs on all nodes and is a InokeHTTP 
processor.  That processor is currently idle.

I can stop/start InvokeHTTP, but the queue size remains.  I can't list what is 
in the queue.  This seems like a bug?  What I'm confused about is - do I have 
data to process or not?  I can empty the queue, and something is removed, but I 
can't see what...so I don't know if I lost messages or not.

<6aueDc0O2l8gSHvp.png>

-Joe

On 11/9/2022 11:50 AM, Joe Witt wrote:
Hello

This likely means the processor consuming from this queue has the flowfiles 
held being processed.  MergeContent is a common processor that would do this 
but others certainly can.  What processor do you have there?

If you stop the target processor then delete it should always work though you 
generally should not need to do so but normally deleting queue content is a 
debug thing so you can.

Thanks

On Wed, Nov 9, 2022 at 9:47 AM Joe Obernberger 
mailto:joseph.obernber...@gmail.com>> wrote:

Hi - I'm using NiFi 1.18.0 in a three node cluster using internal zookeeper.  
Occasionally, I will see a queue showing queued messages, but when I list the 
queue, the UI says that there are no flow files.  Also the consumer of this 
queue seems to be idle.  I can then empty the queue and it reports that n 
messages were deleted.

Any idea what is happening here?






-Joe


[https://s-install.avcdn.net/ipm/preview/icons/icon-envelope-tick-green-avg-v1.png]
 
Virus-free.www.avg.com

Re: Empty Queue - but UI shows messages

[Mark Payne]

Just to add to what Joe said, in the screenshot the queue that you’re listing 
shows that blue load balancing icon, meaning that it is actively load 
balancing. The listing will not include any FlowFiles that are queued up to 
move to another node. So on a connection like that where it’s actively load 
balancing you may not be able to view the contents.

Thanks
-Mark


On Nov 9, 2022, at 11:50 AM, Joe Witt 
mailto:joe.w...@gmail.com>> wrote:

Hello

This likely means the processor consuming from this queue has the flowfiles 
held being processed.  MergeContent is a common processor that would do this 
but others certainly can.  What processor do you have there?

If you stop the target processor then delete it should always work though you 
generally should not need to do so but normally deleting queue content is a 
debug thing so you can.

Thanks

On Wed, Nov 9, 2022 at 9:47 AM Joe Obernberger 
mailto:joseph.obernber...@gmail.com>> wrote:

Hi - I'm using NiFi 1.18.0 in a three node cluster using internal zookeeper.  
Occasionally, I will see a queue showing queued messages, but when I list the 
queue, the UI says that there are no flow files.  Also the consumer of this 
queue seems to be idle.  I can then empty the queue and it reports that n 
messages were deleted.

Any idea what is happening here?






-Joe


[https://s-install.avcdn.net/ipm/preview/icons/icon-envelope-tick-green-avg-v1.png]
 
Virus-free.www.avg.com

Re: Empty Queue - but UI shows messages

[Joe Witt]

Hello

This likely means the processor consuming from this queue has the flowfiles
held being processed.  MergeContent is a common processor that would do
this but others certainly can.  What processor do you have there?

If you stop the target processor then delete it should always work though
you generally should not need to do so but normally deleting queue content
is a debug thing so you can.

Thanks

On Wed, Nov 9, 2022 at 9:47 AM Joe Obernberger 
wrote:

> Hi - I'm using NiFi 1.18.0 in a three node cluster using internal
> zookeeper.  Occasionally, I will see a queue showing queued messages, but
> when I list the queue, the UI says that there are no flow files.  Also the
> consumer of this queue seems to be idle.  I can then empty the queue and it
> reports that n messages were deleted.
>
> Any idea what is happening here?
>
>
> -Joe
>
>
>
> 
> Virus-free.www.avg.com
> 
> <#m_5948377795112736779_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>

Empty Queue - but UI shows messages

[Joe Obernberger]

Hi - I'm using NiFi 1.18.0 in a three node cluster using internal 
zookeeper.  Occasionally, I will see a queue showing queued messages, 
but when I list the queue, the UI says that there are no flow files.  
Also the consumer of this queue seems to be idle.  I can then empty the 
queue and it reports that n messages were deleted.


Any idea what is happening here?


-Joe



--
This email has been checked for viruses by AVG antivirus software.
www.avg.com