[ https://issues.apache.org/jira/browse/NIFI-3897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16061573#comment-16061573 ]
Joseph Percivall commented on NIFI-3897: ---------------------------------------- I believe I hit this state. I debugged a bit with [~markap14] and I'll let him comment on what he's going to do but we believe it to be related to un-caught throwables (ie. out of memory errors) but not 100% sure it only arises under those conditions. Below I'll outline a summary of what the problem was, what I tried and what fixed it: NiFi 1.3.0 standalone instance Apparent problem: * An RPG is routed to a funnel and FlowFiles are not being processed by the funnel from that queue. The RPG continues to grab more FlowFiles Other symptoms: * The funnel will still process flowfiles coming from another connection * There are 2 swap files on disk for that queue (logs for the creation of them have aged off) * There is a distinct cliff of flowfiles getting processed and then stopping * There was a swap for that queue a half second prior to the cliff but is swapped back in a couple seconds later * An out of memory error log is present in the bootstrap logs for a couple days prior to the cliff Actions Tried: * List queue says there are 0 flowfiles in the queue * Moving the connection to another component. Then moving it back. Also tried starting the other component. Neither worked. * Setting the expiration had no effect * Back-pressure works as expected. * Sent in a large group of flowfiles (larger than the swap size). Once it hit the next swap size it flipped and started processing those. It only processed down to 2 swap files worth and then got into the same state. Remedy: * Restarted NiFi and it successfully processed them > Queue with flow files thinks it is empty > ---------------------------------------- > > Key: NIFI-3897 > URL: https://issues.apache.org/jira/browse/NIFI-3897 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework > Affects Versions: 1.2.0 > Reporter: Joseph Gresock > Attachments: queue-has-no-flowfiles.png > > > A particular queue is listed as having 80,000 flow files, but NiFi thinks > there are none to list. I was originally inclined to believe that the queue > is indeed empty, and that there is somehow a pointer to 80,000 flow files > leftover on the queue. > Unfortunately, the logs have rolled over so I can't see what error or warning > messages may have happened at the time, and I can't reproduce it reliably. > Update: I happened to reboot my cluster, and suddenly the 80,000 flow files > went through the flow. This makes the problem considerably worse, since it > appears the flow files were "invisible" until the reboot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)