Peter, I see your point that it feels not natural or at least surprising. There are two challenges I see with what you propose. One is user oriented and the other is technical.
The user oriented one is that penalized objects are penalized as a function of the thing that last operated on them. The further away we let the data get the harder it would be to reason over why they were penalized in the first place. The technical one is that once something is penalized and placed into the queue there is prioritization and polling logic that kicks in as a factor. I'm not sure how we'd tweak it for that to be ok in some cases and in others not. Perhaps we could just make funnels truly a pass-through and when calculating the queue we're storing on figure out the first non-funnel queue provided there is no cloning/branching we'd have to account for. But even then it brings us back to the previous point which is the user challenge of knowing what thing penalized objects in queue in the first place. Alternatively, we should review whether it is obvious enough (or at all) that items within a queue at a given moment in time are penalized. I've worked with NiFi for a very long time and i'll be honest and state I've forgotten that penalization was a thing more than a few times too. What do you think? Thanks On Mon, Oct 9, 2017 at 9:01 PM, Peter Wicks (pwicks) <pwi...@micron.com> wrote: > Bryan, > > Yes, it was the penalty causing the issue. This feels like weird behavior for > Funnel’s, and I’m not sure if it makes sense for penalties to work this way. > > Would it make more sense if penalties were generally kept as is, but not > applied at Funnel’s, then the penalty would kick back in at the first > non-funnel queue? > > Thanks, > Peter > > From: Bryan Bende [mailto:bbe...@gmail.com] > Sent: Monday, October 09, 2017 7:33 PM > To: dev@nifi.apache.org > Subject: [EXT] Re: Funnel Queue Slowness > > Peter, > > The images didn’t come across for me, but since you mentioned that a failure > queue is involved, is it possible all the flow files going to failure are > being penalized which would cause them to not be processed immediately? > > -Bryan > > > On Oct 8, 2017, at 10:49 PM, Peter Wicks (pwicks) > <pwi...@micron.com<mailto:pwi...@micron.com>> wrote: > > I’ve been running into an issue on 1.4.0 where my Funnel sometimes runs slow. > I haven’t been able to create a nice reproducible test case to pass on. > What I’m seeing is that my failure queue on the right will start to fill up, > even though there is plenty of room for them in the next queue. You can see > that the Tasks/Time is fairly low, only 24 in the last 5 minutes (first > image), so it’s not that the FlowFile’s are moving so fast that they just > appear to be in queue. > > If I stop the downstream processor the files slowly trickle through the > funnel into the next queue slowly. I had an Oldest FlowFile First prioritizer > on the downstream queue. I tried removing it but there was no change in > behavior. > One time where I saw this behavior in the past was when my NiFi instance was > thread starved, but there are plenty of threads available on the instance and > all other processors are running fine. I also don’t understand why it > trickles the FlowFile’s in, from what I’ve seen in the code Funnel grabs > large batches at one time… > > Thoughts? > > (Sometimes my images don’t make it, let me know if that happens.) > [cid:image002.png@01D340EC.543FE750] [cid:image004.png@01D340EC.543FE750] >
