Hi Phil, A few things that might be of help here.
It's been a while since I looked at the call chain for a processor but....... I believe what happens is that when a processor is set to have a number of concurrent processors set what happens is that per nifi node the locks around the onTrigger method is set equal to that number, thus this doesn't create any additional instances of a processor. The implication of this being that if you have any instance variables in your custom processor that are synchronized or protected by locks in some way then this may cause a backlog. Now if you have any libraries that use a singleton pattern or has resources that may also have instance locks (or syncronised) internally this may also cause a backlog while each of the instances race and block to get hold of that lock. The other thing maybe is that, it could be a setting in either in the JVM, or unix limit subsystem that may be blocking nifi from using more than 10% of the CPU.e Edward On Mon, Sep 21, 2020 at 1:06 PM Mike Thomsen <[email protected]> wrote: > Phil, > > > Basically I have two custom processors that are slow (data intensive) > that create backlogs > > Perhaps I missed it, but you didn't say whether the processors are > functionally I/O-bound or CPU-bound. Also, the nature of the output is > important. How many flowfiles are you creating, what are their average > sizes, etc. > > Thanks, > > Mike > > On Sun, Sep 20, 2020 at 11:02 PM Phil H <[email protected]> wrote: > > > > Hi there, > > > > The issue I have isn’t my processor, it’s the fact I can’t get Nifi to > run > > more threads. I’ve noticed that if I bump the max thread counts in the > > controller settings to total over 256, then nifi as a whole locks up and > I > > have to clear the state and restart it. > > > > Thanks, > > Phil > > > > On Mon, 21 Sep 2020 at 11:01, Mark Payne <[email protected]> wrote: > > > > > Phil, > > > > > > > > > > > > It sounds like you’re dealing with a large amount of lock contention. > > > Either using explicit java.util.concurrency.Lock or synchronized > blocks. > > > Would review the code for lock contention or run a profiler against > nifi > > > and see what that shows. > > > > > > > > > > > > Sent from my iPhone > > > > > > > > > > > > > On Sep 20, 2020, at 8:30 PM, Phil H <[email protected]> wrote: > > > > > > > > > > > > > > Various numbers - max of 96 so far. > > > > > > > > > > > > > >> On Sat, 19 Sep 2020 at 21:44, Pierre Villard < > > > [email protected]> > > > > > > >> wrote: > > > > > > >> > > > > > > >> How many concurrent tasks did you set on the two custom processors? > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> Le sam. 19 sept. 2020 à 02:16, Phil H <[email protected]> a > écrit : > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >>> Basically I have two custom processors that are slow (data > intensive) > > > > > > >> that > > > > > > >> > > > > > > >>> create backlogs. All other aspects of the flow work well - I am > running > > > > > > >> SSD > > > > > > >> > > > > > > >>> disks so that sort of I/O is not biting me. The process itself is > > > highly > > > > > > >> > > > > > > >>> ‘parallelisable’ but I can’t get NiFi to use more than about 10% > of the > > > > > > >> CPU > > > > > > >> > > > > > > >>> load. This is the only purpose this machine has - I want it to work > > > much > > > > > > >> > > > > > > >>> harder! > > > > > > >> > > > > > > >>> > > > > > > >> > > > > > > >>> On Fri, 18 Sep 2020 at 20:56, Pierre Villard < > > > > > > >> [email protected]> > > > > > > >> > > > > > > >>> wrote: > > > > > > >> > > > > > > >>> > > > > > > >> > > > > > > >>>> Hi, > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> Do you have backpressure with flow files accumulating somewhere in > > > your > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> flow? If not, then you probably don't need more threads. > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> If you do find bottlenecks in your flow design, then you can look > at > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> increasing the concurrent tasks on specific processors to have > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> multi-threading. Note that it's not always the solution. > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> Changing the pool size does not mean NiFi will use more resources. > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> Actually, I usually recommend changing the pool size only when > you see > > > > > > >> > > > > > > >>> that > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> all the threads available are used. > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> Overall we'd need more details about your flow and observations. > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> Thanks, > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> Pierre > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> Le ven. 18 sept. 2020 à 03:49, Phil H <[email protected]> a > écrit : > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>>> Hi there, > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>>> > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>>> Regardless of what I set in the maximum thread count settings, I > > > > > > >> never > > > > > > >> > > > > > > >>>> see > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>>> more than 128 active threads. The CPU load of the system is only > 5-8% > > > > > > >> > > > > > > >>>> when > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>>> NiFi is running at this rate. It’s a Solaris system with 64 > cores. > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>>> > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>>> Any help getting more performance would be appreciated! > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>>> > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>>> Regards, > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>>> Phil > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>>> > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>>> > > > > > > >> > > > > > > >>> > > > > > > >> > > > > > > >> > > > > > > >
