Phil, > Basically I have two custom processors that are slow (data intensive) that > create backlogs
Perhaps I missed it, but you didn't say whether the processors are functionally I/O-bound or CPU-bound. Also, the nature of the output is important. How many flowfiles are you creating, what are their average sizes, etc. Thanks, Mike On Sun, Sep 20, 2020 at 11:02 PM Phil H <[email protected]> wrote: > > Hi there, > > The issue I have isn’t my processor, it’s the fact I can’t get Nifi to run > more threads. I’ve noticed that if I bump the max thread counts in the > controller settings to total over 256, then nifi as a whole locks up and I > have to clear the state and restart it. > > Thanks, > Phil > > On Mon, 21 Sep 2020 at 11:01, Mark Payne <[email protected]> wrote: > > > Phil, > > > > > > > > It sounds like you’re dealing with a large amount of lock contention. > > Either using explicit java.util.concurrency.Lock or synchronized blocks. > > Would review the code for lock contention or run a profiler against nifi > > and see what that shows. > > > > > > > > Sent from my iPhone > > > > > > > > > On Sep 20, 2020, at 8:30 PM, Phil H <[email protected]> wrote: > > > > > > > > > > Various numbers - max of 96 so far. > > > > > > > > > >> On Sat, 19 Sep 2020 at 21:44, Pierre Villard < > > [email protected]> > > > > >> wrote: > > > > >> > > > > >> How many concurrent tasks did you set on the two custom processors? > > > > >> > > > > >> > > > > >> > > > > >> Le sam. 19 sept. 2020 à 02:16, Phil H <[email protected]> a écrit : > > > > >> > > > > >> > > > > >> > > > > >>> Basically I have two custom processors that are slow (data intensive) > > > > >> that > > > > >> > > > > >>> create backlogs. All other aspects of the flow work well - I am running > > > > >> SSD > > > > >> > > > > >>> disks so that sort of I/O is not biting me. The process itself is > > highly > > > > >> > > > > >>> ‘parallelisable’ but I can’t get NiFi to use more than about 10% of the > > > > >> CPU > > > > >> > > > > >>> load. This is the only purpose this machine has - I want it to work > > much > > > > >> > > > > >>> harder! > > > > >> > > > > >>> > > > > >> > > > > >>> On Fri, 18 Sep 2020 at 20:56, Pierre Villard < > > > > >> [email protected]> > > > > >> > > > > >>> wrote: > > > > >> > > > > >>> > > > > >> > > > > >>>> Hi, > > > > >> > > > > >>>> > > > > >> > > > > >>>> > > > > >> > > > > >>>> > > > > >> > > > > >>>> Do you have backpressure with flow files accumulating somewhere in > > your > > > > >> > > > > >>>> > > > > >> > > > > >>>> flow? If not, then you probably don't need more threads. > > > > >> > > > > >>>> > > > > >> > > > > >>>> If you do find bottlenecks in your flow design, then you can look at > > > > >> > > > > >>>> > > > > >> > > > > >>>> increasing the concurrent tasks on specific processors to have > > > > >> > > > > >>>> > > > > >> > > > > >>>> multi-threading. Note that it's not always the solution. > > > > >> > > > > >>>> > > > > >> > > > > >>>> Changing the pool size does not mean NiFi will use more resources. > > > > >> > > > > >>>> > > > > >> > > > > >>>> Actually, I usually recommend changing the pool size only when you see > > > > >> > > > > >>> that > > > > >> > > > > >>>> > > > > >> > > > > >>>> all the threads available are used. > > > > >> > > > > >>>> > > > > >> > > > > >>>> Overall we'd need more details about your flow and observations. > > > > >> > > > > >>>> > > > > >> > > > > >>>> > > > > >> > > > > >>>> > > > > >> > > > > >>>> Thanks, > > > > >> > > > > >>>> > > > > >> > > > > >>>> Pierre > > > > >> > > > > >>>> > > > > >> > > > > >>>> > > > > >> > > > > >>>> > > > > >> > > > > >>>> Le ven. 18 sept. 2020 à 03:49, Phil H <[email protected]> a écrit : > > > > >> > > > > >>>> > > > > >> > > > > >>>> > > > > >> > > > > >>>> > > > > >> > > > > >>>>> Hi there, > > > > >> > > > > >>>> > > > > >> > > > > >>>>> > > > > >> > > > > >>>> > > > > >> > > > > >>>>> Regardless of what I set in the maximum thread count settings, I > > > > >> never > > > > >> > > > > >>>> see > > > > >> > > > > >>>> > > > > >> > > > > >>>>> more than 128 active threads. The CPU load of the system is only 5-8% > > > > >> > > > > >>>> when > > > > >> > > > > >>>> > > > > >> > > > > >>>>> NiFi is running at this rate. It’s a Solaris system with 64 cores. > > > > >> > > > > >>>> > > > > >> > > > > >>>>> > > > > >> > > > > >>>> > > > > >> > > > > >>>>> Any help getting more performance would be appreciated! > > > > >> > > > > >>>> > > > > >> > > > > >>>>> > > > > >> > > > > >>>> > > > > >> > > > > >>>>> Regards, > > > > >> > > > > >>>> > > > > >> > > > > >>>>> Phil > > > > >> > > > > >>>> > > > > >> > > > > >>>>> > > > > >> > > > > >>>> > > > > >> > > > > >>>> > > > > >> > > > > >>> > > > > >> > > > > >> > > > >
