Phil, It sounds like you’re dealing with a large amount of lock contention. Either using explicit java.util.concurrency.Lock or synchronized blocks. Would review the code for lock contention or run a profiler against nifi and see what that shows.
Sent from my iPhone > On Sep 20, 2020, at 8:30 PM, Phil H <[email protected]> wrote: > > Various numbers - max of 96 so far. > >> On Sat, 19 Sep 2020 at 21:44, Pierre Villard <[email protected]> >> wrote: >> >> How many concurrent tasks did you set on the two custom processors? >> >> >> >> Le sam. 19 sept. 2020 à 02:16, Phil H <[email protected]> a écrit : >> >> >> >>> Basically I have two custom processors that are slow (data intensive) >> that >> >>> create backlogs. All other aspects of the flow work well - I am running >> SSD >> >>> disks so that sort of I/O is not biting me. The process itself is highly >> >>> ‘parallelisable’ but I can’t get NiFi to use more than about 10% of the >> CPU >> >>> load. This is the only purpose this machine has - I want it to work much >> >>> harder! >> >>> >> >>> On Fri, 18 Sep 2020 at 20:56, Pierre Villard < >> [email protected]> >> >>> wrote: >> >>> >> >>>> Hi, >> >>>> >> >>>> >> >>>> >> >>>> Do you have backpressure with flow files accumulating somewhere in your >> >>>> >> >>>> flow? If not, then you probably don't need more threads. >> >>>> >> >>>> If you do find bottlenecks in your flow design, then you can look at >> >>>> >> >>>> increasing the concurrent tasks on specific processors to have >> >>>> >> >>>> multi-threading. Note that it's not always the solution. >> >>>> >> >>>> Changing the pool size does not mean NiFi will use more resources. >> >>>> >> >>>> Actually, I usually recommend changing the pool size only when you see >> >>> that >> >>>> >> >>>> all the threads available are used. >> >>>> >> >>>> Overall we'd need more details about your flow and observations. >> >>>> >> >>>> >> >>>> >> >>>> Thanks, >> >>>> >> >>>> Pierre >> >>>> >> >>>> >> >>>> >> >>>> Le ven. 18 sept. 2020 à 03:49, Phil H <[email protected]> a écrit : >> >>>> >> >>>> >> >>>> >> >>>>> Hi there, >> >>>> >> >>>>> >> >>>> >> >>>>> Regardless of what I set in the maximum thread count settings, I >> never >> >>>> see >> >>>> >> >>>>> more than 128 active threads. The CPU load of the system is only 5-8% >> >>>> when >> >>>> >> >>>>> NiFi is running at this rate. It’s a Solaris system with 64 cores. >> >>>> >> >>>>> >> >>>> >> >>>>> Any help getting more performance would be appreciated! >> >>>> >> >>>>> >> >>>> >> >>>>> Regards, >> >>>> >> >>>>> Phil >> >>>> >> >>>>> >> >>>> >> >>>> >> >>> >> >>
