Basically I have two custom processors that are slow (data intensive) that create backlogs. All other aspects of the flow work well - I am running SSD disks so that sort of I/O is not biting me. The process itself is highly ‘parallelisable’ but I can’t get NiFi to use more than about 10% of the CPU load. This is the only purpose this machine has - I want it to work much harder!
On Fri, 18 Sep 2020 at 20:56, Pierre Villard <[email protected]> wrote: > Hi, > > > > Do you have backpressure with flow files accumulating somewhere in your > > flow? If not, then you probably don't need more threads. > > If you do find bottlenecks in your flow design, then you can look at > > increasing the concurrent tasks on specific processors to have > > multi-threading. Note that it's not always the solution. > > Changing the pool size does not mean NiFi will use more resources. > > Actually, I usually recommend changing the pool size only when you see that > > all the threads available are used. > > Overall we'd need more details about your flow and observations. > > > > Thanks, > > Pierre > > > > Le ven. 18 sept. 2020 à 03:49, Phil H <[email protected]> a écrit : > > > > > Hi there, > > > > > > Regardless of what I set in the maximum thread count settings, I never > see > > > more than 128 active threads. The CPU load of the system is only 5-8% > when > > > NiFi is running at this rate. It’s a Solaris system with 64 cores. > > > > > > Any help getting more performance would be appreciated! > > > > > > Regards, > > > Phil > > > > >
