Probably can't answer how much it would increase processing speed, but processor thread count is limited by the nifi system thread count.
Normally you can increase the nifi system thread count a decent amount, 100+, depending on the system. On Tue, Mar 17, 2020 at 4:19 PM Midhun Mohan <midhun.mo...@esginc.us> wrote: > Yeah got it thank you. > How will Changing that values affect the processing speed > > On Wed, 18 Mar 2020, 1:45 am Chad Zobrisky, <czobri...@gmail.com> wrote: > > > Hamburger menu in the top right -> Controller Settings -> under the > general > > tab -> Maximum Timer Driven Thread Count. > > > > On Tue, Mar 17, 2020 at 4:11 PM Midhun Mohan <midhun.mo...@esginc.us> > > wrote: > > > > > No overall thread pool I haven't changed. How can I change that? > > > > > > Nifi is running on a standalone aws instance. T2.medium I remember so > > that > > > leaves us with 2 cores I guess > > > > > > On Wed, 18 Mar 2020, 1:29 am Pierre Villard, < > > pierre.villard...@gmail.com> > > > wrote: > > > > > > > Did you change the size of the overall thread pool at NiFi level? If > > yes, > > > > to which number? How many cores do you have on NiFi? > > > > > > > > Thanks > > > > > > > > Le mar. 17 mars 2020 à 20:55, Midhun Mohan <midhun.mo...@esginc.us> > a > > > > écrit : > > > > > > > > > Yeah the endpoint which am sending right now has plenty of > resources > > > > > available. Only thing is I need to send more records > > > > > > > > > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <czobri...@gmail.com> > > > wrote: > > > > > > > > > > > I have not test throughput of InvokeHTTP so am not sure what the > > > > maximum > > > > > > is, but can give some general guidance. > > > > > > > > > > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to figure > > out > > > > > system > > > > > > resources usage while your flow is running. > > > > > > > > > > > > You should be able to increase both your nifi count by more and > > > adjust > > > > > your > > > > > > processors until you are limited by your system resources. > > > > > > > > > > > > Have you verified the endpoint you are sending to is not the > bottle > > > > neck? > > > > > > > > > > > > Chad > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan < > > midhun.mo...@esginc.us > > > > > > > > > > wrote: > > > > > > > > > > > > > Size of file around kb not more than that, > > > > > > > How can I find the bottle neck, > > > > > > > > > > > > > > Yes I adjust the count to 11 then all other processors stopped > > but > > > > > > > processing improved. > > > > > > > Totally at present 6 is showing not more than that. > > > > > > > > > > > > > > I just need a way to post more records to endpoint to make it > > > > realtime > > > > > > > > > > > > > > Hope i was able to give more details > > > > > > > > > > > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, < > czobri...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > Midhun, > > > > > > > > A little more information would help. > > > > > > > > > > > > > > > > What size files are you sending? > > > > > > > > Have you looked at resource usage to see what the bottleneck > > is? > > > > > > > > Did you adjust your nifi system thread count from the > hamburger > > > > menu? > > > > > > > > How many threads are running total for nifi? It's the number > in > > > the > > > > > top > > > > > > > > left of the screen. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Chad > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan < > > > > midhun.mo...@esginc.us > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Do anyone have better idea on this > > > > > > > > > > > > > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, < > > > > midhun.mo...@esginc.us > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > I have a queue which will be filling up with realtime > > records > > > > of > > > > > > > around > > > > > > > > > > 20k records. Which is posting to an endpoint using > > Invokehttp > > > > > > > > processor. > > > > > > > > > > > > > > > > > > > > currently the average throughput is for 10k records it > > takes > > > > > around > > > > > > > 20 > > > > > > > > > > minutes to complete the invokehttp processor posting. > > > > > > > > > > I Increased the concurrent thread to larger number which > > is > > > > > > > increasing > > > > > > > > > > the throughput, but it is causing other processor to > > > > terminate. I > > > > > > > think > > > > > > > > > > that can be solved by running on larger instances. > > > > > > > > > > > > > > > > > > > > What all things I need to take care to make sure I get a > > > real > > > > > time > > > > > > > > > > posting on endpoint. > > > > > > > > > > > > > > > > > > > > I need 20k records to be processed as they receive it > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks in advance > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > Midhun Mohan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
