Do you have some other client calling that proves your API endpoint isn’t the bottleneck? Sorry if I missed that part and you already provided that.
On March 18, 2020 at 09:29:27, Midhun Mohan (midhun.mo...@esginc.us) wrote: Hey Mike , I meant like when I try posting that many records it is taking bit time. Just checking did you tweak around the thread count and what is your instance size. CPU and RAM On Wed, 18 Mar 2020 at 18:57, Mike Thomsen <mikerthom...@gmail.com> wrote: > By setting the HTTP verb to POST in InvokeHTTP. > > On Wed, Mar 18, 2020 at 1:29 AM Midhun Mohan <midhun.mo...@esginc.us> > wrote: > > > How did you post 50k flowfiles, that is what am looking for > > > > On Wed, 18 Mar 2020 at 02:36, Mike Thomsen <mikerthom...@gmail.com> > wrote: > > > > > That's probably a lot of the issue, especially if it's an evented > service > > > like a Node service running with one thread. I just did a simple test > by > > > posting 50k flowfiles with 4k of JSON in them to an Express hello world > > app > > > and it was able to respond to 50k flowfiles in under 10s using > InvokeHttp > > > with only 3 threads. All on my MacBook Pro. > > > > > > On Tue, Mar 17, 2020 at 4:54 PM Midhun Mohan <midhun.mo...@esginc.us> > > > wrote: > > > > > > > Endpoint does execute db query in a transaction, yes it is in the > same > > > > setup but different cluster > > > > > > > > On Wed, 18 Mar 2020, 2:20 am Mike Thomsen, <mikerthom...@gmail.com> > > > wrote: > > > > > > > > > How is the endpoint implemented and what does it do? Also, is it > > > located > > > > in > > > > > the same data center as the EC2 instance running NiFi? > > > > > > > > > > On Tue, Mar 17, 2020 at 3:55 PM Midhun Mohan < > midhun.mo...@esginc.us > > > > > > > > wrote: > > > > > > > > > > > Yeah the endpoint which am sending right now has plenty of > > resources > > > > > > available. Only thing is I need to send more records > > > > > > > > > > > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <czobri...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > I have not test throughput of InvokeHTTP so am not sure what > the > > > > > maximum > > > > > > > is, but can give some general guidance. > > > > > > > > > > > > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to > figure > > > out > > > > > > system > > > > > > > resources usage while your flow is running. > > > > > > > > > > > > > > You should be able to increase both your nifi count by more and > > > > adjust > > > > > > your > > > > > > > processors until you are limited by your system resources. > > > > > > > > > > > > > > Have you verified the endpoint you are sending to is not the > > bottle > > > > > neck? > > > > > > > > > > > > > > Chad > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan < > > > midhun.mo...@esginc.us > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Size of file around kb not more than that, > > > > > > > > How can I find the bottle neck, > > > > > > > > > > > > > > > > Yes I adjust the count to 11 then all other processors > stopped > > > but > > > > > > > > processing improved. > > > > > > > > Totally at present 6 is showing not more than that. > > > > > > > > > > > > > > > > I just need a way to post more records to endpoint to make it > > > > > realtime > > > > > > > > > > > > > > > > Hope i was able to give more details > > > > > > > > > > > > > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, < > > czobri...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Midhun, > > > > > > > > > A little more information would help. > > > > > > > > > > > > > > > > > > What size files are you sending? > > > > > > > > > Have you looked at resource usage to see what the > bottleneck > > > is? > > > > > > > > > Did you adjust your nifi system thread count from the > > hamburger > > > > > menu? > > > > > > > > > How many threads are running total for nifi? It's the > number > > in > > > > the > > > > > > top > > > > > > > > > left of the screen. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Chad > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan < > > > > > midhun.mo...@esginc.us > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Do anyone have better idea on this > > > > > > > > > > > > > > > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, < > > > > > midhun.mo...@esginc.us > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > I have a queue which will be filling up with realtime > > > records > > > > > of > > > > > > > > around > > > > > > > > > > > 20k records. Which is posting to an endpoint using > > > Invokehttp > > > > > > > > > processor. > > > > > > > > > > > > > > > > > > > > > > currently the average throughput is for 10k records it > > > takes > > > > > > around > > > > > > > > 20 > > > > > > > > > > > minutes to complete the invokehttp processor posting. > > > > > > > > > > > I Increased the concurrent thread to larger number > which > > > is > > > > > > > > increasing > > > > > > > > > > > the throughput, but it is causing other processor to > > > > > terminate. I > > > > > > > > think > > > > > > > > > > > that can be solved by running on larger instances. > > > > > > > > > > > > > > > > > > > > > > What all things I need to take care to make sure I get > a > > > > real > > > > > > time > > > > > > > > > > > posting on endpoint. > > > > > > > > > > > > > > > > > > > > > > I need 20k records to be processed as they receive it > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks in advance > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > Midhun Mohan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Regards, > > Midhun Mohan > > > -- Regards, Midhun Mohan
