Looping over curl with bash would work too. On Wed, Mar 18, 2020 at 3:31 PM Midhun Mohan <midhun.mo...@esginc.us> wrote:
> Thanks for that suggestion will look into it as well > > On Thu, 19 Mar 2020 at 00:59, Mike Thomsen <mikerthom...@gmail.com> wrote: > > > Quad core MacBook Pro that's about 3 years ago. So laptop grade i7 w/ > 16GB > > of RAM and Xmx was the default of 512MB. > > > > Easy way to debug this is to write a python script to hit the endpoint > > repeatedly. Something based on this: > > > > import json > > import requests > > > > requests.post(url, json = json_data_var) > > > > And then wrap it like this: > > > > $> time python benchmark.py > > > > On Wed, Mar 18, 2020 at 9:29 AM Midhun Mohan <midhun.mo...@esginc.us> > > wrote: > > > > > Hey Mike , I meant like when I try posting that many records it is > taking > > > bit time. Just checking did you tweak around the thread count and what > is > > > your instance size. CPU and RAM > > > > > > On Wed, 18 Mar 2020 at 18:57, Mike Thomsen <mikerthom...@gmail.com> > > wrote: > > > > > > > By setting the HTTP verb to POST in InvokeHTTP. > > > > > > > > On Wed, Mar 18, 2020 at 1:29 AM Midhun Mohan <midhun.mo...@esginc.us > > > > > > wrote: > > > > > > > > > How did you post 50k flowfiles, that is what am looking for > > > > > > > > > > On Wed, 18 Mar 2020 at 02:36, Mike Thomsen <mikerthom...@gmail.com > > > > > > wrote: > > > > > > > > > > > That's probably a lot of the issue, especially if it's an evented > > > > service > > > > > > like a Node service running with one thread. I just did a simple > > test > > > > by > > > > > > posting 50k flowfiles with 4k of JSON in them to an Express hello > > > world > > > > > app > > > > > > and it was able to respond to 50k flowfiles in under 10s using > > > > InvokeHttp > > > > > > with only 3 threads. All on my MacBook Pro. > > > > > > > > > > > > On Tue, Mar 17, 2020 at 4:54 PM Midhun Mohan < > > midhun.mo...@esginc.us > > > > > > > > > > wrote: > > > > > > > > > > > > > Endpoint does execute db query in a transaction, yes it is in > the > > > > same > > > > > > > setup but different cluster > > > > > > > > > > > > > > On Wed, 18 Mar 2020, 2:20 am Mike Thomsen, < > > mikerthom...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > How is the endpoint implemented and what does it do? Also, is > > it > > > > > > located > > > > > > > in > > > > > > > > the same data center as the EC2 instance running NiFi? > > > > > > > > > > > > > > > > On Tue, Mar 17, 2020 at 3:55 PM Midhun Mohan < > > > > midhun.mo...@esginc.us > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Yeah the endpoint which am sending right now has plenty of > > > > > resources > > > > > > > > > available. Only thing is I need to send more records > > > > > > > > > > > > > > > > > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, < > > > czobri...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > I have not test throughput of InvokeHTTP so am not sure > > what > > > > the > > > > > > > > maximum > > > > > > > > > > is, but can give some general guidance. > > > > > > > > > > > > > > > > > > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to > > > > figure > > > > > > out > > > > > > > > > system > > > > > > > > > > resources usage while your flow is running. > > > > > > > > > > > > > > > > > > > > You should be able to increase both your nifi count by > more > > > and > > > > > > > adjust > > > > > > > > > your > > > > > > > > > > processors until you are limited by your system > resources. > > > > > > > > > > > > > > > > > > > > Have you verified the endpoint you are sending to is not > > the > > > > > bottle > > > > > > > > neck? > > > > > > > > > > > > > > > > > > > > Chad > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan < > > > > > > midhun.mo...@esginc.us > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Size of file around kb not more than that, > > > > > > > > > > > How can I find the bottle neck, > > > > > > > > > > > > > > > > > > > > > > Yes I adjust the count to 11 then all other processors > > > > stopped > > > > > > but > > > > > > > > > > > processing improved. > > > > > > > > > > > Totally at present 6 is showing not more than that. > > > > > > > > > > > > > > > > > > > > > > I just need a way to post more records to endpoint to > > make > > > it > > > > > > > > realtime > > > > > > > > > > > > > > > > > > > > > > Hope i was able to give more details > > > > > > > > > > > > > > > > > > > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, < > > > > > czobri...@gmail.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Midhun, > > > > > > > > > > > > A little more information would help. > > > > > > > > > > > > > > > > > > > > > > > > What size files are you sending? > > > > > > > > > > > > Have you looked at resource usage to see what the > > > > bottleneck > > > > > > is? > > > > > > > > > > > > Did you adjust your nifi system thread count from the > > > > > hamburger > > > > > > > > menu? > > > > > > > > > > > > How many threads are running total for nifi? It's the > > > > number > > > > > in > > > > > > > the > > > > > > > > > top > > > > > > > > > > > > left of the screen. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Chad > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan < > > > > > > > > midhun.mo...@esginc.us > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Do anyone have better idea on this > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, < > > > > > > > > midhun.mo...@esginc.us > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have a queue which will be filling up with > > realtime > > > > > > records > > > > > > > > of > > > > > > > > > > > around > > > > > > > > > > > > > > 20k records. Which is posting to an endpoint > using > > > > > > Invokehttp > > > > > > > > > > > > processor. > > > > > > > > > > > > > > > > > > > > > > > > > > > > currently the average throughput is for 10k > records > > > it > > > > > > takes > > > > > > > > > around > > > > > > > > > > > 20 > > > > > > > > > > > > > > minutes to complete the invokehttp processor > > posting. > > > > > > > > > > > > > > I Increased the concurrent thread to larger > number > > > > which > > > > > > is > > > > > > > > > > > increasing > > > > > > > > > > > > > > the throughput, but it is causing other processor > > to > > > > > > > > terminate. I > > > > > > > > > > > think > > > > > > > > > > > > > > that can be solved by running on larger > instances. > > > > > > > > > > > > > > > > > > > > > > > > > > > > What all things I need to take care to make sure > I > > > get > > > > a > > > > > > > real > > > > > > > > > time > > > > > > > > > > > > > > posting on endpoint. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I need 20k records to be processed as they > receive > > it > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks in advance > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > Midhun Mohan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > Regards, > > > > > Midhun Mohan > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Regards, > > > Midhun Mohan > > > > > > > > -- > > > Regards, > Midhun Mohan >
