Hi Venkat, The drop of the performance of LB compared to GW Framework seems to be way too much. I think we can't afford to lose nearly 50% of throughput because of the LB components. Let's try to identify the bottlenecks related to LB code.
On Tue, Aug 9, 2016 at 4:18 PM, Venkat Raman <vraman2...@gmail.com> wrote: > Hi Isuru & Kasun, > > Please find the attached results document. As discussed, I created a new > VM for bench-marking. > > It seems like TPS of 20,000 (from yesterday's results) even at higher > concurrency level is not accurate. Sorry for the confusion caused. > Most of the time endpoints were marked as unHealthy and direct error > response is returned by LB Mediator which resulted in high TPS. I tried > multiple bench-marking from yesterday and I was never able to achieve that > result. > > In this test, to avoid such cases, higher no of unHealthyRetries count has > been configured. > > Also, I've bench-marked performance of GW-FMW using i-server with this > simple > <https://github.com/Venkat2811/product-http-load-balancer/blob/master/performance-benchmark/gw-framework/router.iflow> > configuration. It is a simple route even without if-else conditions. > > As you can see, It is performing 2X faster than LB. > > Next steps would be to do memory benchmark and plot graphs with these > values. Once repo, documentation and blog is ready, I'll be using JFR to > identify bottle necks and on fine-tuning LB's performance. > > Will be looking forward to hear your feedback on this. > > > > > > *Thanks,* > *Venkat.* > > On Tue, Aug 9, 2016 at 1:13 AM, Venkat Raman <vraman2...@gmail.com> wrote: > >> Sure Kasun, I'll do a perf-benchmark between iserver and LB in a new VM >> as discussed. >> >> Thanks, >> Venkat. >> On Aug 9, 2016 12:55 AM, "Kasun Indrasiri" <ka...@wso2.com> wrote: >> >>> >>> - Compare GW framework perf vs LB (need to identify if any perf impact >>> from the LB related code). >>> - Identify the reason for the apparent perf bottleneck with high >>> concurrency. >>> >>> On Mon, Aug 8, 2016 at 10:55 AM, Venkat Raman <vraman2...@gmail.com> >>> wrote: >>> >>>> Hi Kasun, >>>> >>>> Please find the latest results after Saturday's code review. >>>> >>>> >>>> >>>> >>>> *Thanks,* >>>> *Venkat.* >>>> >>>> On Mon, Aug 8, 2016 at 10:06 AM, Venkat Raman <vraman2...@gmail.com> >>>> wrote: >>>> >>>>> Hi Isuru, >>>>> >>>>> Good morning. Please find 11th week's progress. >>>>> >>>>> 1) Had code reviews and made few suggested corrections. >>>>> 2) Did some groundwork for using JFR >>>>> >>>>> Will be continuing to work on performance tuning. >>>>> >>>>> @Kasun - Tomorrow is August 9th. Can we have demo ? >>>>> >>>>> >>>>> >>>>> *Thanks,* >>>>> *Venkat.* >>>>> >>>>> On Sat, Aug 6, 2016 at 10:16 PM, Venkat Raman <vraman2...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Isuru, >>>>>> >>>>>> Here are the findings from today's review: >>>>>> >>>>>> 1) Change CallMediatorMap from ConcurrentHashMap to HashMap >>>>>> 2) Remove unnecessary Synchronized block while checking >>>>>> areAllEndpointsUnhealthy() >>>>>> 3) Rename LoadBalancerCallMediator to LBEndpointsCallMediator >>>>>> 4) Give a PR by adding getUri() method to gateway-framework >>>>>> 5) Use JavaFlightRecorder while doing benchmark to identify >>>>>> bottlenecks >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *Venkat.* >>>>>> >>>>>> On Fri, Aug 5, 2016 at 1:43 PM, Venkat Raman <vraman2...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Isuru, >>>>>>> >>>>>>> Please find the attached latest bench-mark without synchronization, >>>>>>> callBackpool, healthcheck. >>>>>>> >>>>>>> Throughput is just 1000 times faster than my current >>>>>>> implementation. >>>>>>> >>>>>>> It is drastically falling because of some other reason. >>>>>>> >>>>>>> >>>>>>> *Thanks,* >>>>>>> *Venkat.* >>>>>>> >>>>>>> On Fri, Aug 5, 2016 at 10:09 AM, Venkat Raman <vraman2...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi IsuruU, >>>>>>>> >>>>>>>> FYI >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *Thanks,* >>>>>>>> *Venkat.* >>>>>>>> >>>>>>>> ---------- Forwarded message ---------- >>>>>>>> From: Venkat Raman <vraman2...@gmail.com> >>>>>>>> Date: Thu, Aug 4, 2016 at 10:39 AM >>>>>>>> Subject: Re: GSoC Project: HTTP Load Balancer on Top of WSO2 >>>>>>>> Gateway Discussion >>>>>>>> To: Isuru Ranawaka <isu...@wso2.com>, Kasun Indrasiri < >>>>>>>> ka...@wso2.com> >>>>>>>> Cc: DEV <dev@wso2.org>, Senduran Balasubramaniyam < >>>>>>>> sendu...@wso2.com> >>>>>>>> >>>>>>>> >>>>>>>> Hi Isuru & Kasun, >>>>>>>> >>>>>>>> Please find the attached result document >>>>>>>> (raw-engine-transport.xlsx). I've done test with raw-engine-transport >>>>>>>> without any BE. It is performing great and is close to Netty based >>>>>>>> BE !! >>>>>>>> >>>>>>>> Problem is with LB only. >>>>>>>> >>>>>>>> My guess is that CallbackPool (using concurrent HashMap) that we >>>>>>>> are using to determine timeout is the bottle neck. I'll disable >>>>>>>> Callback >>>>>>>> pool and do bench-mark and update you on that. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *Thanks,* >>>>>>>> *Venkat.* >>>>>>>> >>>>>>>> On Thu, Aug 4, 2016 at 9:48 AM, Venkat Raman <vraman2...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Okay Isuru. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *Thanks,* >>>>>>>>> *Venkat.* >>>>>>>>> >>>>>>>>> On Thu, Aug 4, 2016 at 9:42 AM, Isuru Ranawaka <isu...@wso2.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi venkat, >>>>>>>>>> >>>>>>>>>> Yes we can. lets have a call today around 9.30 p.m >>>>>>>>>> >>>>>>>>>> On Thu, Aug 4, 2016 at 9:34 AM, Venkat Raman < >>>>>>>>>> vraman2...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Isuru, >>>>>>>>>>> >>>>>>>>>>> Good morning. Yesterday night, I spoke with Kasun regarding the >>>>>>>>>>> latest update on bench-mark results. Even without any locking >>>>>>>>>>> performance >>>>>>>>>>> is not good after concurrency of 5000. >>>>>>>>>>> >>>>>>>>>>> As you have done bench-mark till concurrency of 3000, we both >>>>>>>>>>> would like to do bench-marking on raw carbon-transport upto >>>>>>>>>>> concurrency of >>>>>>>>>>> 10,000 and 1,00,000 requests so that we get an idea on this. >>>>>>>>>>> >>>>>>>>>>> How do we do that ? Will a simple response from engine suffice >>>>>>>>>>> ? Can I use LB to send simple response directly without doing any >>>>>>>>>>> mediation ? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *Thanks,* >>>>>>>>>>> *Venkat.* >>>>>>>>>>> >>>>>>>>>>> On Thu, Aug 4, 2016 at 12:19 AM, Venkat Raman < >>>>>>>>>>> vraman2...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Isuru, >>>>>>>>>>>> >>>>>>>>>>>> Please find the attached bench-mark results. As discussed, >>>>>>>>>>>> I've disabled health-checking and removed synchronized block and >>>>>>>>>>>> used >>>>>>>>>>>> atomic Integer in one test and also did a test without any kind of >>>>>>>>>>>> lock or >>>>>>>>>>>> use of atomic integers. >>>>>>>>>>>> >>>>>>>>>>>> Throughput and latency results are positive. But, after >>>>>>>>>>>> concurrency level of 5000 it is not that good. So even If we use >>>>>>>>>>>> read-write lock or stamped lock, we will get performance little >>>>>>>>>>>> performance >>>>>>>>>>>> gain only. >>>>>>>>>>>> >>>>>>>>>>>> I feel that If we can do bench-mark with integration-server >>>>>>>>>>>> upto 10000 concurrent connections we'll get a better idea. Is >>>>>>>>>>>> that okay ? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *Thanks,* >>>>>>>>>>>> *Venkat.* >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Aug 2, 2016 at 9:39 PM, Venkat Raman < >>>>>>>>>>>> vraman2...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Isuru & Kasun, >>>>>>>>>>>>> >>>>>>>>>>>>> Please find the findings from today's code review. >>>>>>>>>>>>> >>>>>>>>>>>>> 1) Locking in getNextLBOutboundEndpoint() method in algorithm >>>>>>>>>>>>> implementation is causing over-head. We have to find a way to >>>>>>>>>>>>> efficiently >>>>>>>>>>>>> handle communication between threads to reduce locking overhead. >>>>>>>>>>>>> >>>>>>>>>>>>> 2) Code repo freeze by August 15th for the sake of GSoC. If >>>>>>>>>>>>> we can find a way to overcome locking over-head before August >>>>>>>>>>>>> 15th that >>>>>>>>>>>>> changes will be added to code repo. Otherwise it will be added >>>>>>>>>>>>> after GSoC. >>>>>>>>>>>>> >>>>>>>>>>>>> 3) TPS, Latency and Memory graphs to be added. >>>>>>>>>>>>> >>>>>>>>>>>>> 4) Blog post and PDF documentation. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> *Thanks,* >>>>>>>>>>>>> *Venkat.* >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Aug 1, 2016 at 9:25 AM, Venkat Raman < >>>>>>>>>>>>> vraman2...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Isuru, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Good morning. Please find 10th week's progress. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1) Had discussion with Kasun. >>>>>>>>>>>>>> 2) As suggested, did performance bench-marking using Netty >>>>>>>>>>>>>> BE, and it turns out that our LB is beating Nginx till >>>>>>>>>>>>>> concurrency level of >>>>>>>>>>>>>> 6000 after which it is not performing well. >>>>>>>>>>>>>> I've attached the results. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I've started a new thread as Conversation arrangement is not >>>>>>>>>>>>>> good in previous one. >>>>>>>>>>>>>> >>>>>>>>>>>>>> It would be great if we can have a code review Isuru. Based >>>>>>>>>>>>>> on your feedback I'll be abe to make changes and we can do >>>>>>>>>>>>>> bench-marking >>>>>>>>>>>>>> again. Can we do it today 9:30 PM ? We have only 2 full weeks >>>>>>>>>>>>>> more. The >>>>>>>>>>>>>> last week will be for documentation. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> *Thanks,* >>>>>>>>>>>>>> *Venkat.* >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Best Regards >>>>>>>>>> Isuru Ranawaka >>>>>>>>>> M: +94714629880 >>>>>>>>>> Blog : http://isurur.blogspot.com/ >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>> >>> -- >>> Kasun Indrasiri >>> Director, Integration Technologies >>> WSO2, Inc.; http://wso2.com >>> lean.enterprise.middleware >>> >>> cell: +1 650 450 2293 >>> Blog : http://kasunpanorama.blogspot.com/ >>> >> > -- Kasun Indrasiri Director, Integration Technologies WSO2, Inc.; http://wso2.com lean.enterprise.middleware cell: +1 650 450 2293 Blog : http://kasunpanorama.blogspot.com/
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
