Our initial assumptions were wrong and we were able to narrow down this issue to a wrong configuration parameter name. Due to that, we were always running with a single disruptor event handler thread instead of a thread pool. Now we are seeing better perf values with disruptor as opposed to the Netty executor thread pool, with some tuning of disruptor configuration parameters.
Azeez On Mon, Mar 14, 2016 at 1:55 PM, Sagara Gunathunga <sag...@wso2.com> wrote: > > > On Mon, Mar 14, 2016 at 1:53 PM, Srinath Perera <srin...@wso2.com> wrote: > >> I talked to Ranawaka and Isuru in detail. >> >> Disrupter helps a lot when tasks are CPU bound. In such cases, in can >> works with very few threads and reduce the overhead of communication >> between threads. >> >> However, when threads block for I/O this advantage is reduced a lot. In >> those cases, we need to have multiple disrupter workers ( batch >> processors). We are doing that. >> >> However, doing test with 500ms sleep is not fair IMO. Sleep often waits >> more than the given value. With 200 threads, it can only do 400TPS with >> 500ms sleep. I think we should compare against a DB backend. >> >> Shell we test disrupter and java work pool model against a DB backend? >> > > +1 I also think we should use more realistic backend such as DB. > > Thanks ! > >> >> --Srinath >> >> >> >> On Mon, Mar 14, 2016 at 10:26 AM, Kasun Indrasiri <ka...@wso2.com> wrote: >> >>> Hi, >>> >>> Let me try to clarify few things here. >>> >>> - Initially we implemented Netty HTTP transport with conventional thread >>> model (workers) and at the same time we also tested the Disruptor based >>> model for the same Gateway/Header based routing use case. Disruptor based >>> approach gave us around ~20k of perf gain on perf testing environments. >>> - MSF4J 1.0 GA didn't use GW's HTTP transport code as it is. It reused >>> basic transport runtime but with a custom Netty handler that is used to >>> dispatch the requests. So, MSFJ 1.0 GA, didn't have disruptor and >>> performance/latency of MSF4J 1.0 GA has nothing to do with Disruprtor. >>> >>> - Now we are trying to migrate MSF4J into the exact same transport code >>> as it with the GW core (carbon-transport's HTTP transport). And that's >>> where we came across the above perf issue. >>> - So, unlike GW scenario, for MSF4J and even for any other content-aware >>> ESB scenario, the above approach is not the optimum it seems. Hence we are >>> now looking into how such scenarios are handled with Disruptor. >>> >>> In that context, if we look at the original LMAX use case[1] is also >>> quite similar to what we are trying with content aware scenarios. In their >>> use case they had heavy CPU intensive components(such as Business Logic >>> component) as well as IO bound components (such as Un-marshaller, >>> Replicator). And they get better performance for the same use case with >>> Disruptor over a conventional worker-thread model. >>> >>> >>> [image: Inline image 1] >>> >>> So, we need to have a close look into that how we can implement a >>> dependent consumer scenario[2] (one consumer is IO bound and the other is >>> CPU bound) and check whether we can get more perf gain compared to the >>> conventional worker thread model. >>> >>> Ranawaka is current looking into this. >>> >>> [1] http://martinfowler.com/articles/lmax.html >>> [2] >>> http://mechanitis.blogspot.com/2011/07/dissecting-disruptor-wiring-up.html >>> >>> On Sun, Mar 13, 2016 at 8:11 AM, Isuru Ranawaka <isu...@wso2.com> wrote: >>> >>>> Hi Azeez, >>>> >>>> In GW Disruptor threads are not used for make calls for backends. >>>> Backends are called by Netty worker pool and those calls are non blocking >>>> calls. So if backend responds after a delay it won't be a problem. >>>> >>>> >>>> In MSF4J if it includes IO operations or delayed operations then it >>>> causes problems because processing happens through Disruptor threads and >>>> by occupying all the limited disruptor threads. But this should be solved >>>> by operating Disruptor Event Handlers through workerpool and now we are >>>> looking in to that why it does not provide expected results. >>>> >>>> thanks >>>> IsuruR >>>> >>>> On Sat, Mar 12, 2016 at 6:46 PM, Afkham Azeez <az...@wso2.com> wrote: >>>> >>>>> Kasun et al, >>>>> In MSF4J, the threads from the disruptor pool itself are used for >>>>> processing in the MSF4J operation. In the case of the GW passthrough & HBR >>>>> scenarios, are those disruptor threads used to make the calls to the >>>>> actual >>>>> backends? Is that a blocking call? What if the backend service is changed >>>>> so that it responds after a delay rather than instantaneously? >>>>> >>>>> On Sat, Mar 12, 2016 at 6:21 PM, Afkham Azeez <az...@wso2.com> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Sat, Mar 12, 2016 at 1:40 PM, Sanjiva Weerawarana < >>>>>> sanj...@wso2.com> wrote: >>>>>> >>>>>>> On Thu, Mar 10, 2016 at 6:20 PM, Sagara Gunathunga <sag...@wso2.com> >>>>>>> wrote: >>>>>>> >>>>>>>> On Thu, Mar 10, 2016 at 10:26 AM, Afkham Azeez <az...@wso2.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> No from day 1, we have decided that GW & MSF4J will use the same >>>>>>>>> Netty transport component so that the config file will be the same as >>>>>>>>> well >>>>>>>>> as improvements made to that transport will be automatically >>>>>>>>> available for >>>>>>>>> both products. So now at least for MSF4J, we have issues in using the >>>>>>>>> Netty >>>>>>>>> transport in its current state, so we have to fix those issues. >>>>>>>>> >>>>>>>> >>>>>>>> Reuse of same config files and components provide an advantage to >>>>>>>> us as the F/W developers/maintainers but my question was what are the >>>>>>>> benefits grant to end users of MSF4J through Carbon transport ? >>>>>>>> >>>>>>> >>>>>>> We are writing MSF4J and the rest of the platform. Not someone else. >>>>>>> As such we have to keep them consistent. >>>>>>> >>>>>>> For end users our target has to be to give the best performance >>>>>>> possible. >>>>>>> >>>>>>> >>>>>>>> I don't think we can compromise performance numbers for a reason >>>>>>>> that is more important for F/W maintainers than end users, IMHO if we >>>>>>>> continue to use Carbon transport at least it should perform as same >>>>>>>> level >>>>>>>> as vanilla Netty. >>>>>>>> >>>>>>> >>>>>>> There's no reason why that cannot be the case. >>>>>>> >>>>>>> Can't we keep Disruptor while improve performance of Carbon >>>>>>>> transport ? >>>>>>>> >>>>>>> >>>>>>> Disruptor is a technique to make things more performant not less >>>>>>> performant. We have to figure out what's wrong and fix it - not throw >>>>>>> the >>>>>>> baby out with the bathwater. >>>>>>> >>>>>> >>>>>> Yes, we are in the process of trying to figure out why disruptor as >>>>>> opposed to the Netty executor threadpool gives better performance for the >>>>>> gateway (dispatching to a zero delay backend), while for an MSF4J service >>>>>> which has a sleep in it, it is the other way around. >>>>>> >>>>>> >>>>>>> >>>>>>> Sanjiva. >>>>>>> -- >>>>>>> Sanjiva Weerawarana, Ph.D. >>>>>>> Founder, CEO & Chief Architect; WSO2, Inc.; http://wso2.com/ >>>>>>> email: sanj...@wso2.com; office: (+1 650 745 4499 | +94 11 214 >>>>>>> 5345) x5700; cell: +94 77 787 6880 | +1 408 466 5099; voip: +1 650 >>>>>>> 265 8311 >>>>>>> blog: http://sanjiva.weerawarana.org/; twitter: @sanjiva >>>>>>> Lean . Enterprise . Middleware >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> Architecture@wso2.org >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Afkham Azeez* >>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com >>>>>> Member; Apache Software Foundation; http://www.apache.org/ >>>>>> * <http://www.apache.org/>* >>>>>> *email: **az...@wso2.com* <az...@wso2.com> >>>>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: * >>>>>> *http://blog.afkham.org* <http://blog.afkham.org> >>>>>> *twitter: **http://twitter.com/afkham_azeez* >>>>>> <http://twitter.com/afkham_azeez> >>>>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez >>>>>> <http://lk.linkedin.com/in/afkhamazeez>* >>>>>> >>>>>> *Lean . Enterprise . Middleware* >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Afkham Azeez* >>>>> Director of Architecture; WSO2, Inc.; http://wso2.com >>>>> Member; Apache Software Foundation; http://www.apache.org/ >>>>> * <http://www.apache.org/>* >>>>> *email: **az...@wso2.com* <az...@wso2.com> >>>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: * >>>>> *http://blog.afkham.org* <http://blog.afkham.org> >>>>> *twitter: **http://twitter.com/afkham_azeez* >>>>> <http://twitter.com/afkham_azeez> >>>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez >>>>> <http://lk.linkedin.com/in/afkhamazeez>* >>>>> >>>>> *Lean . Enterprise . Middleware* >>>>> >>>> >>>> >>>> >>>> -- >>>> Best Regards >>>> Isuru Ranawaka >>>> M: +94714629880 >>>> Blog : http://isurur.blogspot.com/ >>>> >>> >>> >>> >>> -- >>> Kasun Indrasiri >>> Software Architect >>> WSO2, Inc.; http://wso2.com >>> lean.enterprise.middleware >>> >>> cell: +94 77 556 5206 >>> Blog : http://kasunpanorama.blogspot.com/ >>> >> >> >> >> -- >> ============================ >> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >> Site: http://people.apache.org/~hemapani/ >> Photos: http://www.flickr.com/photos/hemapani/ >> Phone: 0772360902 >> > > > > -- > Sagara Gunathunga > > Architect; WSO2, Inc.; http://wso2.com > V.P Apache Web Services; http://ws.apache.org/ > Linkedin; http://www.linkedin.com/in/ssagara > Blog ; http://ssagara.blogspot.com > > -- *Afkham Azeez* Director of Architecture; WSO2, Inc.; http://wso2.com Member; Apache Software Foundation; http://www.apache.org/ * <http://www.apache.org/>* *email: **az...@wso2.com* <az...@wso2.com> * cell: +94 77 3320919blog: **http://blog.afkham.org* <http://blog.afkham.org> *twitter: **http://twitter.com/afkham_azeez* <http://twitter.com/afkham_azeez> *linked-in: **http://lk.linkedin.com/in/afkhamazeez <http://lk.linkedin.com/in/afkhamazeez>* *Lean . Enterprise . Middleware*
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
