> > > > * This is with 3 ESB nodes publishing data to DAS. So the above graph > shows the overall TPS for all 3 ESB nodes. The peak to start with is the > initial TPS we set. Then in decreases gradually and converges to some low > amount. > > @ESB Team: Would like to know is this an expected behavior? > This is not exepected bahviour. Chanaka, shall we do a long running test?
> > > Thanks, > Supun > > > On Wed, Apr 20, 2016 at 12:51 PM, Sriskandarajah Suhothayan <s...@wso2.com > > wrote: > >> +1 for the approach. Lets test it and see. >> >> Regards >> Suho >> >> On Wed, Apr 20, 2016 at 12:30 PM, Anjana Fernando <anj...@wso2.com> >> wrote: >> >>> Hi, >>> >>> Good progress Supun! .. do keep pushing the parameters to find the >>> limits we can go to. >>> >>> @Suho, the idea was to all together eliminate the batch script and just >>> store/index the data for later lookup, and do the computation purely in >>> Siddhi. I don't think we will get a big scaling problem, since the data >>> needs to be stored in-memory when we go to upper layers of summarization is >>> smaller, and stops at yearly granularity. So it would be at that time, we >>> having data in-memory for last years worth of data, in a way of last 12 >>> records of summary data for 12 months for a specific artifact, last day's >>> worth, that is 30 entries etc.. so growing of data slows immensely, and >>> also it has a upper limit, which I guess should be comfortability within >>> usual memory capacity. >>> >>> So if we can get a proper checkpoint and replay mechanism figured out >>> for data processed, we can do all the things in CEP, then we just don't >>> have the complexity of maintaining two mechanism of doing the processing. >>> >>> Cheers, >>> Anjana. >>> >>> On Wed, Apr 20, 2016 at 12:11 PM, Sriskandarajah Suhothayan < >>> s...@wso2.com> wrote: >>> >>>> I think it will make more sense to run seconds and minutes from siddhi, >>>> and run the spark every hour, when there are lots of date on the system >>>> this will be much more scalable. >>>> >>>> WDYT? >>>> >>>> Regards >>>> Suho >>>> >>>> On Wed, Apr 20, 2016 at 11:50 AM, Supun Sethunga <sup...@wso2.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> This is a follow-up mail of [1], to give an update on the status with >>>>> the performance issue [2] . So as mentioned in the previous mail, with >>>>> Spark-script doing the summary stat generation as a batch process, creates >>>>> a bottleneck at a higher TPS. More precisely, with our findings, it cannot >>>>> handle a throughput of more than 30 TPS as a batch process. (i.e: events >>>>> published to DAS within 10 mins with a TPS of 30, take more than 10 mins >>>>> to >>>>> process. Means, if we schedule a script every 10 mins, the events to be >>>>> processed grows over time). >>>>> >>>>> To overcome this, thought of doing the summarizing up to a certain >>>>> extent (upto second-wise summary) using siddhi, and to generate remaining >>>>> stats (per-minute/hour/day/month), using spark. With this enhancement, ran >>>>> some load tests locally to evaluate this approach, and the results are as >>>>> follows. >>>>> >>>>> Backend DB : MySQL >>>>> ESB analytics nodes: 1 >>>>> >>>>> With InnoDB >>>>> >>>>> - With *80 TPS*: (script scheduled every 1 min) : Avg time taken >>>>> for completion of the script = ~ *20 sec*. >>>>> - With* 500 TPS* (script scheduled every 2 min) : Avg time taken >>>>> for completion of the script = ~ *45 sec*. >>>>> >>>>> >>>>> With MyISAM >>>>> >>>>> - With *80 TPS* (script scheduled every 1 min) : Avg time taken >>>>> for completion of the script = ~ *24 sec*. >>>>> - With *80 TPS *(script scheduled every 2 min) : Avg time taken >>>>> for completion of the script = ~ *20 sec*. >>>>> - With *500 TPS* (script scheduled every 2 min) : Avg time taken >>>>> for completion of the script = ~ *35 sec*. >>>>> >>>>> As a further improvement, we would be trying out to do summarizing >>>>> upto minute/hour level (eventually do all the summarizing using siddhi). >>>>> >>>>> [1] [Dev] ESB Analytics - Verifying the common production use cases >>>>> [2] https://wso2.org/jira/browse/ANLYESB-15 >>>>> >>>>> Thanks, >>>>> Supun >>>>> >>>>> -- >>>>> *Supun Sethunga* >>>>> Software Engineer >>>>> WSO2, Inc. >>>>> http://wso2.com/ >>>>> lean | enterprise | middleware >>>>> Mobile : +94 716546324 >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> Architecture@wso2.org >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> *S. Suhothayan* >>>> Technical Lead & Team Lead of WSO2 Complex Event Processor >>>> *WSO2 Inc. *http://wso2.com >>>> * <http://wso2.com/>* >>>> lean . enterprise . middleware >>>> >>>> >>>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: >>>> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/>twitter: >>>> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | linked-in: >>>> http://lk.linkedin.com/in/suhothayan >>>> <http://lk.linkedin.com/in/suhothayan>* >>>> >>> >>> >>> >>> -- >>> *Anjana Fernando* >>> Senior Technical Lead >>> WSO2 Inc. | http://wso2.com >>> lean . enterprise . middleware >>> >> >> >> >> -- >> >> *S. Suhothayan* >> Technical Lead & Team Lead of WSO2 Complex Event Processor >> *WSO2 Inc. *http://wso2.com >> * <http://wso2.com/>* >> lean . enterprise . middleware >> >> >> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: >> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/>twitter: >> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | linked-in: >> http://lk.linkedin.com/in/suhothayan <http://lk.linkedin.com/in/suhothayan>* >> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > *Supun Sethunga* > Software Engineer > WSO2, Inc. > http://wso2.com/ > lean | enterprise | middleware > Mobile : +94 716546324 > > _______________________________________________ > Architecture mailing list > Architecture@wso2.org > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- ============================ Srinath Perera, Ph.D. http://people.apache.org/~hemapani/ http://srinathsview.blogspot.com/
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
