Hi, Good progress Supun! .. do keep pushing the parameters to find the limits we can go to.
@Suho, the idea was to all together eliminate the batch script and just store/index the data for later lookup, and do the computation purely in Siddhi. I don't think we will get a big scaling problem, since the data needs to be stored in-memory when we go to upper layers of summarization is smaller, and stops at yearly granularity. So it would be at that time, we having data in-memory for last years worth of data, in a way of last 12 records of summary data for 12 months for a specific artifact, last day's worth, that is 30 entries etc.. so growing of data slows immensely, and also it has a upper limit, which I guess should be comfortability within usual memory capacity. So if we can get a proper checkpoint and replay mechanism figured out for data processed, we can do all the things in CEP, then we just don't have the complexity of maintaining two mechanism of doing the processing. Cheers, Anjana. On Wed, Apr 20, 2016 at 12:11 PM, Sriskandarajah Suhothayan <s...@wso2.com> wrote: > I think it will make more sense to run seconds and minutes from siddhi, > and run the spark every hour, when there are lots of date on the system > this will be much more scalable. > > WDYT? > > Regards > Suho > > On Wed, Apr 20, 2016 at 11:50 AM, Supun Sethunga <sup...@wso2.com> wrote: > >> Hi, >> >> This is a follow-up mail of [1], to give an update on the status with the >> performance issue [2] . So as mentioned in the previous mail, with >> Spark-script doing the summary stat generation as a batch process, creates >> a bottleneck at a higher TPS. More precisely, with our findings, it cannot >> handle a throughput of more than 30 TPS as a batch process. (i.e: events >> published to DAS within 10 mins with a TPS of 30, take more than 10 mins to >> process. Means, if we schedule a script every 10 mins, the events to be >> processed grows over time). >> >> To overcome this, thought of doing the summarizing up to a certain extent >> (upto second-wise summary) using siddhi, and to generate remaining >> stats (per-minute/hour/day/month), using spark. With this enhancement, ran >> some load tests locally to evaluate this approach, and the results are as >> follows. >> >> Backend DB : MySQL >> ESB analytics nodes: 1 >> >> With InnoDB >> >> - With *80 TPS*: (script scheduled every 1 min) : Avg time taken for >> completion of the script = ~ *20 sec*. >> - With* 500 TPS* (script scheduled every 2 min) : Avg time taken for >> completion of the script = ~ *45 sec*. >> >> >> With MyISAM >> >> - With *80 TPS* (script scheduled every 1 min) : Avg time taken for >> completion of the script = ~ *24 sec*. >> - With *80 TPS *(script scheduled every 2 min) : Avg time taken for >> completion of the script = ~ *20 sec*. >> - With *500 TPS* (script scheduled every 2 min) : Avg time taken for >> completion of the script = ~ *35 sec*. >> >> As a further improvement, we would be trying out to do summarizing upto >> minute/hour level (eventually do all the summarizing using siddhi). >> >> [1] [Dev] ESB Analytics - Verifying the common production use cases >> [2] https://wso2.org/jira/browse/ANLYESB-15 >> >> Thanks, >> Supun >> >> -- >> *Supun Sethunga* >> Software Engineer >> WSO2, Inc. >> http://wso2.com/ >> lean | enterprise | middleware >> Mobile : +94 716546324 >> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > > *S. Suhothayan* > Technical Lead & Team Lead of WSO2 Complex Event Processor > *WSO2 Inc. *http://wso2.com > * <http://wso2.com/>* > lean . enterprise . middleware > > > *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: > http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/>twitter: > http://twitter.com/suhothayan <http://twitter.com/suhothayan> | linked-in: > http://lk.linkedin.com/in/suhothayan <http://lk.linkedin.com/in/suhothayan>* > -- *Anjana Fernando* Senior Technical Lead WSO2 Inc. | http://wso2.com lean . enterprise . middleware
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
