Iranga, The optimization applies to siddhi queries that were written for IS analytics. Other product analytics would have different queries so this is not applicable there. This is not a siddhi level change, rather an optimization of the previously written queries.
Seshi On 27 Jun 2016 16:48, "Iranga Muthuthanthri" <ira...@wso2.com> wrote: > > > On Mon, Jun 27, 2016 at 4:35 PM, Mohanadarshan Vivekanandalingam < > mo...@wso2.com> wrote: > >> >> >> On Mon, Jun 27, 2016 at 4:30 PM, Seshika Fernando <sesh...@wso2.com> >> wrote: >> >>> Hey saith, >>> >>> This is great. So when you removed the duplicate windows, there were no >>> OOM issues? >>> >> Yes, we have introduced per-second windows and eventually per-minute >> event count get reduced. Then, there will not be an OOM.. >> > > is this specific for IS-Analytics or common to all other 'product > analytics'. > > >> Latest number is also seems good.. SajithR, can share those numbers.. >> >> Thanks, >> Mohan >> >> >>> Seshi >>> On 24 Jun 2016 14:28, "Sajith Ravindra" <saji...@wso2.com> wrote: >>> >>>> Hi Malith, >>>> >>>> Thanks for the explanation. >>>>> >>>>> I would expect some variation in the throughput. The aim should be >>>>> minimize the variation in the throughput (while maintaining the throughput >>>>> at its highest level). >>>>> >>>> Agreed. Actually, our expectation is to minimize the fluctuation and >>>> to increase the throughput. >>>> >>>>> >>>>> It is possible to measure the latency as well? >>>>> >>>> In IS-Analytics we don't generate any output events or alerts >>>> currently. Therefore, we don't calculate the latency. What we do is >>>> summarize events using siddhi queries and persist them in a DB and further >>>> summarize persisted data using Spark and then displayed through >>>> dashbaords. >>>> >>>> >>>> Thanks >>>> *,Sajith Ravindra* >>>> Senior Software Engineer >>>> WSO2 Inc.; http://wso2.com >>>> lean.enterprise.middleware >>>> >>>> mobile: +94 77 2273550 >>>> blog: http://sajithr.blogspot.com/ >>>> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> >>>> >>>> On Fri, Jun 24, 2016 at 1:57 PM, Malith Jayasinghe <mali...@wso2.com> >>>> wrote: >>>> >>>>> Hi Sajith, >>>>> >>>>> Thanks for the explanation. >>>>> >>>>> I would expect some variation in the throughput. The aim should be >>>>> minimize the variation in the throughput (while maintaining the throughput >>>>> at its highest level). >>>>> >>>>> It is possible to measure the latency as well? >>>>> >>>>> regards >>>>> >>>>> Malith >>>>> >>>>> >>>>> >>>>> On Thu, Jun 23, 2016 at 11:13 PM, Sajith Ravindra <saji...@wso2.com> >>>>> wrote: >>>>> >>>>>> Hi Malith, >>>>>> >>>>>> The deployment is just a standalone DAS server, and we are planning >>>>>> to do a test for HA deployment in recent future. >>>>>> >>>>>> The workload is generated by a .csv data file which has 100K sample >>>>>> events, 10M events are generated by iterating through the same data set >>>>>> 100 >>>>>> times. But we keep increasing the timestamp. A simple thrift client is >>>>>> used >>>>>> to publish data. >>>>>> >>>>>> >>>>>> >>>>>> Thanks >>>>>> *,Sajith Ravindra* >>>>>> Senior Software Engineer >>>>>> WSO2 Inc.; http://wso2.com >>>>>> lean.enterprise.middleware >>>>>> >>>>>> mobile: +94 77 2273550 >>>>>> blog: http://sajithr.blogspot.com/ >>>>>> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> >>>>>> >>>>>> On Fri, Jun 24, 2016 at 10:34 AM, Malith Jayasinghe <mali...@wso2.com >>>>>> > wrote: >>>>>> >>>>>>> Hi Sajith, >>>>>>> >>>>>>> Could you please provide some details about how you are actually >>>>>>> doing these performance tests. For example, what is deployment model? >>>>>>> How >>>>>>> are you generating these workloads/events? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Malith >>>>>>> >>>>>>> On Thu, Jun 23, 2016 at 11:28 AM, Sajith Ravindra <saji...@wso2.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> This is to give an update on the performance study we conducted on >>>>>>>> is-analytics server on last. The idea of this test round was to >>>>>>>> evaluate >>>>>>>> the performance of Siddhi queries used for is-analytics, therefore we >>>>>>>> disabled event stream persistence and spark for this test. >>>>>>>> >>>>>>>> This test was conducted on a standalone DAS server with Xms2g and >>>>>>>> Xmx4g. >>>>>>>> >>>>>>>> On the initial round when input TPS reaches ~20K, the server went >>>>>>>> OOM after consuming around 1M events. The reason for this was the >>>>>>>> events >>>>>>>> accumulated inside 7 1min time batch windows used inside. To overcome >>>>>>>> this >>>>>>>> we implemented an extension to siddhi which allows us to avoid >>>>>>>> duplicating >>>>>>>> the window. >>>>>>>> >>>>>>>> After removing duplicate windows the server was able to consume >>>>>>>> events at a rate of ~22K, but there were fluctuations (see the graph >>>>>>>> bellow) of the throughput. With analysis, we found that intense GC >>>>>>>> causes >>>>>>>> this. We suspect that this intense GC is caused when expiring a large >>>>>>>> number of events accumulated inside 1-minute window. To overcome this >>>>>>>> we >>>>>>>> are planning to batch events in 1-second windows and then accumulate >>>>>>>> 1second batches in 1 min window in order to stop accumulating a large >>>>>>>> number of events. >>>>>>>> >>>>>>>> >>>>>>>> As the next steps, we are planning to test the performance with >>>>>>>> event stream persistence and then move on to check the performance in >>>>>>>> DAS >>>>>>>> minimum HA mode. >>>>>>>> >>>>>>>> We will keep updating this thread with our findings. >>>>>>>> >>>>>>>> Please share your thought, suggestions on this. >>>>>>>> >>>>>>>> Thanks >>>>>>>> *,Sajith Ravindra* >>>>>>>> Senior Software Engineer >>>>>>>> WSO2 Inc.; http://wso2.com >>>>>>>> lean.enterprise.middleware >>>>>>>> >>>>>>>> mobile: +94 77 2273550 >>>>>>>> blog: http://sajithr.blogspot.com/ >>>>>>>> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Architecture mailing list >>>>>>>> Architecture@wso2.org >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Malith Jayasinghe >>>>>>> >>>>>>> >>>>>>> WSO2, Inc. (http://wso2.com) >>>>>>> Email : mali...@wso2.com >>>>>>> Mobile : 0770704040 >>>>>>> Lean . Enterprise . Middleware >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> Architecture@wso2.org >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> Architecture@wso2.org >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Malith Jayasinghe >>>>> >>>>> >>>>> WSO2, Inc. (http://wso2.com) >>>>> Email : mali...@wso2.com >>>>> Mobile : 0770704040 >>>>> Lean . Enterprise . Middleware >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> Architecture@wso2.org >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> Architecture@wso2.org >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> _______________________________________________ >>> Architecture mailing list >>> Architecture@wso2.org >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> *V. Mohanadarshan* >> *Associate Tech Lead,* >> *Data Technologies Team,* >> *WSO2, Inc. http://wso2.com <http://wso2.com> * >> *lean.enterprise.middleware.* >> >> email: mo...@wso2.com >> phone:(+94) 771117673 >> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Thanks & Regards > > Iranga Muthuthanthri > (M) -0777-255773 > Team Product Management > > > _______________________________________________ > Architecture mailing list > Architecture@wso2.org > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > >
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
