Hi Rukshan, On Wed, Jun 20, 2018 at 9:40 AM Rukshan Premathunga <ruks...@wso2.com> wrote:
> Hi Sinthuja, > > Per min summary within local node will be fine and reduce the traffic to > Analyzer. But can we grantee that, Siddhi apps will not slow down other > functionalities(gateway request)? > IMHO it should not slow down. :) Addtionally we should be simply calculating a time batch window with sum aggregations and group bys therefore I believe, it shouldn't have a great impact on the gateway. This can be try out in ballerina based gateway since streams is already in > there. > Well, ballerina based gateway, IMHO we should use the Ballerina observabilty (metrics and tracing) APIs to calculate the statistics. We are also thinking about merging the Ballerina observability to streams, but still not yet functional upto that level. > But if we take c4 based products this will not be easy right? we have to > release the products with siddhi features. > We just simply need siddhi librabry, not other SP related features. As we promote siddhi as edge analytics libabry for IoT devices as well, I beleieve it will not have a major impact. Anyhow, IMO it's worth to put some effort and see. IMHO, we are simply moving some parts of the existing queries to the gateway. Since we are already doing some APIM analytics SP migration, why not try this as well? Shouldn't be a very hard thing to try. :) Ofcourse, if we have enough bandwidth to the release timelines. Thanks, Sinthuja. > > Thanks and Regards > > On Tue, Jun 19, 2018 at 5:04 PM, Sinthuja Rajendran <sinth...@wso2.com> > wrote: > >> Hi, >> >> With my recent work with metrics and other monitoring systems, I'm >> thinking whether our model of sending everything and calculate in the >> analytics server side is correct. >> >> Basically IMHO, the majority of the product analytics use cases are >> statistics calculation. For example, in APIM, we are calculating API >> statistics, Subscription statistics, etc. And for this, we are sending >> events for every request/response/fault to analytics server, and analytics >> server is performing the actual statistics calculation on the events. >> Therefore in the high traffic scenario, there are a lot of events needs to >> be published to analytics server, and we are getting issues like "Event >> Queue Full" in the gateway. >> >> My proposal is, what if we calculate the statistics on the local node >> itself (similar to the edge analytics by running Siddhi within the >> products), and only send the minute statistics summary to analytics server >> to do the global (across all nodes) calculation. With this way, the traffic >> to analytics server will be reduced drastically as only the final >> calculated value for each group by combination will be reported. Therefore, >> the analytics server can focus on global summarization and other global >> monitoring aspects. >> >> Thanks, >> Sinthuja. >> >> -- >> *Sinthuja Rajendran* >> Senior Technical Lead >> WSO2, Inc.:http://wso2.com >> >> Blog: http://sinthu-rajan.blogspot.com/ >> Mobile: +94774273955 >> >> >> > > > -- > Rukshan Chathuranga. > Software Engineer. > WSO2, Inc. > +94711822074 > -- *Sinthuja Rajendran* Senior Technical Lead WSO2, Inc.:http://wso2.com Blog: http://sinthu-rajan.blogspot.com/ Mobile: +94774273955
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
