Hi, With my recent work with metrics and other monitoring systems, I'm thinking whether our model of sending everything and calculate in the analytics server side is correct.
Basically IMHO, the majority of the product analytics use cases are statistics calculation. For example, in APIM, we are calculating API statistics, Subscription statistics, etc. And for this, we are sending events for every request/response/fault to analytics server, and analytics server is performing the actual statistics calculation on the events. Therefore in the high traffic scenario, there are a lot of events needs to be published to analytics server, and we are getting issues like "Event Queue Full" in the gateway. My proposal is, what if we calculate the statistics on the local node itself (similar to the edge analytics by running Siddhi within the products), and only send the minute statistics summary to analytics server to do the global (across all nodes) calculation. With this way, the traffic to analytics server will be reduced drastically as only the final calculated value for each group by combination will be reported. Therefore, the analytics server can focus on global summarization and other global monitoring aspects. Thanks, Sinthuja. -- *Sinthuja Rajendran* Senior Technical Lead WSO2, Inc.:http://wso2.com Blog: http://sinthu-rajan.blogspot.com/ Mobile: +94774273955
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
