Hi Sanjeewa, We did consider that option, but didn't proceed since it appeared error prone due to the following reasons. But if the following concerns have been addressed with an eventing approach (like with the recommendation feature), we can consider using events.
If we go with the eventing approach, we'd have to reconstruct much of the information entirely based on events. For example, the APIs currently in the system, the resources in the APIs, with whom these APIs are shared etc.. . Now since each event reflects a more persistent change on the system, missing one would reflect an incorrect state about the system. For a simple example missing one API deletion would indicate that the API hasn't received any traffic while the actual case is it's not being any more in the system. So adopting this approach places more stringent requirements on the underlying eventing mechanism too. Another suggestion came up was to periodically push information from AM DBs to the Analytics side. So in case one cycle fails, the changes can be still published in a subsequent cycle. And it would create connections from additional components. Mainly from Store, Publisher (and possibly from IS). If events published from these are analysed for monitoring/analytics purposes (to identify access patterns, to detect any fraudulent activities) then configuring the additional connections is justifiable. But if those are only used to duplicate/reconstruct already available data, doubt if it will be a good justification. Then there are different views provided through Analytics Dashboards. Managers would see an overall view of the system while publishers would only see a subset they are allowed through publisher access control. Similar to this, Subscribers would see additional statistics depending on the group they belong to. Even if we move to an event based (or to a data syncing ) mechanism, processing access controls would be an unnecessary overhead IMO. So Analytics would have to rely on AM at least to get the filtering done. We can discuss further about this before moving with this option. But this seemed to be a faster and a stable option. Hence the suggestion for additional APIs. On Thu, Apr 23, 2020 at 12:52 PM Sanjeewa Malalgoda <sanje...@wso2.com> wrote: > When I looked at some other solutions I found we can do the same using > event publishing for API Management related operations. For an example if > someone created an API then there will be an event published to the > analytics side with API details. Same applies for apps and other entries > created in API Manager. If that case then we can easily do anything we need > in analytics side(join or create complex views etc). Analytics will be > completely independent from API Management. Didn't we considered this when > we think about removing APIM DB? > > Thanks, > sanjeewa. > > On Thu, Apr 23, 2020 at 10:43 AM Ruwini Wijesiri <ruw...@wso2.com> wrote: > >> Hi All, >> >> APIM Analytics 3.0.0 onwards has a database dependency to the APIM >> database(AM_DB). This dependency was introduced because of the following >> reasons: >> >> - Data required for certain widgets are persisted only in the AM_DB. >> For example, the complete list of resources of an API is available only in >> the AM_DB. The Analytics_DB only persists the resources which have been >> invoked. >> - Information required to enforce publisher access control and >> application sharing are not persisted in the Analytics_DB. >> >> The suggested solution to remove this AM_DB dependency is to introduce a >> new Analytics REST API( a new webapp) to APIM. This REST API will be solely >> responsible for providing the information required for analytics. The >> analytics APIs will only have the scope for 'internal/analytics' role, >> which is a role used only for analytics. >> >> The advantages of having a dedicated REST API for analytics are as >> follows: >> >> *Introducing time based filtering:* >> >> Most of the analytics widgets require time based data filtering. The >> existing publisher APIs do not provide this functionality. By implementing >> the new REST APIs with time based filtering functionality, we can move the >> burden of data filtering to the database level. >> >> *Move data processing burden from the front end to the** backend/* >> *database* >> >> Certain widgets require data from multiple tables (For example, the Top >> Subscription per Provider widget needs data from both AM_API table and >> AM_SUBSCRIPTIONS table to identify the providers with the highest number of >> subscriptions). If we use the generic REST APIs provided in the publisher, >> we will have to do further processing at the frontend to arrive at the >> final information required. This could cause memory and performance issues >> when the data load increases. However, if we introduce analytics specific >> REST APIs tailor made to return the information required by these widgets, >> we could move the data processing burden to the backend/database level. >> >> *Ease of introducing new REST APIs* >> >> When introducing new widgets which require data from AM_DB, if there is >> no existing API to get the required information, we can easily add a new >> REST API to the analytics webapp to get the required information. >> >> Appreciate your thoughts on the above. >> >> Regards, >> Ruwini >> -- >> Ruwini Wijesiri >> Senior Software Engineer, >> WSO2 Inc. >> >> Mobile : +94716133480 >> >> <http://wso2.com/signature> >> > > > -- > *Sanjeewa Malalgoda* > Software Architect | Associate Director, Engineering - WSO2 Inc. > (m) +94 712933253 | (e) sanje...@wso2.com | (b) Blogger > <http://sanjeewamalalgoda.blogspot.com>, Medium > <https://medium.com/@sanjeewa190> > > GET INTEGRATION AGILE <https://wso2.com/signature> > Integration Agility for Digitally Driven Business > -- *Amila De Silva* Software Architect | Associate Director, Engineering - WSO2 Inc. (m) +94 775119302 | (e) ami...@wso2.com <http://wso2.com/signature>
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
