Re: [Architecture] [APIM] [APIM-Analytics] Removing APIM database(AM_DB) dependency for analytics

[Sanjeewa Malalgoda]

On Friday, April 24, 2020, Amila De Silva  wrote:

> Hi Sanjeewa,
> We did consider that option, but didn't  proceed since it appeared error
> prone due to the following reasons. But if the following concerns have been
> addressed with an eventing approach (like with the recommendation feature),
> we can consider using events.
>
> If we go with the eventing approach, we'd have to reconstruct much of the
> information entirely based on events. For example, the APIs currently in
> the system, the resources in the APIs, with whom these APIs are shared
> etc.. . Now since each event reflects a more persistent change on the
> system, missing one would reflect an incorrect state about the system. For
> a simple example missing one API deletion would indicate that the API
> hasn't received any traffic while the actual case is it's not being any
> more in the system. So adopting this approach places more stringent
> requirements on the underlying eventing mechanism too. Another suggestion
> came up was to periodically push information from AM DBs to the Analytics
> side. So in case one cycle fails, the changes can be still published in a
> subsequent cycle.
>
>
> And it would create connections from additional components. Mainly from
> Store, Publisher (and possibly from IS). If events published from these are
> analysed for monitoring/analytics purposes (to identify access patterns, to
> detect any fraudulent activities) then configuring the additional
> connections is justifiable. But if those are only used to
> duplicate/reconstruct  already available data, doubt if it will be a good
> justification.
>
> Then there are different views provided through Analytics Dashboards.
> Managers would see an overall view of the system while publishers would
> only see a subset they are allowed through publisher access control.
> Similar to this, Subscribers would see additional statistics depending on
> the group they belong to. Even if we move to an event based (or to a data
> syncing ) mechanism, processing access controls would be an unnecessary
> overhead IMO. So Analytics would have to rely on AM at least to get the
> filtering done.
>
> We can discuss further about this before moving with this option. But this
> seemed to be a faster and a stable option. Hence the suggestion for
> additional APIs.
>
If we cosider our billing capability  which is directly connected with
revenue we use external eventing to create plans etc. And those compinents
linked from many different  places. In that case also plan entey and
subscription  entry duplicate at billing engine side as well. I see this is
also as same. Recommendation  feature is also very good example as you
pointed. It can work independently without connecting  always.
Also having data in analytics  side give them great flexibility  to write
join quaries and implement specific indexing optimization  etc. And most
importantly we can completely  decouple. I feel its something to consider
before make a decision  on api. Thoughts?

>
> On Thu, Apr 23, 2020 at 12:52 PM Sanjeewa Malalgoda 
> wrote:
>
>> When I looked at some other solutions I found we can do the same using
>> event publishing for API Management related operations. For an example if
>> someone created an API then there will be an event published to the
>> analytics side with API details. Same applies for apps and other entries
>> created in API Manager. If that case then we can easily do anything we need
>> in analytics side(join or create complex views etc). Analytics will be
>> completely independent from API Management. Didn't we considered this when
>> we think about removing APIM DB?
>>
>> Thanks,
>> sanjeewa.
>>
>> On Thu, Apr 23, 2020 at 10:43 AM Ruwini Wijesiri  wrote:
>>
>>> Hi All,
>>>
>>> APIM Analytics 3.0.0 onwards has a database dependency to the APIM
>>> database(AM_DB). This dependency was introduced because of the following
>>> reasons:
>>>
>>>- Data required for certain widgets are persisted only in the AM_DB.
>>>For example, the complete list of resources of an API is available only 
>>> in
>>>the AM_DB. The Analytics_DB only persists the resources which have been
>>>invoked.
>>>- Information required to enforce publisher access control and
>>>application sharing are not persisted in the Analytics_DB.
>>>
>>> The suggested solution to remove this AM_DB dependency is to introduce a
>>> new Analytics REST API( a new webapp) to APIM. This REST API will be solely
>>> responsible for providing the information required for analytics. The
>>> analytics APIs will only have the scope for 'internal/analytics' role,
>>> which is a role used only for analytics.
>>>
>>> The advantages of having a dedicated REST API for analytics are as
>>> follows:
>>>
>>> *Introducing time based filtering:*
>>>
>>> Most of the analytics widgets require time based data filtering. The
>>> existing publisher APIs do not provide this functionality.  By implementing
>>> the new REST APIs with time bas

Re: [Architecture] [APIM] [APIM-Analytics] Removing APIM database(AM_DB) dependency for analytics

[Amila De Silva]

Hi Sanjeewa,
We did consider that option, but didn't  proceed since it appeared error
prone due to the following reasons. But if the following concerns have been
addressed with an eventing approach (like with the recommendation feature),
we can consider using events.

If we go with the eventing approach, we'd have to reconstruct much of the
information entirely based on events. For example, the APIs currently in
the system, the resources in the APIs, with whom these APIs are shared
etc.. . Now since each event reflects a more persistent change on the
system, missing one would reflect an incorrect state about the system. For
a simple example missing one API deletion would indicate that the API
hasn't received any traffic while the actual case is it's not being any
more in the system. So adopting this approach places more stringent
requirements on the underlying eventing mechanism too. Another suggestion
came up was to periodically push information from AM DBs to the Analytics
side. So in case one cycle fails, the changes can be still published in a
subsequent cycle.

And it would create connections from additional components. Mainly from
Store, Publisher (and possibly from IS). If events published from these are
analysed for monitoring/analytics purposes (to identify access patterns, to
detect any fraudulent activities) then configuring the additional
connections is justifiable. But if those are only used to
duplicate/reconstruct  already available data, doubt if it will be a good
justification.

Then there are different views provided through Analytics Dashboards.
Managers would see an overall view of the system while publishers would
only see a subset they are allowed through publisher access control.
Similar to this, Subscribers would see additional statistics depending on
the group they belong to. Even if we move to an event based (or to a data
syncing ) mechanism, processing access controls would be an unnecessary
overhead IMO. So Analytics would have to rely on AM at least to get the
filtering done.

We can discuss further about this before moving with this option. But this
seemed to be a faster and a stable option. Hence the suggestion for
additional APIs.

On Thu, Apr 23, 2020 at 12:52 PM Sanjeewa Malalgoda 
wrote:

> When I looked at some other solutions I found we can do the same using
> event publishing for API Management related operations. For an example if
> someone created an API then there will be an event published to the
> analytics side with API details. Same applies for apps and other entries
> created in API Manager. If that case then we can easily do anything we need
> in analytics side(join or create complex views etc). Analytics will be
> completely independent from API Management. Didn't we considered this when
> we think about removing APIM DB?
>
> Thanks,
> sanjeewa.
>
> On Thu, Apr 23, 2020 at 10:43 AM Ruwini Wijesiri  wrote:
>
>> Hi All,
>>
>> APIM Analytics 3.0.0 onwards has a database dependency to the APIM
>> database(AM_DB). This dependency was introduced because of the following
>> reasons:
>>
>>- Data required for certain widgets are persisted only in the AM_DB.
>>For example, the complete list of resources of an API is available only in
>>the AM_DB. The Analytics_DB only persists the resources which have been
>>invoked.
>>- Information required to enforce publisher access control and
>>application sharing are not persisted in the Analytics_DB.
>>
>> The suggested solution to remove this AM_DB dependency is to introduce a
>> new Analytics REST API( a new webapp) to APIM. This REST API will be solely
>> responsible for providing the information required for analytics. The
>> analytics APIs will only have the scope for 'internal/analytics' role,
>> which is a role used only for analytics.
>>
>> The advantages of having a dedicated REST API for analytics are as
>> follows:
>>
>> *Introducing time based filtering:*
>>
>> Most of the analytics widgets require time based data filtering. The
>> existing publisher APIs do not provide this functionality.  By implementing
>> the new REST APIs with time based filtering functionality, we can move the
>> burden of data filtering to the database level.
>>
>> *Move data processing burden from the front end to the** backend/*
>> *database*
>>
>> Certain widgets require data from multiple tables (For example, the Top
>> Subscription per Provider widget needs data from both AM_API table and
>> AM_SUBSCRIPTIONS table to identify the providers with the highest number of
>> subscriptions). If we use the generic REST APIs provided in the publisher,
>> we will have to do further processing at the frontend to arrive at the
>> final information required. This could cause memory and performance issues
>> when the data load increases. However, if we introduce analytics specific
>> REST APIs tailor made to return the information required by these widgets,
>> we could move the data processing burden to the backend/database leve

Re: [Architecture] [APIM] [APIM-Analytics] Removing APIM database(AM_DB) dependency for analytics

[Sanjeewa Malalgoda]

When I looked at some other solutions I found we can do the same using
event publishing for API Management related operations. For an example if
someone created an API then there will be an event published to the
analytics side with API details. Same applies for apps and other entries
created in API Manager. If that case then we can easily do anything we need
in analytics side(join or create complex views etc). Analytics will be
completely independent from API Management. Didn't we considered this when
we think about removing APIM DB?

Thanks,
sanjeewa.

On Thu, Apr 23, 2020 at 10:43 AM Ruwini Wijesiri  wrote:

> Hi All,
>
> APIM Analytics 3.0.0 onwards has a database dependency to the APIM
> database(AM_DB). This dependency was introduced because of the following
> reasons:
>
>- Data required for certain widgets are persisted only in the AM_DB.
>For example, the complete list of resources of an API is available only in
>the AM_DB. The Analytics_DB only persists the resources which have been
>invoked.
>- Information required to enforce publisher access control and
>application sharing are not persisted in the Analytics_DB.
>
> The suggested solution to remove this AM_DB dependency is to introduce a
> new Analytics REST API( a new webapp) to APIM. This REST API will be solely
> responsible for providing the information required for analytics. The
> analytics APIs will only have the scope for 'internal/analytics' role,
> which is a role used only for analytics.
>
> The advantages of having a dedicated REST API for analytics are as follows:
>
> *Introducing time based filtering:*
>
> Most of the analytics widgets require time based data filtering. The
> existing publisher APIs do not provide this functionality.  By implementing
> the new REST APIs with time based filtering functionality, we can move the
> burden of data filtering to the database level.
>
> *Move data processing burden from the front end to the** backend/*
> *database*
>
> Certain widgets require data from multiple tables (For example, the Top
> Subscription per Provider widget needs data from both AM_API table and
> AM_SUBSCRIPTIONS table to identify the providers with the highest number of
> subscriptions). If we use the generic REST APIs provided in the publisher,
> we will have to do further processing at the frontend to arrive at the
> final information required. This could cause memory and performance issues
> when the data load increases. However, if we introduce analytics specific
> REST APIs tailor made to return the information required by these widgets,
> we could move the data processing burden to the backend/database level.
>
> *Ease of introducing new REST APIs*
>
> When introducing new widgets which require data from AM_DB, if there is no
> existing API to get the required information, we can easily add a new REST
> API to the analytics webapp to get the required information.
>
> Appreciate your thoughts on the above.
>
> Regards,
> Ruwini
> --
> Ruwini Wijesiri
> Senior Software Engineer,
> WSO2 Inc.
>
> Mobile : +94716133480
>
> 
>


-- 
*Sanjeewa Malalgoda*
Software Architect | Associate Director, Engineering - WSO2 Inc.
(m) +94 712933253 | (e) sanje...@wso2.com | (b) Blogger
, Medium


GET INTEGRATION AGILE 
Integration Agility for Digitally Driven Business
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

[Architecture] [APIM] [APIM-Analytics] Removing APIM database(AM_DB) dependency for analytics

[Ruwini Wijesiri]

Hi All,

APIM Analytics 3.0.0 onwards has a database dependency to the APIM
database(AM_DB). This dependency was introduced because of the following
reasons:

   - Data required for certain widgets are persisted only in the AM_DB. For
   example, the complete list of resources of an API is available only in
   the AM_DB. The Analytics_DB only persists the resources which have been
   invoked.
   - Information required to enforce publisher access control and
   application sharing are not persisted in the Analytics_DB.

The suggested solution to remove this AM_DB dependency is to introduce a
new Analytics REST API( a new webapp) to APIM. This REST API will be solely
responsible for providing the information required for analytics. The
analytics APIs will only have the scope for 'internal/analytics' role,
which is a role used only for analytics.

The advantages of having a dedicated REST API for analytics are as follows:

*Introducing time based filtering:*

Most of the analytics widgets require time based data filtering. The
existing publisher APIs do not provide this functionality.  By implementing
the new REST APIs with time based filtering functionality, we can move the
burden of data filtering to the database level.

*Move data processing burden from the front end to the** backend/**database*

Certain widgets require data from multiple tables (For example, the Top
Subscription per Provider widget needs data from both AM_API table and
AM_SUBSCRIPTIONS table to identify the providers with the highest number of
subscriptions). If we use the generic REST APIs provided in the publisher,
we will have to do further processing at the frontend to arrive at the
final information required. This could cause memory and performance issues
when the data load increases. However, if we introduce analytics specific
REST APIs tailor made to return the information required by these widgets,
we could move the data processing burden to the backend/database level.

*Ease of introducing new REST APIs*

When introducing new widgets which require data from AM_DB, if there is no
existing API to get the required information, we can easily add a new REST
API to the analytics webapp to get the required information.

Appreciate your thoughts on the above.

Regards,
Ruwini
-- 
Ruwini Wijesiri
Senior Software Engineer,
WSO2 Inc.

Mobile : +94716133480


___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture