Re: [Architecture] [EMM] Enabling application white listing and application blacklisting support

[Lakshman Udayakantha]

Hi Chathura,

The approach you mentioned can be broken for two incidents mentioned below.

1. Remove a role from a user
2. Remove a role from an application

If I explained first case,
Let's suppose *application1* has restricted to *role1* and *user1* has
*role1*. User will be able to see the *application1* and subscribed to it.
After subscribed to *application1*, *role1* has removed from *user1*. After
that also user can see *application1* is a subscribed application. If we
get subscribed application list in this moment in order to send to the
device for white listing, It should list *application1* (which is wrong)
also. So shall we go with the approach that I mentioned or is there any
better of doing this?

Thanks

On Tue, Mar 1, 2016 at 7:28 PM, Chathura Dilan  wrote:

> Hi Lakshman,
>
>
> +1 for two options in EMM
>
> 1. White list is enabled or not
> 2. Which app store is used to provide the white list
>
>
> But the operation I think it is very expensive.
> 1. Getting app list from the device ( If user has multiple devices, you
> need to get them separately)
> 2. Sending all of them to app manager to check the compliance.
> 3. Then block apps which do not comply with each devices.
>
> Following operation would be much easy.
> 1. Getting subscribed app list from the user in app manager.
> 2. Sending that app list to user's devices for white listing.
>
>
>
> On Tue, Mar 1, 2016 at 5:32 AM, Lakshman Udayakantha 
> wrote:
>
>> Hi Chathura et al,
>>
>> @Chathura:Thanks for the detailed information.
>>
>> As per the offline discussion with PrabathA, EMM should specify
>> explicitly that EMM is using app manager white list. Therefore when policy
>> is created, below information should provided.
>>
>> 1. White list is enabled or not
>> 2. Which app store is used to provide the white list
>>
>> When consider policy compliance, It will get installed application list
>> from device and will pass that application list to app manager. There
>> should be an API in app manager to return true if application list has
>> access to the user and if not return false with application list which is
>> not access by user and if there is mismatch emm will pass with relevant
>> application list to device to block them or uninstall.
>>
>> Please comment if you have any concern about this approach.
>>
>> Thanks
>>
>> On Mon, Feb 29, 2016 at 5:47 PM, Lakshman Udayakantha > > wrote:
>>
>>> Hi All,
>>>
>>> After bit of discussion with EMM team, it is decided to move creation of
>>> white list policy to the app manager it self. We will provide a
>>> configuration whether white list is enabled or not. If white list enabled
>>> in the configuration, a new policy will created in EMM when new application
>>> is going to publish. All other later application will be added to existing
>>> white list policy when they are going to publish. This new policy will list
>>> in policy list and can be edit later to add a new policy criteria.
>>>
>>> When considering policy compliance, EMM will request application list
>>> with roles and update the existing policy and will request installed
>>> application list from device and will compare both for compliance Or
>>> another option would be to update the existing policy when something happen
>>> in app manager that would break existing app to role mapping. Ex:
>>>
>>> 1. Remove a role from a user
>>> 2. Remove a role from an application
>>>
>>> and request installed application list from device and check both for
>>> compliance.
>>>
>>> Another thing is these application restrictions will be implement based
>>> on COPE at initial stage. BYOD scenario will be considered later because of
>>> the complications mentioned in previous reply.
>>>
>>> WDYT about this approach?
>>>
>>> Thanks
>>>
>>> On Tue, Feb 23, 2016 at 3:26 PM, Lakshman Udayakantha <
>>> lakshm...@wso2.com> wrote:
>>>
 Hi DilanA/EMM Team,

 @DilanA :Thanks for the information.

 I have assumed policy creator know the package names of the
 applications which need to be restricted in the device and implemented the
 mdm policy UI for app restriction list and able to publish the restriction
 list to the device successfully as the first step.

 Some terminology has been changed after this thread initialised. As of
 now If AWL is enabled we will provide role based application access. Policy
 creator will define application white list with set of roles along with the
 application. Only those roles will be able to access the application. If
 ABL is enabled, policy creator will define black list via the UI and those
 application list will not be allowed to run on any device.

 @EMM Team: I got several questions regarding the restriction apps using
 mobile agent app.
 1. If we provide AWL, only those applications will show in app manager
 store. Other app stores, side loading and google play store needs to 

Re: [Architecture] [AppCloud] Defining the Service Name in Kubernetes

[Imesh Gunaratne]

On Fri, Mar 4, 2016 at 11:38 AM, Pirinthapan Mahendran  wrote:

> Hi Imesh and Lakmal,
>
> Thanks for the replies. Since the generated hash value is sufficient, we
> are going with that solution.
>

+1 It might be better to add labels to the services including their
metadata (ex: application-id, version, etc).

Thanks

>
> Thanks,
> Pirinthapan.
>
>
>
> Mahendran Pirinthapan
> Software Engineer | WSO2 Inc.
> Mobile +94772378732.
>
> On Fri, Mar 4, 2016 at 11:13 AM, Lakmal Warusawithana 
> wrote:
>
>>
>>
>> On Fri, Mar 4, 2016 at 9:33 AM, Lakmal Warusawithana 
>> wrote:
>>
>>>
>>>
>>> On Thu, Mar 3, 2016 at 9:42 PM, Pirinthapan Mahendran <
>>> pirintha...@wso2.com> wrote:
>>>
 Hi all,

 In AppCloud, when deploying application in Kubernetes we are using the
 following pattern as the service name.

 *--*

 We are using the above pattern because a particular Application can be
 identified by Application Name and Version within a tenant.

 But Kubernetes limits the number of characters of this service name to
 24. According to our current implementation, the version can have maximum
 of 13 character. So if we consider these limitations, the Application Name
 can have only 6 characters. So we are trying to come up with a solution to
 increase the length of the application name.

 Initially we thought about using UUID as primary key and replacing 
 *'>>> Name>-' *with the UUID. But according to the standared, UUID
 has 36 characters. So we cannot use UUID.

 Currently we have the follwoing two options to solve this issue.

 1. Using auto generated id of mysql database, which is generated during
 adding application, instead of *'-'*. But
 its not the recommonded way to use the auto generated id outside to
 database.

 2. Defining our own hash function to generate a 24 character string
 instead of *'**--'*.


>>> Do we need port listed in service name? If not we can get 10
>>> characters.  For readable and traceable purpose better to have app name and
>>> version. Why do we need 13 character version here, can't we reduce it?
>>>
>>
>> As we discussed, since app name, version and port are in the service
>> labels, we can use generated hash value for service names
>>
>>
>>
>>>
>>>
 Could you please share your opinion on the above options?

 Thanks & Regards,
 Mahendran Pirinthapan
 Software Engineer | WSO2 Inc.
 Mobile +94772378732.

>>>
>>>
>>>
>>> --
>>> Lakmal Warusawithana
>>> Director - Cloud Architecture; WSO2 Inc.
>>> Mobile : +94714289692
>>> Blog : http://lakmalsview.blogspot.com/
>>>
>>>
>>
>>
>> --
>> Lakmal Warusawithana
>> Director - Cloud Architecture; WSO2 Inc.
>> Mobile : +94714289692
>> Blog : http://lakmalsview.blogspot.com/
>>
>>
>


-- 
*Imesh Gunaratne*
Senior Technical Lead
WSO2 Inc: http://wso2.com
T: +94 11 214 5345 M: +94 77 374 2057
W: http://imesh.io
Lean . Enterprise . Middleware
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [AppCloud] Defining the Service Name in Kubernetes

[Pirinthapan Mahendran]

Hi Imesh and Lakmal,

Thanks for the replies. Since the generated hash value is sufficient, we
are going with that solution.

Thanks,
Pirinthapan.



Mahendran Pirinthapan
Software Engineer | WSO2 Inc.
Mobile +94772378732.

On Fri, Mar 4, 2016 at 11:13 AM, Lakmal Warusawithana 
wrote:

>
>
> On Fri, Mar 4, 2016 at 9:33 AM, Lakmal Warusawithana 
> wrote:
>
>>
>>
>> On Thu, Mar 3, 2016 at 9:42 PM, Pirinthapan Mahendran <
>> pirintha...@wso2.com> wrote:
>>
>>> Hi all,
>>>
>>> In AppCloud, when deploying application in Kubernetes we are using the
>>> following pattern as the service name.
>>>
>>> *--*
>>>
>>> We are using the above pattern because a particular Application can be
>>> identified by Application Name and Version within a tenant.
>>>
>>> But Kubernetes limits the number of characters of this service name to
>>> 24. According to our current implementation, the version can have maximum
>>> of 13 character. So if we consider these limitations, the Application Name
>>> can have only 6 characters. So we are trying to come up with a solution to
>>> increase the length of the application name.
>>>
>>> Initially we thought about using UUID as primary key and replacing 
>>> *'>> Name>-' *with the UUID. But according to the standared, UUID
>>> has 36 characters. So we cannot use UUID.
>>>
>>> Currently we have the follwoing two options to solve this issue.
>>>
>>> 1. Using auto generated id of mysql database, which is generated during
>>> adding application, instead of *'-'*. But
>>> its not the recommonded way to use the auto generated id outside to
>>> database.
>>>
>>> 2. Defining our own hash function to generate a 24 character string
>>> instead of *'**--'*.
>>>
>>>
>> Do we need port listed in service name? If not we can get 10 characters.
>> For readable and traceable purpose better to have app name and version. Why
>> do we need 13 character version here, can't we reduce it?
>>
>
> As we discussed, since app name, version and port are in the service
> labels, we can use generated hash value for service names
>
>
>
>>
>>
>>> Could you please share your opinion on the above options?
>>>
>>> Thanks & Regards,
>>> Mahendran Pirinthapan
>>> Software Engineer | WSO2 Inc.
>>> Mobile +94772378732.
>>>
>>
>>
>>
>> --
>> Lakmal Warusawithana
>> Director - Cloud Architecture; WSO2 Inc.
>> Mobile : +94714289692
>> Blog : http://lakmalsview.blogspot.com/
>>
>>
>
>
> --
> Lakmal Warusawithana
> Director - Cloud Architecture; WSO2 Inc.
> Mobile : +94714289692
> Blog : http://lakmalsview.blogspot.com/
>
>
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [AppCloud] Defining the Service Name in Kubernetes

[Lakmal Warusawithana]

On Fri, Mar 4, 2016 at 9:33 AM, Lakmal Warusawithana 
wrote:

>
>
> On Thu, Mar 3, 2016 at 9:42 PM, Pirinthapan Mahendran <
> pirintha...@wso2.com> wrote:
>
>> Hi all,
>>
>> In AppCloud, when deploying application in Kubernetes we are using the
>> following pattern as the service name.
>>
>> *--*
>>
>> We are using the above pattern because a particular Application can be
>> identified by Application Name and Version within a tenant.
>>
>> But Kubernetes limits the number of characters of this service name to
>> 24. According to our current implementation, the version can have maximum
>> of 13 character. So if we consider these limitations, the Application Name
>> can have only 6 characters. So we are trying to come up with a solution to
>> increase the length of the application name.
>>
>> Initially we thought about using UUID as primary key and replacing 
>> *'> Name>-' *with the UUID. But according to the standared, UUID
>> has 36 characters. So we cannot use UUID.
>>
>> Currently we have the follwoing two options to solve this issue.
>>
>> 1. Using auto generated id of mysql database, which is generated during
>> adding application, instead of *'-'*. But its
>> not the recommonded way to use the auto generated id outside to database.
>>
>> 2. Defining our own hash function to generate a 24 character string
>> instead of *'**--'*.
>>
>>
> Do we need port listed in service name? If not we can get 10 characters.
> For readable and traceable purpose better to have app name and version. Why
> do we need 13 character version here, can't we reduce it?
>

As we discussed, since app name, version and port are in the service
labels, we can use generated hash value for service names



>
>
>> Could you please share your opinion on the above options?
>>
>> Thanks & Regards,
>> Mahendran Pirinthapan
>> Software Engineer | WSO2 Inc.
>> Mobile +94772378732.
>>
>
>
>
> --
> Lakmal Warusawithana
> Director - Cloud Architecture; WSO2 Inc.
> Mobile : +94714289692
> Blog : http://lakmalsview.blogspot.com/
>
>


-- 
Lakmal Warusawithana
Director - Cloud Architecture; WSO2 Inc.
Mobile : +94714289692
Blog : http://lakmalsview.blogspot.com/
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [AppCloud] Defining the Service Name in Kubernetes

[Lakmal Warusawithana]

On Thu, Mar 3, 2016 at 9:42 PM, Pirinthapan Mahendran 
wrote:

> Hi all,
>
> In AppCloud, when deploying application in Kubernetes we are using the
> following pattern as the service name.
>
> *--*
>
> We are using the above pattern because a particular Application can be
> identified by Application Name and Version within a tenant.
>
> But Kubernetes limits the number of characters of this service name to 24.
> According to our current implementation, the version can have maximum of 13
> character. So if we consider these limitations, the Application Name can
> have only 6 characters. So we are trying to come up with a solution to
> increase the length of the application name.
>
> Initially we thought about using UUID as primary key and replacing 
> *' Name>-' *with the UUID. But according to the standared, UUID has
> 36 characters. So we cannot use UUID.
>
> Currently we have the follwoing two options to solve this issue.
>
> 1. Using auto generated id of mysql database, which is generated during
> adding application, instead of *'-'*. But its
> not the recommonded way to use the auto generated id outside to database.
>
> 2. Defining our own hash function to generate a 24 character string
> instead of *'**--'*.
>
>
Do we need port listed in service name? If not we can get 10 characters.
For readable and traceable purpose better to have app name and version. Why
do we need 13 character version here, can't we reduce it?


> Could you please share your opinion on the above options?
>
> Thanks & Regards,
> Mahendran Pirinthapan
> Software Engineer | WSO2 Inc.
> Mobile +94772378732.
>



-- 
Lakmal Warusawithana
Director - Cloud Architecture; WSO2 Inc.
Mobile : +94714289692
Blog : http://lakmalsview.blogspot.com/
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [API Analytics] Health Monitoring

[Malintha Amarasinghe]

Hi,

Are we using the same handler we publish data to DAS (AFAIU the events go
through CEP before publishing to DAS)?

If so I have a small doubt if we can cover the following scenario as the
synapse response handlers are executed before the outSequence of the API.

1. Backend returns an invalid payload with 200 OK
2. But at the gateway when sending it to the client it fails. May be its
trying to build the payload at the outSequence and fails.

Now actually the API has been failed to respond to the client but i'm
afraid it is tracked by our events sent to CEP.

Thanks,
Malintha


On Thu, Mar 3, 2016 at 11:10 PM, Nirmal Fernando  wrote:

> +1 for API Availability Dashboard.
>
> On Thu, Mar 3, 2016 at 10:03 PM, Janaka Ranabahu  wrote:
>
>> Hi Maheshakya,
>>
>> On Thu, Mar 3, 2016 at 4:30 PM, Maheshakya Wijewardena <
>> mahesha...@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> Due to the potential security issues and complexity issues arose in the
>>> discussions  among APIM analytics team members, we have decided to get rid
>>> of the separate Java client for API health monitoring. Instead, this will
>>> be implemented as CEP execution plans like other APIM analytics tasks. The
>>> following information will be extracted from the APIM event streams.
>>>
>> ​I believe we agreed not to call this API Health monitoring as we can not
>> guarantee the health of the API using only these values. Instead we opted
>> to call this as something like API Availability.
>>
>> Lets finalize a new name for this as well. IMO, we should be calling this
>> as "API Availability".
>> WDYT?
>>
>> Thanks,
>> Janaka  ​
>>
>>
>>>
>>>1. API request count frequency.
>>>2. API response time
>>>3. API response error codes
>>>
>>> The criteria to categorize healthy/unhealthy scenario is as follows:
>>>
>>>1. If the API request count frequency gets lower than the 90th
>>>percentile of the overall, that API is not healthy within the given 
>>> period
>>>of time.
>>>2. If the API response times get higher than 90th percentile of the
>>>overall, that API is not healthy within the given period of time.
>>>3. If the API response contains error codes (given in configuration,
>>>eg: 5xx, 4xx, that API is not healthy within the given period of time.
>>>
>>> If an API is stated as no healthy, the reason will be provided (one or
>>> more of the above three).
>>>
>>> Best regards.
>>>
>>> On Tue, Mar 1, 2016 at 10:07 AM, Maheshakya Wijewardena <
>>> mahesha...@wso2.com> wrote:
>>>
 Hi,

 We have decided not to include the HTTP methods that affects the
 backends of APIs, such as POST, DELETE since the health checker should not
 cause any differences in the backend.

 Best regards.

 On Tue, Feb 23, 2016 at 10:53 AM, Maheshakya Wijewardena <
 mahesha...@wso2.com> wrote:

> Hi,
>
> We have discussed (APIM team and analytics team) and came to a
> decision to implement the health monitor for APIM as a separate Java
> client. The following diagram depicts the architecture of the health
> monitor.
>
>
> ​
>
>- Health checker incorporates the full life cycle of an API,
>therefore it is located outside the firewall and can be viewed by the 
> admin
>of the APIM.
>- This client consumes quota from the whole API, therefore the
>client needs to manage it.
>- This Java client will ping (call API methods from the client)
>the APIs periodically (hourly, daily, weekly) and tracks the status 
> codes
>of the responses.
>- For each API, there will be a separate thread spawned and that
>thread will keep track of that API.
>- This information will be published to DAS inside the firewall
>using HTTPS protocal.
>- Based on the status code, a status will be reported as one of
>the following (These are to be decided):
>   1. Healthy - 2xx
>   2. Some problems occurred during the time period - 4xx
>   3. Failures reported - 5xx
>-  Following configuration information will be given the the
>health monitor using a yaml file:
>   - APIM credentials:
>  - APIM username
>  - APIM password
>   - APIM configuration info:
>  - APIM host
>  - APIM port
>  - Application name
>  - Application tier
>  - Application key type
>  - Application key validity time (default -1, infinitely
>  valid)
>  - Information of APIs:
> - API name
> - API url
> - API version
> - API provider
> - API tier
> - API parameters (if available)
> - API payload (if available)
>  - DAS creadentials:
>  - DAS username
>  - DAS 

Re: [Architecture] [API Analytics] Health Monitoring

[Nirmal Fernando]

+1 for API Availability Dashboard.

On Thu, Mar 3, 2016 at 10:03 PM, Janaka Ranabahu  wrote:

> Hi Maheshakya,
>
> On Thu, Mar 3, 2016 at 4:30 PM, Maheshakya Wijewardena <
> mahesha...@wso2.com> wrote:
>
>> Hi,
>>
>> Due to the potential security issues and complexity issues arose in the
>> discussions  among APIM analytics team members, we have decided to get rid
>> of the separate Java client for API health monitoring. Instead, this will
>> be implemented as CEP execution plans like other APIM analytics tasks. The
>> following information will be extracted from the APIM event streams.
>>
> ​I believe we agreed not to call this API Health monitoring as we can not
> guarantee the health of the API using only these values. Instead we opted
> to call this as something like API Availability.
>
> Lets finalize a new name for this as well. IMO, we should be calling this
> as "API Availability".
> WDYT?
>
> Thanks,
> Janaka  ​
>
>
>>
>>1. API request count frequency.
>>2. API response time
>>3. API response error codes
>>
>> The criteria to categorize healthy/unhealthy scenario is as follows:
>>
>>1. If the API request count frequency gets lower than the 90th
>>percentile of the overall, that API is not healthy within the given period
>>of time.
>>2. If the API response times get higher than 90th percentile of the
>>overall, that API is not healthy within the given period of time.
>>3. If the API response contains error codes (given in configuration,
>>eg: 5xx, 4xx, that API is not healthy within the given period of time.
>>
>> If an API is stated as no healthy, the reason will be provided (one or
>> more of the above three).
>>
>> Best regards.
>>
>> On Tue, Mar 1, 2016 at 10:07 AM, Maheshakya Wijewardena <
>> mahesha...@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> We have decided not to include the HTTP methods that affects the
>>> backends of APIs, such as POST, DELETE since the health checker should not
>>> cause any differences in the backend.
>>>
>>> Best regards.
>>>
>>> On Tue, Feb 23, 2016 at 10:53 AM, Maheshakya Wijewardena <
>>> mahesha...@wso2.com> wrote:
>>>
 Hi,

 We have discussed (APIM team and analytics team) and came to a decision
 to implement the health monitor for APIM as a separate Java client. The
 following diagram depicts the architecture of the health monitor.


 ​

- Health checker incorporates the full life cycle of an API,
therefore it is located outside the firewall and can be viewed by the 
 admin
of the APIM.
- This client consumes quota from the whole API, therefore the
client needs to manage it.
- This Java client will ping (call API methods from the client) the
APIs periodically (hourly, daily, weekly) and tracks the status codes of
the responses.
- For each API, there will be a separate thread spawned and that
thread will keep track of that API.
- This information will be published to DAS inside the firewall
using HTTPS protocal.
- Based on the status code, a status will be reported as one of the
following (These are to be decided):
   1. Healthy - 2xx
   2. Some problems occurred during the time period - 4xx
   3. Failures reported - 5xx
-  Following configuration information will be given the the health
monitor using a yaml file:
   - APIM credentials:
  - APIM username
  - APIM password
   - APIM configuration info:
  - APIM host
  - APIM port
  - Application name
  - Application tier
  - Application key type
  - Application key validity time (default -1, infinitely valid)
  - Information of APIs:
 - API name
 - API url
 - API version
 - API provider
 - API tier
 - API parameters (if available)
 - API payload (if available)
  - DAS creadentials:
  - DAS username
  - DAS password
   - DAS configuration info:
  - DAS receiver url
   - An email notification can be configured to be sent when a
failure is reported for an API.

 WDYT?

 Best regards.

 References:

 [1] API metrics: http://apimetrics.io/check-api-health/
 --
 Pruthuvi Maheshakya Wijewardena
 mahesha...@wso2.com
 +94711228855



>>>
>>>
>>> --
>>> Pruthuvi Maheshakya Wijewardena
>>> mahesha...@wso2.com
>>> +94711228855
>>>
>>>
>>>
>>
>>
>> --
>> Pruthuvi Maheshakya Wijewardena
>> mahesha...@wso2.com
>> +94711228855
>>
>>
>>
>
>
> --
> *Janaka Ranabahu*
> Associate Technical Lead, WSO2 Inc.
> http://wso2.com
>
>
> *E-mail: jan...@wso2.com **M: **+94 718370861
> <%2B94%20718370861>*
>
> Lean . 

Re: [Architecture] [AppCloud] Defining the Service Name in Kubernetes

[Imesh Gunaratne]

Hi Pirinthapan,

We had the same problem in Stratos when creating K8S services. The way we
resolved that is using a sequence number and a prefix [1]:
*"service-"*

[1]
https://github.com/apache/stratos/blob/master/components/org.apache.stratos.cloud.controller/src/main/java/org/apache/stratos/cloud/controller/iaases/kubernetes/KubernetesIaas.java#L683

Thanks

On Thu, Mar 3, 2016 at 9:42 PM, Pirinthapan Mahendran 
wrote:

> Hi all,
>
> In AppCloud, when deploying application in Kubernetes we are using the
> following pattern as the service name.
>
> *--*
>
> We are using the above pattern because a particular Application can be
> identified by Application Name and Version within a tenant.
>
> But Kubernetes limits the number of characters of this service name to 24.
> According to our current implementation, the version can have maximum of 13
> character. So if we consider these limitations, the Application Name can
> have only 6 characters. So we are trying to come up with a solution to
> increase the length of the application name.
>
> Initially we thought about using UUID as primary key and replacing 
> *' Name>-' *with the UUID. But according to the standared, UUID has
> 36 characters. So we cannot use UUID.
>
> Currently we have the follwoing two options to solve this issue.
>
> 1. Using auto generated id of mysql database, which is generated during
> adding application, instead of *'-'*. But its
> not the recommonded way to use the auto generated id outside to database.
>
> 2. Defining our own hash function to generate a 24 character string
> instead of *'**--'*.
>
> Could you please share your opinion on the above options?
>
> Thanks & Regards,
> Mahendran Pirinthapan
> Software Engineer | WSO2 Inc.
> Mobile +94772378732.
>



-- 
*Imesh Gunaratne*
Senior Technical Lead
WSO2 Inc: http://wso2.com
T: +94 11 214 5345 M: +94 77 374 2057
W: http://imesh.io
Lean . Enterprise . Middleware
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [API Analytics] Health Monitoring

[Janaka Ranabahu]

Hi Maheshakya,

On Thu, Mar 3, 2016 at 4:30 PM, Maheshakya Wijewardena 
wrote:

> Hi,
>
> Due to the potential security issues and complexity issues arose in the
> discussions  among APIM analytics team members, we have decided to get rid
> of the separate Java client for API health monitoring. Instead, this will
> be implemented as CEP execution plans like other APIM analytics tasks. The
> following information will be extracted from the APIM event streams.
>
​I believe we agreed not to call this API Health monitoring as we can not
guarantee the health of the API using only these values. Instead we opted
to call this as something like API Availability.

Lets finalize a new name for this as well. IMO, we should be calling this
as "API Availability".
WDYT?

Thanks,
Janaka  ​


>
>1. API request count frequency.
>2. API response time
>3. API response error codes
>
> The criteria to categorize healthy/unhealthy scenario is as follows:
>
>1. If the API request count frequency gets lower than the 90th
>percentile of the overall, that API is not healthy within the given period
>of time.
>2. If the API response times get higher than 90th percentile of the
>overall, that API is not healthy within the given period of time.
>3. If the API response contains error codes (given in configuration,
>eg: 5xx, 4xx, that API is not healthy within the given period of time.
>
> If an API is stated as no healthy, the reason will be provided (one or
> more of the above three).
>
> Best regards.
>
> On Tue, Mar 1, 2016 at 10:07 AM, Maheshakya Wijewardena <
> mahesha...@wso2.com> wrote:
>
>> Hi,
>>
>> We have decided not to include the HTTP methods that affects the backends
>> of APIs, such as POST, DELETE since the health checker should not cause any
>> differences in the backend.
>>
>> Best regards.
>>
>> On Tue, Feb 23, 2016 at 10:53 AM, Maheshakya Wijewardena <
>> mahesha...@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> We have discussed (APIM team and analytics team) and came to a decision
>>> to implement the health monitor for APIM as a separate Java client. The
>>> following diagram depicts the architecture of the health monitor.
>>>
>>>
>>> ​
>>>
>>>- Health checker incorporates the full life cycle of an API,
>>>therefore it is located outside the firewall and can be viewed by the 
>>> admin
>>>of the APIM.
>>>- This client consumes quota from the whole API, therefore the
>>>client needs to manage it.
>>>- This Java client will ping (call API methods from the client) the
>>>APIs periodically (hourly, daily, weekly) and tracks the status codes of
>>>the responses.
>>>- For each API, there will be a separate thread spawned and that
>>>thread will keep track of that API.
>>>- This information will be published to DAS inside the firewall
>>>using HTTPS protocal.
>>>- Based on the status code, a status will be reported as one of the
>>>following (These are to be decided):
>>>   1. Healthy - 2xx
>>>   2. Some problems occurred during the time period - 4xx
>>>   3. Failures reported - 5xx
>>>-  Following configuration information will be given the the health
>>>monitor using a yaml file:
>>>   - APIM credentials:
>>>  - APIM username
>>>  - APIM password
>>>   - APIM configuration info:
>>>  - APIM host
>>>  - APIM port
>>>  - Application name
>>>  - Application tier
>>>  - Application key type
>>>  - Application key validity time (default -1, infinitely valid)
>>>  - Information of APIs:
>>> - API name
>>> - API url
>>> - API version
>>> - API provider
>>> - API tier
>>> - API parameters (if available)
>>> - API payload (if available)
>>>  - DAS creadentials:
>>>  - DAS username
>>>  - DAS password
>>>   - DAS configuration info:
>>>  - DAS receiver url
>>>   - An email notification can be configured to be sent when a
>>>failure is reported for an API.
>>>
>>> WDYT?
>>>
>>> Best regards.
>>>
>>> References:
>>>
>>> [1] API metrics: http://apimetrics.io/check-api-health/
>>> --
>>> Pruthuvi Maheshakya Wijewardena
>>> mahesha...@wso2.com
>>> +94711228855
>>>
>>>
>>>
>>
>>
>> --
>> Pruthuvi Maheshakya Wijewardena
>> mahesha...@wso2.com
>> +94711228855
>>
>>
>>
>
>
> --
> Pruthuvi Maheshakya Wijewardena
> mahesha...@wso2.com
> +94711228855
>
>
>


-- 
*Janaka Ranabahu*
Associate Technical Lead, WSO2 Inc.
http://wso2.com


*E-mail: jan...@wso2.com **M: **+94 718370861*

Lean . Enterprise . Middleware
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

[Architecture] [AppCloud] Defining the Service Name in Kubernetes

[Pirinthapan Mahendran]

Hi all,

In AppCloud, when deploying application in Kubernetes we are using the
following pattern as the service name.

*--*

We are using the above pattern because a particular Application can be
identified by Application Name and Version within a tenant.

But Kubernetes limits the number of characters of this service name to 24.
According to our current implementation, the version can have maximum of 13
character. So if we consider these limitations, the Application Name can
have only 6 characters. So we are trying to come up with a solution to
increase the length of the application name.

Initially we thought about using UUID as primary key and replacing
*'-' *with the UUID. But according to the standared, UUID has
36 characters. So we cannot use UUID.

Currently we have the follwoing two options to solve this issue.

1. Using auto generated id of mysql database, which is generated during
adding application, instead of *'-'*. But its
not the recommonded way to use the auto generated id outside to database.

2. Defining our own hash function to generate a 24 character string instead
of *'**--'*.

Could you please share your opinion on the above options?

Thanks & Regards,
Mahendran Pirinthapan
Software Engineer | WSO2 Inc.
Mobile +94772378732.
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [Dev] Web Application Statistics Monitoring for Application Server 6.0.0

[Kishanthan Thangarajah]

Can we profile and check why there is a drop in performance in high
concurrency?

On Wed, Mar 2, 2016 at 2:51 PM, Nathasha Naranpanawa 
wrote:

> Hi,
>
> As initially planned, the feature was implemented to collect web
> application statistics and publish them to DAS.
>
> Given below is an illustration of the high-level architecture of the
> feature;
>
>
> Currently, all configuration parameters required to create a connection to
> DAS (such as DAS username, password, urls, etc.) are provided as valve
> attributes in the server.xml file. The events are published through a
> Thrift data agent and publishing is handled by the DataPublisher API which
> is asynchronous.
>
> If DAS is not reachable, all the events will be buffered in order. When
> DAS is made available, the buffered events will be pushed to DAS and
> published. This is handled by the DataPublisher API itself.
>
> A performance testing was also carried out using Jmeter in order to
> compare the throughput with and without the publisher valve. Following is
> an indication of the throughput of the server according to a set of data
> obtained;
>
>
>
> The Spark script which is currently in the Capp is a sample one. The Cron
> expression in the script is scheduled to execute the script at every
> second. The script presents two tables, raw statistics data and request
> summary per minute. The table that holds raw statistics data are updated at
> every invoke of a web application. The request summary per minute table is
> only updated at each minute of invoke. Summarizing information published to
> DAS and displaying them to users is done as a separate project by another
> intern by creating gadgets to be deployed on the Dashboard server.
>
> Any further suggestions and feedback are highly appreciated.
>
> Thanks,
> Nathasha.
>
>
> On Tue, Dec 15, 2015 at 5:04 PM, Imesh Gunaratne  wrote:
>
>>
>> On Tue, Dec 15, 2015 at 11:29 AM, Nathasha Naranpanawa  wrote:

>
>- Adding a custom Valve to the tomcat server in order to collect
>data from the server.
>
> Can you please explain more details about this valve?
>>
>>- What would be the approach taken to publish data in asynchronous
>>mode?
>>- How do we plan to share data between the thread that accept the
>>request and the thread that publish statistics?
>>- How would that scale/affect if the incoming requests grow?
>>- How would this valve would handle if DAS is not reachable?
>>
>> Thanks
>>
>>>
>- Creating necessary CApp packages in order to define means of
>publishing data from tomcat server to DAS, receiving and storing the 
> data.
>
> The custom tomcat Valve behaves as an Event Publisher and publishes
> data to DAS through Thrift data agent. For each request from the server, 
> an
> Event is created including metadata and payload data and is published to
> the DAS by a DataPublisher.
>
> A CApp was created by including the definitions for an Event Stream,
> Event Receiver and Event Store to receive data from the publisher valve 
> and
> store them. This is deployed in the
> wso2das-3.0.0/repository/deployment/server/carbonapps folder.
>
> A Spark script is also added to the CApp in order to query the data
> received by DAS. The script presents a summary of the requests received by
> DAS per minute. The summary includes the following information;
>
>- average request count
>- average response time
>- HTTP request success count
>- HTTP request error count
>
> Any suggestions and feedback are highly appreciated.
>
> Thanks,
>
>
> --
> Nathasha Naranpanawa
> Software Engineering Intern
> WSO2 Inc.
>
> Email: natha...@wso2.com
> Mobile: +94775496142
> LinkedIn: https://lk.linkedin.com/in/nathashanaranpanawa
>
>
> ___
> Dev mailing list
> d...@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


 --
 --

 *Nuwan Chamara Pallewela*


 *Software Engineer*

 *WSO2, Inc. *http://wso2.com
 *lean . enterprise . middleware*

 Email   *nuw...@wso2.com *
 Mobile  *+94719079739 <%2B94719079739>@*



>>>
>>>
>>> --
>>> --
>>>
>>> *Nuwan Chamara Pallewela*
>>>
>>>
>>> *Software Engineer*
>>>
>>> *WSO2, Inc. *http://wso2.com
>>> *lean . enterprise . middleware*
>>>
>>> Email   *nuw...@wso2.com *
>>> Mobile  *+94719079739 <%2B94719079739>@*
>>>
>>>
>>>
>>> ___
>>> Dev mailing list
>>> d...@wso2.org
>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>
>>>
>>
>>
>> --
>> *Imesh Gunaratne*
>> 

[Architecture] Invitation: Tier Implementation Discussion @ Thu Mar 3, 2016 4pm - 5pm (nadeesha...@wso2.com)

[Nadeeshani Pathirennehelage]

BEGIN:VCALENDAR
PRODID:-//Google Inc//Google Calendar 70.9054//EN
VERSION:2.0
CALSCALE:GREGORIAN
METHOD:REQUEST
BEGIN:VEVENT
DTSTART:20160303T103000Z
DTEND:20160303T113000Z
DTSTAMP:20160303T091900Z
ORGANIZER;CN=Nadeeshani Pathirennehelage:mailto:nadeesha...@wso2.com
UID:q7cb5e81lgfruj5m5lomccj...@google.com
ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=
 TRUE;CN=Manjula Rathnayake;X-NUM-GUESTS=0:mailto:manju...@wso2.com
ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=
 TRUE;CN=Deependra Ariyadewa;X-NUM-GUESTS=0:mailto:d...@wso2.com
ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=
 TRUE;CN=Kasun De Silva;X-NUM-GUESTS=0:mailto:kas...@wso2.com
ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=ACCEPTED;RSVP=TRUE
 ;CN=Nadeeshani Pathirennehelage;X-NUM-GUESTS=0:mailto:nadeesha...@wso2.com
ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=
 TRUE;CN=Lahiru Manohara;X-NUM-GUESTS=0:mailto:lahir...@wso2.com
ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=
 TRUE;CN=Pirinthapan Mahendran;X-NUM-GUESTS=0:mailto:pirintha...@wso2.com
ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=
 TRUE;CN=Punnadi Gunarathna;X-NUM-GUESTS=0:mailto:punn...@wso2.com
ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=
 TRUE;CN=WSO2 Engineering Group;X-NUM-GUESTS=0:mailto:engineering-group@wso2
 .com
ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=
 TRUE;CN=architecture@wso2.org;X-NUM-GUESTS=0:mailto:architecture@wso2.org
CREATED:20160303T090346Z
DESCRIPTION:This event has a Google Hangouts video call.\nJoin: https://plu
 s.google.com/hangouts/_/wso2.com/tier?hceid=bmFkZWVzaGFuaXBAd3NvMi5jb20.q7c
 b5e81lgfruj5m5lomccjsjo=121\n\nView your event at https://www.google.com
 /calendar/event?action=VIEW=cTdjYjVlODFsZ2ZydWo1bTVsb21jY2pzam8gYXJjaGl
 0ZWN0dXJlQHdzbzIub3Jn=MjAjbmFkZWVzaGFuaXBAd3NvMi5jb21kYTQxMDRlZTRmY2QzY
 TVlN2ZjYzFmNDQ2NzgzMDZlMTY2OGFhYWY5=Asia/Colombo=en.
LAST-MODIFIED:20160303T091900Z
LOCATION:6th Floor Meeting Room - Turing 
SEQUENCE:1
STATUS:CONFIRMED
SUMMARY:Tier Implementation Discussion
TRANSP:OPAQUE
END:VEVENT
END:VCALENDAR


invite.ics
Description: application/ics
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [Java8] Checked Exceptions and Stream, Lambdas, Functional Interfaces

[Lochana Ranaweera]

Hi Sameera,

I came across a the topic of exception handling in functional interfaces
when going through the book 'Java 8 for the Really Impatient - Cay S.
Horstmann' which was recommended by my mentor Azeez.
The book however says that what is suggested is a solution but not a
complete fix.

The suggested solution is to use functional interfaces whose methods allow
checked exceptions. For example, the *T get( ) *method of functional
interface *Supplier* do not allow checked exceptions. But, the
functional interface *Callable* has a method that is declared as *T
call( ) throws exception*, which allows checked exceptions. Therefore, the
fix would be to use *Callable* instead of *Supplier*.  I tried this
fix for an example. [1]

In the example given, stream.map( ) method maps objects to another value as
specified by a Function object. Since a functional interface that
allows checked exceptions corresponding to* Function* is not
available, I created a functional interface called *CheckedFunction* which
has a single abstract method that throws a *ClassNotFoundException*.  I
have replaced the *Function* interface using *CheckedFunction* by
writing the *unchecked* method.

import java.util.function.Function;
import java.util.stream.Stream;

public class Example {

public static void main(String[] args) {
Stream.of("java.lang.String", "java.lang.Integer")
.map(unchecked(Class::forName))
.forEach(System.out::println);
}

public static  Function unchecked(CheckedFunction function) {
return arg -> {
try {
return function.apply(arg);
} catch (Exception e) {
throw new RuntimeException(e);
}
};
}
}

interface CheckedFunction {
R apply(T t) throws ClassNotFoundException;
}


​
According to the book, we can implement a version of the *unchecked *method
to deal with lambdas that throws checked exceptions, given that we can find
or provide a functional interface whose abstract method throws arbitrary
exceptions.

[1] https://lochanaranaweera.wordpress.com/2015/11/27/417/


Thanks and regards,

On Wed, Mar 2, 2016 at 11:41 AM, Sameera Jayasoma  wrote:

> You cannot throw checked exceptions from inside Lambda expression/Streams
> in Java 8. This is a well-known issue and there are tons of discussions
> about this. Oracle has not provided a solution yet. Some people say that
> Oracle messed it up. Anyway developers around the globe have come up with
> several workarounds. With C5 effort, we are starting to use Java 8 heavily.
> Therefore we need to agree on a workaround soon.
>
> You probably need to have some experience with Java 8 Lambdas and Streams
> in order to understand this issue. Lets me explain some of the Java 8
> concepts briefly. I am using few sample code snippets extracted from
> various sources.
>
> *Introduction to Java8*
>
> *Streams* - Represents a sequence of elements. Streams supports
> sequential and parallel aggregate operations. In the following example, If
> an element starts with "c", then the element is converted to uppercase. All
> such converted elements are sorted and output to the System.out. JVM
> invokes these operations in an optimized manner, not necessarily in the
> order we've specified.
>
> List list = Arrays.asList("a1", "a2", "b1", "c2", "c1");
>
> list
> .stream()
> .filter(s -> s.startsWith("c"))
> .map(String::toUpperCase)
> .sorted()
> .forEach(System.out::println);
>
> // C1
> // C2
>
>
> *Lambda Expression* - Represents a function which can be created and
> passed around without belonging to any class. This function can be executed
> on demand as well. Take a look at the following example.
>
> list
> .stream()
> .filter(s -> s.startsWith("c"))
>
> Expression inside the filter method is called a lambda expression which
> takes a String object as an argument and returns a boolean. Lambda
> expressions are equivalent to interfaces with one abstract method. This is
> how lambda expressions are modeled in Java8. These interfaces are called
> Functional Interfaces in Java 8. Following is a sample functional
> interface. Above lambda expression is mapped to the following functional
> interface that comes with Java by default. A point to note is that, None of
> these standard functional interfaces available in Java 8 allow to forward
> checked exceptions to the caller.
>
> @FunctionalInterfacepublic interface Predicate {
> boolean test(T t);}
>
> Now that you have a brief understanding of some of the Java 8 features,
> let's look at the problem related to checked exceptions.
>
>
> *Checked Exceptions and Stream, Lambdas, Functional Interfaces*
>
> The test method of the Predicate interface has not declared any exceptions
> in the method signature. What if a checked exception is thrown inside an
> implementation of the test method, how can we forward this