Re: [Architecture] [APIM][IS] A new permission to create OAuth app through Dynamic Client Registration module

[Johann Nallathamby]

Hi Nuwan,

On Wed, Mar 2, 2016 at 2:32 PM, Nuwan Dias  wrote:

> Yes, having a dedicated permissions string would be best IMO. Can we
> finalize on a permissions string for this?
>
> @IS Team, what is the plan of applying the Dynamic Client Registration PR
> to the release branch? We need to make a decision whether we're going to
> use this one or the one we have locally at the moment to proceed. Also, we
> need to decided where to provide feature enhancements as the one being
> discussed on this thread.
>

Tharika didn't have enough time to complete the feature. She was only
assigned for two weeks, in that also last 3 days she was involved in a APIM
support issue. She did port the EMM implementation and got it working with
IS. However there were lot of improvements identified in code review, which
was held in her last week. So I believe we need some one to complete this
off until such time we can't add it to the release.

Currently there is an effort going on in IS team to separate the
identity-framework and connectors. DCR also will be a connector, so if this
is completed at some point we will be able to install this as a connector
to a released product even.

[1] Updated Invitation: [Code Review][IS] OAuth 2.0 Dynamic Client Registration
@ Tue Feb 9, 2016 11:30am - 12:30pm (thar...@wso2.com)

Thanks,
Johann.


> Thanks,
> NuwanD.
>
> On Tue, Mar 1, 2016 at 6:28 PM, Malintha Amarasinghe 
> wrote:
>
>> Hi Darshana,
>>
>> Thanks for the quick response.  I will go through the link.
>>
>> Thanks,
>> Malintha
>> On Mar 1, 2016 5:09 PM, "Darshana Gunawardana"  wrote:
>>
>>> Hi Malintha,
>>>
>>> Yes, the better option is to create new permissions for DCR, rather
>>> reusing already defined permissions. You can refer [1] to see how the
>>> recently developed IS workflow component defined its permission model and
>>> it's hierarchy.
>>>
>>> [1]
>>> http://cdwijayarathna.blogspot.com/2016/01/permission-model-of-wso2-is-workflow.html
>>>
>>> Thanks,
>>>
>>> On Tue, Mar 1, 2016 at 6:20 AM, Malintha Amarasinghe >> > wrote:
>>>
 Hi All,

 Currently Dynamic Client Registration (DCR) module in API Manager [1]
 allows to create OAuth applications irrespective of user permissions. That
 might lead to problems as any user can directly create Apps which might be
 unusable and they can flood the system too.

 Currently in API Manager we have following permissions defined.

 /permission/admin/manage/api/create
 /permission/admin/manage/api/publish
 /permission/admin/manage/api/subscribe

 We initially thought of letting user to create OAuth apps through DCR
 only if user has any of the above permissions. But it then allows *ALL*
 creators/subscribers and publishers to create OAuth apps through DCR and we
 cannot restrict that.

 Hence, we are suggesting to use a new permission for create an OAuth
 app using DCR. Then we can specifically choose which user can access DCR.

 Please share your thoughts.

 PS:
 As per [2] current DCR module of API Manager will be moved as an IS
 component.

 Thanks,
 Malintha

 [1]
 https://github.com/wso2/carbon-apimgt/tree/master/components/apimgt/org.wso2.carbon.apimgt.rest.api.dcr/src/main/java/org/wso2/carbon/apimgt/rest/api/dcr/web
 [2] https://github.com/wso2/carbon-identity/pull/1712/files

 --
 Malintha Amarasinghe
 Software Engineer
 *WSO2, Inc. - lean | enterprise | middleware*
 http://wso2.com/

 Mobile : +94 712383306

>>>
>>>
>>>
>>> --
>>> Regards,
>>>
>>>
>>> *Darshana Gunawardana*Senior Software Engineer
>>> WSO2 Inc.; http://wso2.com
>>>
>>> *E-mail: darsh...@wso2.com *
>>> *Mobile: +94718566859 <%2B94718566859>*Lean . Enterprise . Middleware
>>>
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Nuwan Dias
>
> Technical Lead - WSO2, Inc. http://wso2.com
> email : nuw...@wso2.com
> Phone : +94 777 775 729
>



-- 
Thanks & Regards,

*Johann Dilantha Nallathamby*
Technical Lead & Product Lead of WSO2 Identity Server
Governance Technologies Team
WSO2, Inc.
lean.enterprise.middleware

Mobile - *+9476950*
Blog - *http://nallaa.wordpress.com *
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [PET]Improvement On BigQuery Connector

[Hariprasath Thanarajah]

Yes Malaka,

 It will only be using rest services.

On Thu, Mar 3, 2016 at 10:38 AM, Malaka Silva  wrote:

> Hi Hariprasath,
>
> Sounds good.
>
> So will be adding a new authentication mechanism to the connector. This
> connector will only be using rest services not Google SDK I guess?
>
> On Wed, Mar 2, 2016 at 10:16 PM, Hariprasath Thanarajah <
> haripras...@wso2.com> wrote:
>
>> Hi All,
>>
>> I have planned to do an improvement on BigQuery connector, Since the
>> BigQuery API connector is only supports to authenticated the API call as an
>> end user, and is also can support to authenticate the API calls using a
>> service account instead of an individual end user.
>>
>> For that we need to create a service account to a particular project in
>> BigQuery to get the clientId and the private key.
>>
>> Then,
>>
>> [image: Inline image 1]
>>
>>
>> 1.Create a JSON Web Token , which includes a header, a claim set, and a
>> signature.
>> 2.Request an access token from the Google OAuth 2.0 Authorization Server.
>> 3.Handle the JSON response that the Authorization Server returns.
>> 4.If the response includes an access token, you can use the access token
>> to call a Google API.
>>
>> For more information on this you can follow up this below link [1],
>>
>> [1]https://developers.google.com/identity/protocols/OAuth2ServiceAccount
>>
>> Please suggest, if need any improvements.
>>
>> --
>>
>>
>> *Thank you and Regards**Hariprasath Thanarajah*
>> Associate Software Engineer | WSO2
>> E: haripras...@wso2.com
>> M: +94752806528, 0777216903
>>
>>
>
>
> --
>
> Best Regards,
>
> Malaka Silva
> Senior Tech Lead
> M: +94 777 219 791
> Tel : 94 11 214 5345
> Fax :94 11 2145300
> Skype : malaka.sampath.silva
> LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
> Blog : http://mrmalakasilva.blogspot.com/
>
> WSO2, Inc.
> lean . enterprise . middleware
> http://www.wso2.com/
> http://www.wso2.com/about/team/malaka-silva/
> 
> https://store.wso2.com/store/
>
> Save a tree -Conserve nature & Save the world for your future. Print this
> email only if it is absolutely necessary.
>



-- 


*Thank you and Regards**Hariprasath Thanarajah*
Associate Software Engineer | WSO2
E: haripras...@wso2.com
M: +94752806528, 0777216903
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [PET]Improvement On BigQuery Connector

[Malaka Silva]

Hi Hariprasath,

Sounds good.

So will be adding a new authentication mechanism to the connector. This
connector will only be using rest services not Google SDK I guess?

On Wed, Mar 2, 2016 at 10:16 PM, Hariprasath Thanarajah <
haripras...@wso2.com> wrote:

> Hi All,
>
> I have planned to do an improvement on BigQuery connector, Since the
> BigQuery API connector is only supports to authenticated the API call as an
> end user, and is also can support to authenticate the API calls using a
> service account instead of an individual end user.
>
> For that we need to create a service account to a particular project in
> BigQuery to get the clientId and the private key.
>
> Then,
>
> [image: Inline image 1]
>
>
> 1.Create a JSON Web Token , which includes a header, a claim set, and a
> signature.
> 2.Request an access token from the Google OAuth 2.0 Authorization Server.
> 3.Handle the JSON response that the Authorization Server returns.
> 4.If the response includes an access token, you can use the access token
> to call a Google API.
>
> For more information on this you can follow up this below link [1],
>
> [1]https://developers.google.com/identity/protocols/OAuth2ServiceAccount
>
> Please suggest, if need any improvements.
>
> --
>
>
> *Thank you and Regards**Hariprasath Thanarajah*
> Associate Software Engineer | WSO2
> E: haripras...@wso2.com
> M: +94752806528, 0777216903
>
>


-- 

Best Regards,

Malaka Silva
Senior Tech Lead
M: +94 777 219 791
Tel : 94 11 214 5345
Fax :94 11 2145300
Skype : malaka.sampath.silva
LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
Blog : http://mrmalakasilva.blogspot.com/

WSO2, Inc.
lean . enterprise . middleware
http://www.wso2.com/
http://www.wso2.com/about/team/malaka-silva/

https://store.wso2.com/store/

Save a tree -Conserve nature & Save the world for your future. Print this
email only if it is absolutely necessary.
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [Dev] Web Application Statistics Monitoring for Application Server 6.0.0

[Imesh Gunaratne]

Hi Nathasha,

According to the above graph, IMO Thrift Data Publisher is not asynchronous
enough.

Are we currently invoking the data publisher in the same thread where the
request is being processed? If so shall we try to move that logic to a
separate thread?

Thanks

On Wed, Mar 2, 2016 at 2:51 PM, Nathasha Naranpanawa 
wrote:

> Hi,
>
> As initially planned, the feature was implemented to collect web
> application statistics and publish them to DAS.
>
> Given below is an illustration of the high-level architecture of the
> feature;
>
>
> Currently, all configuration parameters required to create a connection to
> DAS (such as DAS username, password, urls, etc.) are provided as valve
> attributes in the server.xml file. The events are published through a
> Thrift data agent and publishing is handled by the DataPublisher API which
> is asynchronous.
>
> If DAS is not reachable, all the events will be buffered in order. When
> DAS is made available, the buffered events will be pushed to DAS and
> published. This is handled by the DataPublisher API itself.
>
> A performance testing was also carried out using Jmeter in order to
> compare the throughput with and without the publisher valve. Following is
> an indication of the throughput of the server according to a set of data
> obtained;
>
>
>
> The Spark script which is currently in the Capp is a sample one. The Cron
> expression in the script is scheduled to execute the script at every
> second. The script presents two tables, raw statistics data and request
> summary per minute. The table that holds raw statistics data are updated at
> every invoke of a web application. The request summary per minute table is
> only updated at each minute of invoke. Summarizing information published to
> DAS and displaying them to users is done as a separate project by another
> intern by creating gadgets to be deployed on the Dashboard server.
>
> Any further suggestions and feedback are highly appreciated.
>
> Thanks,
> Nathasha.
>
>
> On Tue, Dec 15, 2015 at 5:04 PM, Imesh Gunaratne  wrote:
>
>>
>> On Tue, Dec 15, 2015 at 11:29 AM, Nathasha Naranpanawa >>> > wrote:

>
>- Adding a custom Valve to the tomcat server in order to collect
>data from the server.
>
> Can you please explain more details about this valve?
>>
>>- What would be the approach taken to publish data in asynchronous
>>mode?
>>- How do we plan to share data between the thread that accept the
>>request and the thread that publish statistics?
>>- How would that scale/affect if the incoming requests grow?
>>- How would this valve would handle if DAS is not reachable?
>>
>> Thanks
>>
>>>
>- Creating necessary CApp packages in order to define means of
>publishing data from tomcat server to DAS, receiving and storing the 
> data.
>
> The custom tomcat Valve behaves as an Event Publisher and publishes
> data to DAS through Thrift data agent. For each request from the server, 
> an
> Event is created including metadata and payload data and is published to
> the DAS by a DataPublisher.
>
> A CApp was created by including the definitions for an Event Stream,
> Event Receiver and Event Store to receive data from the publisher valve 
> and
> store them. This is deployed in the
> wso2das-3.0.0/repository/deployment/server/carbonapps folder.
>
> A Spark script is also added to the CApp in order to query the data
> received by DAS. The script presents a summary of the requests received by
> DAS per minute. The summary includes the following information;
>
>- average request count
>- average response time
>- HTTP request success count
>- HTTP request error count
>
> Any suggestions and feedback are highly appreciated.
>
> Thanks,
>
>
> --
> Nathasha Naranpanawa
> Software Engineering Intern
> WSO2 Inc.
>
> Email: natha...@wso2.com
> Mobile: +94775496142
> LinkedIn: https://lk.linkedin.com/in/nathashanaranpanawa
>
>
> ___
> Dev mailing list
> d...@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


 --
 --

 *Nuwan Chamara Pallewela*


 *Software Engineer*

 *WSO2, Inc. *http://wso2.com
 *lean . enterprise . middleware*

 Email   *nuw...@wso2.com *
 Mobile  *+94719079739 <%2B94719079739>@*



>>>
>>>
>>> --
>>> --
>>>
>>> *Nuwan Chamara Pallewela*
>>>
>>>
>>> *Software Engineer*
>>>
>>> *WSO2, Inc. *http://wso2.com
>>> *lean . enterprise . middleware*
>>>
>>> Email   *nuw...@wso2.com *
>>> Mobile  *+94719079739 <%2B94719079739>@*
>>>
>>>
>>>
>>> ___
>>> Dev mailing list
>>> d...@wso2.

Re: [Architecture] [Dev] Creating Gadgets for Web Application Statistics Monitoring Dashboard - Application Server 6.0.0

[Udara Rathnayake]

Hi All,

Try to stick to the existing libraries within platform as much as
possible(d3,vega,viz etc..). These are proven to provide all necessary
functionalities to render gadget UIs.

Regards,
UdaraR

On Wed, Mar 2, 2016 at 7:15 AM, Lochana Ranaweera  wrote:

> Hi Tharik,
>
> Ideally, the gadgets should directly call the DAS Analytics REST API to
> get the summarized information. At the moment, the gadgets are working with
> our own API to retrieve data out of a mysql database. However, the final
> outcome of this project will have the gadgets calling our own API which in
> turn calls the DAS REST API.
>
> As of now, flot is being used as the javascript plotting library. Also I
> will look in to incorporating VizGrammer when creating chart gadgets. :)
>
> Thanks.
>
> On Tue, Mar 1, 2016 at 8:56 AM, Tharik Kanaka  wrote:
>
>> Hi Kasun,
>>
>> If new charts are going to be implemented for these gadgets, it is better
>> to use VizGrammar [1] as it has been using as generic charts for
>> implementing analytics for products such as ESB.
>>
>> [1] http://wso2.github.io/VizGrammar/samples/
>>
>> Regards,
>>
>> On Mon, Feb 29, 2016 at 7:54 PM, KasunG Gajasinghe 
>> wrote:
>>
>>>
>>> It seems these gadgets should directly call the DAS analytics API to get
>>> the summarized info.
>>>
>>> For the folder structure, may be we can create a top-level module called
>>> webapp-analytics and create the CApp there. If there are other artifacts
>>> other than the CApp, then we'll have to add the capp as a sub-module.
>>>
>>> On Fri, Feb 26, 2016 at 4:45 PM, Tharik Kanaka  wrote:
>>>
 Hi Lochana,

 Are you going to use existing analytics API used in DAS? Apart from
 that what are the javascript chart libraries are you going to use for these
 gadgets?

 Regards,

 On Fri, Feb 26, 2016 at 3:46 PM, Lochana Ranaweera 
 wrote:

>
> Hi KasunG,
>
> The final deliverable of the dashboard will be a set of gadgets that
> can be deployed in the DS separately. It was suggested that ideally with
> the AS build the gadgets could be packed as a car file.
>
> As of now, the gadgets are working along with an API that provides the
> gadgets with properly structured data obtained out of a mysql database. 
> The
> table structure for the tables in the mysql database is to be provided
> using a spark script.
>
> The following will have to be done by a user in order view web
> application statistics by means of a dashboard created using these gadgets
>
> - Publish webapp statistics to the DAS using publisher valve at AS
> - Execute the given Spark script to summarize the event data in to a
> database
> - Configure the datasource definition in Dashboard Server to retrieve
> data from the database
> - Create and publish a dashboard using preferred HTTP monitoring
> gadgets through dashboard server portal
>
> Thanks.
>
> On Fri, Feb 26, 2016 at 2:08 PM, KasunG Gajasinghe 
> wrote:
>
>> +architecture
>>
>> Hi Lochana,
>>
>> Can you describe what is the final deliverable of the dashboard?
>> Also, please explain briefly how the users are going to use these gadgets
>> as well as all other dependencies these need. Based on that, we can 
>> figure
>> out how the folder structure should be.
>>
>> Btw, can you explain the overall effort required to port these
>> gadgets? What needs to be done?
>>
>> Thanks.
>>
>> On Fri, Feb 26, 2016 at 12:42 PM, Lochana Ranaweera <
>> locha...@wso2.com> wrote:
>>
>>> Hi all,
>>>
>>> This is regarding the HTTP Monitoring Dashboard for AS 6.0.0. It is
>>> to be deployed as a collection of user preferred set of gadgets in the
>>> Dashboard Server.
>>> As of now, I have ported the* basic* HTTP monitoring functionality
>>> as separate gadgets for DS.
>>>
>>> A typical HTTP monitoring gadget has the following structure:
>>>
>>> gadget-name
>>>├── css
>>>├── js
>>>├── fonts
>>>├── images
>>>├── index.xml
>>>└── gadget.json
>>>
>>> What will be the best structure to version control thus created
>>> gadgets in the given branch
>>>  of AS 6.0.0?
>>> Should it be in modules or should it be inside contents directory in
>>> the distribution module?
>>> The directory structure of the build for new AS 6.0.0 is given
>>> below:
>>>
>>>
>>> ​
>>> Appreciate your suggestions on this matter.
>>>
>>> Thanks and Regards,
>>> --
>>> Lochana Ranaweera
>>> Intern Software Engineer
>>> WSO2 Inc: http://wso2.com
>>> Blog: https://lochanaranaweera.wordpress.com/
>>> Mobile: +94716487055 
>>>
>>>
>>
>>
>> --
>>
>> *Kasun Gajasinghe*Senior Software En

[Architecture] [PET]Improvement On BigQuery Connector

[Hariprasath Thanarajah]

Hi All,

I have planned to do an improvement on BigQuery connector, Since the
BigQuery API connector is only supports to authenticated the API call as an
end user, and is also can support to authenticate the API calls using a
service account instead of an individual end user.

For that we need to create a service account to a particular project in
BigQuery to get the clientId and the private key.

Then,

[image: Inline image 1]


1.Create a JSON Web Token , which includes a header, a claim set, and a
signature.
2.Request an access token from the Google OAuth 2.0 Authorization Server.
3.Handle the JSON response that the Authorization Server returns.
4.If the response includes an access token, you can use the access token to
call a Google API.

For more information on this you can follow up this below link [1],

[1]https://developers.google.com/identity/protocols/OAuth2ServiceAccount

Please suggest, if need any improvements.

-- 


*Thank you and Regards**Hariprasath Thanarajah*
Associate Software Engineer | WSO2
E: haripras...@wso2.com
M: +94752806528, 0777216903
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] API Manager Throttling Design - How to notify throttling events to Gateway cluster

[Harsha Kumara]

+1 for the approach.

Yes we can avoid using durable subscriptions since we are pre-loading the
throttle database snapshot into gateway nodes at the time they are
spawning. We may need to address how existing users will migrate and adapt
the new design.As Sanjeewa mentioned, some users may won't be need of using
complex deployment. So I think we need to revisit how we can address it. We
may  can implement simple throttling mechanism in our CEP based local
throttle component or support both existing implementation along with our
new implementation. Both involves in maintenance overhead.


On Wed, Mar 2, 2016 at 9:21 AM, Sanjeewa Malalgoda 
wrote:

> +1 for this approach.
> Still i think we need to have other alternative for this(web service call
> and pooling mechanism) as some of our users may not need complex deployment
> like this.
> WDYT?
>
> Thanks,
> sanjeewa.
>
> On Wed, Mar 2, 2016 at 12:12 AM, Sriskandarajah Suhothayan 
> wrote:
>
>>
>>
>> On Tue, Mar 1, 2016 at 11:17 PM, Pamod Sylvester  wrote:
>>
>>> Hi Nuwan,
>>>
>>> AFAIU this will not require *durable* subscriptions created between the
>>> GW and the broker, since each time the GW node starts or reconnects it will
>>> sync/snapshot with the DB to obtain the current state. Also i assume in
>>> this case the keys will first be persisted in the DB and then it'll be
>>> broadcasted to a topic.
>>>
>>> If that is the case in the 2nd approach (d) could be omitted if you
>>> subscribe the GW nodes to a non durable topic.
>>>
>>> +1 for the approach,
>> Need to make sure no events missed and at the end only the latest events
>> remains at GW.
>>
>>
>>
>>> Thanks,
>>> Pamod
>>>
>>> On Tue, Mar 1, 2016 at 7:16 PM, Nuwan Dias  wrote:
>>>
 Hi,

 According to the new throttling architecture in API Manager, the
 throttling decision will be taken by a global throttling engine (CEP). The
 Gateway nodes publishes events to the throttling engine and the throttling
 engine evaluates data within those events to decide whether a particular
 'key' is throttled or not. Note: Each event will have a 'key' against which
 throttling counters are maintained. When the throttling engine decides a
 particular key is to be throttled out, it will persist this data to a
 Database. It will later clean this record from the DB when the state of key
 is reverted later.

 The next problem is on how this decision that is made by the throttling
 engine is communicated to the Gateway cluster. We evaluated two options as
 below.

 1. Each Gateway node keeps polling the throttling engine periodically
 (every 5 secs or so) and gets a snapshot of the DB into local memory.

 2. Using a pub-sub mechanism where the throttling engine publishes a
 state change event of a key to a topic and each Gateway node will have
 subscribed to that topic and hence receive updates as and when state
 changes occur. This pattern is illustrated in a diagram below.

 The first approach has some drawbacks such as
 a) Each Gateway has to poll the engine even if no events are throttled
 out.
 b) The amount of data being transferred over the network can be large.
 c) The load on the throttling engine increases as more Gateways start
 polling it.
 d) The load on the DB reads can be high (can use a cache to avoid this).

 The second approach looks better than the first but still have a few
 other drawbacks such as
 a) We will anyway need to get a snapshot from the DB the first time a
 Gateway starts to get the current state.
 b) Deployment complexities when scaling out. We would probably need to
 have a separate message broker cluster in larger deployments.
 c) Need to maintain a consistent http connection between the Gateway
 and the message broker.
 d) If a Gateway node disconnects, the messages intended for that node
 will pile up on the message broker.

 We have currently opted for option 2. Please share your thoughts,
 suggestions on this approach.


 [image: Inline image 1]

 Thanks,
 NuwanD.

 --
 Nuwan Dias

 Technical Lead - WSO2, Inc. http://wso2.com
 email : nuw...@wso2.com
 Phone : +94 777 775 729

 ___
 Architecture mailing list
 Architecture@wso2.org
 https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


>>>
>>>
>>> --
>>> *Pamod Sylvester *
>>>
>>> *WSO2 Inc.; http://wso2.com *
>>> cell: +94 77 7779495
>>>
>>> ___
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> --
>>
>> *S. Suhothayan*
>> Technical Lead & Team Lead of WSO2 Complex Event Processor
>> *WSO2 Inc. *http://wso2.com
>> * *
>> lean . enterprise . middleware
>>
>>
>> *cel

[Architecture] Multitenancy for device type in CDMF

[Ayyoob Hamza]

Hi All,
Below we have described the current implementation and the proposed
implementation for device type multi tenancy.

In CDMF a device type contains following 3 components:
1) Device type plugin - An OSGi bundle which gets registered to CDMF.
2) Set of JAX-RS application that can be deployed either as a managed api
(This gets published to the api manager) or can be secured through the
authentication framework in tomcat valve.
3) Set of UI components.

*Current Implementation*

We deploy the device type implementation as a service and this gets
registered with the CDMF. When the jax-rs application is deployed and if
the managed-api is enabled in the web.xml then it will deploy the APIs  in
super tenant space and the subscription is set to SHARE_WITH_ALL_TENANTS
mode and when the managed-api mode is disabled then the carbon context flow
is created for the user and in the valve and will allow all the request to
the jax-rs backend API(No restriction for tenant which works similar as
SHARE_WITH_ALL_TENANT mode).

*Requirement*

In device type multi tenancy a device author from a tenant can decide to
share the device type with 3 possible approaches.

   - SHARE_WITH_ALL_TENANTS - eg: Current device type implementation for
   EMM (Android, IOS, Windows)
   - SHARE_WITH_ONLY_CURRENT_TENANTS - eg: Most probable use case for
   device cloud. A device type created by a tenant should be in tenant space.
   - SHARE_WITH_SPECIFIC_TENANTS - There isn't any direct practical use
   case therefore we thought to not to have this capability in CDMF.


*Proposed*

Since a device type has plugin and APIs. We have to solve tenant specific
data separation for the device type and also need to have tenant specific
subscription for the device type API. In here JVM does not cater tenant
specific separation therefore we need to have a mechanism to identify from
which tenant does the device type plugin service is been registered

 Therefore in the OSGi service we will have two methods as stated below:

String getProviderTenantDomain();
boolean isSharedWithAllTenants();

This will be picked up in the CDMF after the device type service is
registered. This registration data will be used to store the device type to
tenant specific.

In the JAX-RS application, we need to have two fields in the web.xml. This
is to capture which tenant provided the webapp is linked and also needs to
know whether the APIs are shared with all or not.

providerTenantDomain
true


isSharedWithAllTenants
true

This information will be captured when the jax-rs application deployed and
in:
managed api use case the api and the subscription will be registered with
the above configs.
In non managed api mode the above information will be grabbed in the
authorization framework in the valve and will allow the request only for
the allowed tenant users.

WDYT about the above flow and if there are anything that we have missed
then please let us know?

Thanks,
*Ayyoob Hamza*
*Software Engineer*
WSO2 Inc.; http://wso2.com
email: ayy...@wso2.com cell: +94 77 1681010 <%2B94%2077%207779495>
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [Dev] Web Application Statistics Monitoring for Application Server 6.0.0

[Nathasha Naranpanawa]

Hi,

As initially planned, the feature was implemented to collect web
application statistics and publish them to DAS.

Given below is an illustration of the high-level architecture of the
feature;


Currently, all configuration parameters required to create a connection to
DAS (such as DAS username, password, urls, etc.) are provided as valve
attributes in the server.xml file. The events are published through a
Thrift data agent and publishing is handled by the DataPublisher API which
is asynchronous.

If DAS is not reachable, all the events will be buffered in order. When DAS
is made available, the buffered events will be pushed to DAS and published.
This is handled by the DataPublisher API itself.

A performance testing was also carried out using Jmeter in order to compare
the throughput with and without the publisher valve. Following is an
indication of the throughput of the server according to a set of data
obtained;



The Spark script which is currently in the Capp is a sample one. The Cron
expression in the script is scheduled to execute the script at every
second. The script presents two tables, raw statistics data and request
summary per minute. The table that holds raw statistics data are updated at
every invoke of a web application. The request summary per minute table is
only updated at each minute of invoke. Summarizing information published to
DAS and displaying them to users is done as a separate project by another
intern by creating gadgets to be deployed on the Dashboard server.

Any further suggestions and feedback are highly appreciated.

Thanks,
Nathasha.


On Tue, Dec 15, 2015 at 5:04 PM, Imesh Gunaratne  wrote:

>
> On Tue, Dec 15, 2015 at 11:29 AM, Nathasha Naranpanawa 
>>> wrote:
>>>

- Adding a custom Valve to the tomcat server in order to collect
data from the server.

 Can you please explain more details about this valve?
>
>- What would be the approach taken to publish data in asynchronous
>mode?
>- How do we plan to share data between the thread that accept the
>request and the thread that publish statistics?
>- How would that scale/affect if the incoming requests grow?
>- How would this valve would handle if DAS is not reachable?
>
> Thanks
>
>>
- Creating necessary CApp packages in order to define means of
publishing data from tomcat server to DAS, receiving and storing the 
 data.

 The custom tomcat Valve behaves as an Event Publisher and publishes
 data to DAS through Thrift data agent. For each request from the server, an
 Event is created including metadata and payload data and is published to
 the DAS by a DataPublisher.

 A CApp was created by including the definitions for an Event Stream,
 Event Receiver and Event Store to receive data from the publisher valve and
 store them. This is deployed in the
 wso2das-3.0.0/repository/deployment/server/carbonapps folder.

 A Spark script is also added to the CApp in order to query the data
 received by DAS. The script presents a summary of the requests received by
 DAS per minute. The summary includes the following information;

- average request count
- average response time
- HTTP request success count
- HTTP request error count

 Any suggestions and feedback are highly appreciated.

 Thanks,


 --
 Nathasha Naranpanawa
 Software Engineering Intern
 WSO2 Inc.

 Email: natha...@wso2.com
 Mobile: +94775496142
 LinkedIn: https://lk.linkedin.com/in/nathashanaranpanawa


 ___
 Dev mailing list
 d...@wso2.org
 http://wso2.org/cgi-bin/mailman/listinfo/dev


>>>
>>>
>>> --
>>> --
>>>
>>> *Nuwan Chamara Pallewela*
>>>
>>>
>>> *Software Engineer*
>>>
>>> *WSO2, Inc. *http://wso2.com
>>> *lean . enterprise . middleware*
>>>
>>> Email   *nuw...@wso2.com *
>>> Mobile  *+94719079739 <%2B94719079739>@*
>>>
>>>
>>>
>>
>>
>> --
>> --
>>
>> *Nuwan Chamara Pallewela*
>>
>>
>> *Software Engineer*
>>
>> *WSO2, Inc. *http://wso2.com
>> *lean . enterprise . middleware*
>>
>> Email   *nuw...@wso2.com *
>> Mobile  *+94719079739 <%2B94719079739>@*
>>
>>
>>
>> ___
>> Dev mailing list
>> d...@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
> *Imesh Gunaratne*
> Senior Technical Lead
> WSO2 Inc: http://wso2.com
> T: +94 11 214 5345 M: +94 77 374 2057
> W: http://imesh.gunaratne.org
> Lean . Enterprise . Middleware
>
>


-- 
Nathasha Naranpanawa
Software Engineering Intern
WSO2 Inc.

Email: natha...@wso2.com
Mobile: +94775496142
LinkedIn: https://lk.linkedin.com/in/nathashanaranpanawa
___
Architecture mailing list
Architecture@wso2.org
https://

Re: [Architecture] [APIM][IS] A new permission to create OAuth app through Dynamic Client Registration module

[Nuwan Dias]

Yes, having a dedicated permissions string would be best IMO. Can we
finalize on a permissions string for this?

@IS Team, what is the plan of applying the Dynamic Client Registration PR
to the release branch? We need to make a decision whether we're going to
use this one or the one we have locally at the moment to proceed. Also, we
need to decided where to provide feature enhancements as the one being
discussed on this thread.

Thanks,
NuwanD.

On Tue, Mar 1, 2016 at 6:28 PM, Malintha Amarasinghe 
wrote:

> Hi Darshana,
>
> Thanks for the quick response.  I will go through the link.
>
> Thanks,
> Malintha
> On Mar 1, 2016 5:09 PM, "Darshana Gunawardana"  wrote:
>
>> Hi Malintha,
>>
>> Yes, the better option is to create new permissions for DCR, rather
>> reusing already defined permissions. You can refer [1] to see how the
>> recently developed IS workflow component defined its permission model and
>> it's hierarchy.
>>
>> [1]
>> http://cdwijayarathna.blogspot.com/2016/01/permission-model-of-wso2-is-workflow.html
>>
>> Thanks,
>>
>> On Tue, Mar 1, 2016 at 6:20 AM, Malintha Amarasinghe 
>> wrote:
>>
>>> Hi All,
>>>
>>> Currently Dynamic Client Registration (DCR) module in API Manager [1]
>>> allows to create OAuth applications irrespective of user permissions. That
>>> might lead to problems as any user can directly create Apps which might be
>>> unusable and they can flood the system too.
>>>
>>> Currently in API Manager we have following permissions defined.
>>>
>>> /permission/admin/manage/api/create
>>> /permission/admin/manage/api/publish
>>> /permission/admin/manage/api/subscribe
>>>
>>> We initially thought of letting user to create OAuth apps through DCR
>>> only if user has any of the above permissions. But it then allows *ALL*
>>> creators/subscribers and publishers to create OAuth apps through DCR and we
>>> cannot restrict that.
>>>
>>> Hence, we are suggesting to use a new permission for create an OAuth app
>>> using DCR. Then we can specifically choose which user can access DCR.
>>>
>>> Please share your thoughts.
>>>
>>> PS:
>>> As per [2] current DCR module of API Manager will be moved as an IS
>>> component.
>>>
>>> Thanks,
>>> Malintha
>>>
>>> [1]
>>> https://github.com/wso2/carbon-apimgt/tree/master/components/apimgt/org.wso2.carbon.apimgt.rest.api.dcr/src/main/java/org/wso2/carbon/apimgt/rest/api/dcr/web
>>> [2] https://github.com/wso2/carbon-identity/pull/1712/files
>>>
>>> --
>>> Malintha Amarasinghe
>>> Software Engineer
>>> *WSO2, Inc. - lean | enterprise | middleware*
>>> http://wso2.com/
>>>
>>> Mobile : +94 712383306
>>>
>>
>>
>>
>> --
>> Regards,
>>
>>
>> *Darshana Gunawardana*Senior Software Engineer
>> WSO2 Inc.; http://wso2.com
>>
>> *E-mail: darsh...@wso2.com *
>> *Mobile: +94718566859 <%2B94718566859>*Lean . Enterprise . Middleware
>>
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
Nuwan Dias

Technical Lead - WSO2, Inc. http://wso2.com
email : nuw...@wso2.com
Phone : +94 777 775 729
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Adding RNN to WSO2 Machine Learner

[Thamali Wijewardhana]

Hi Srinath,

We have decided to  implement only classification first. Once we complete
the classification, we hope to do next value prediction too.
We are basically trying to implement a program to make sure that the
deeplearning4j library we are using is compatible with apache spark
pipeline. And also we are trying to demonstrate all the machine learning
steps with that program.

We are now using aclImdb sentiment analysis data set to verify the accuracy
of the RNN model we create.

Thanks
Thamali


On Wed, Mar 2, 2016 at 10:38 AM, Srinath Perera  wrote:

> Hi Thamali,
>
>
>1. RNN can do both classification and predict next value. Are we
>trying to do both?
>2. When Upul played with it, he had trouble getting deeplearning4j
>implementation work with predict next value scenario. Is it fixed?
>3. What are the data sets we will use to verify the accuracy of RNN
>after integration?
>
>
> --Srinath
>
> On Tue, Mar 1, 2016 at 3:44 PM, Thamali Wijewardhana 
> wrote:
>
>> Hi,
>>
>> Currently we are working on a project to add Recurrent Neural
>> Network(RNN) algorithm to machine learner. RNN is one of deep learning
>> algorithms with record breaking accuracy. For more information on RNN
>> please refer link[1].
>>
>> We have decided to use deeplearning4j which is an open source deep
>> learning library scalable on spark and Hadoop.
>>
>> Since there is a plan to add spark pipeline to machine Learner, we have
>> decided to use spark pipeline concept to our project.
>>
>> I have designed an architecture for the RNN implementation.
>>
>> This architecture is developed to be compatible with spark pipeline.
>>
>> Data set is taken in csv format and then it is converted to spark data
>> frame since apache spark works mostly with data frames.
>>
>> Next step is a transformer which is needed to tokenize the sequential
>> data. A tokenizer is basically used for take a sequence of data and break
>> it into individual units. For example, it can be used to break the words in
>> a sentence to words.
>>
>> Next step is again a transformer used to converts tokens to vectors. This
>> must be done because the features should be added to spark pipeline in
>> org.apache.spark.mllib.linlag.VectorUDT format.
>>
>> Next, the transformed data are fed to the data set iterator. This is an
>> object of a class which implement
>> org.deeplearning4j.datasets.iterator.DataSetIterator. The dataset iterator
>> traverses through a data set and prepares data for neural networks.
>>
>> Next component is the RNN algorithm model which is an estimator. The
>> iterated data from data set iterator is fed to RNN and a model is
>> generated. Then this model can be used for predictions.
>>
>> We have decided to complete this project in two steps :
>>
>>
>>-
>>
>>First create a spark pipeline program containing the steps in machine
>>learner(uploading dataset, generate model, calculating accuracy and
>>prediction) and check whether the project is feasible.
>>-
>>
>>Next add the algorithm to ML
>>
>> Currently we have almost completed the first step and now we are
>> collecting more data and tuning for hyper parameters.
>>
>> [1]
>> https://docs.google.com/document/d/1edg1fdKCYR7-B1oOLy2kon179GSs6x2Zx9oSRDn_NEU/edit
>>
>>
>>
>> ​
>>
>
>
>
> --
> 
> Srinath Perera, Ph.D.
>http://people.apache.org/~hemapani/
>http://srinathsview.blogspot.com/
>
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture