[Architecture] [WSO2 MB] MQTT retained message feature

[Asanka Abeyweera]

Hi all,

I started working on implementing the MQTT retained message feature on WSO2
MB 3.0.0

*Feature Definition*

In MQTT a publisher can set retain flag in any message it sends to the
broker. By setting the retain flag, the message is kept in the broker even
after delivering to all existing subscription, so when a new subscribers
connect to the broker, they get all the relevant retained messages.

*Proposed Design*

A separate table/column-family will be used to store the last published
message for a topic. This table will be updated every time a message is
published with the retained flag set.

When a new client subscribe to a topic, the table is looked up for matching
topics and retained messages are sent to the new subscriber. This is done
only when initializing the subscription.

Please share your thoughts on this.

-- 
Asanka Abeyweera
Software Engineer
WSO2 Inc.

Phone: +94 712228648
Blog: a5anka.github.io
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

[Architecture] Login Interceptor Interface of API Manager

[Lakshman Udayakantha]

Hi All,

I am working on [1]. In that I have to implement the application and
subscription sharing capability based on the organisation of the login
user. I have to create a Login interceptor interface to determine the
organisation of login user. We have to extract the organisation identifier
from login response using login interceptor. below is a proposed login
interceptor interface.

public interface LoginInterceptor {

public String getOrganization(String response);

}

When using WSO2 identity server we can add organisation as an attribute to
SAML response and we can extract it in the implementation of the
interceptor in API Manager side. Appreciate your thought on this.

[1]https://redmine.wso2.com/issues/2917

-- 
Lakshman Udayakantha
WSO2 Inc. www.wso2.com
lean.enterprise.middleware
Mobile: *0711241005*
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Should the OPTIONS call be terminated at the API gateway.

[Nuwan Dias]

On Tue, Jan 13, 2015 at 11:40 AM, Joseph Fonseka  wrote:

>
> Hi
>
> Currently in API Manager we send the OPTIONS call to the back-end server
> to respond. But if the API does not expose all the HTTP methods in
> back-end, OPTIONS call response will be invalid.
>
> 1. Should we respond to the OPTIONS call from the gateway with the Allow
> header including only the HTTP methods exposed by the API.
>
> 2. If we are doing above should we give an option to the user to override
> in case if he want to send it to back-end.
>

Whether the OPTIONS call goes to the back-end or not is dominated by the
Max-Forwards header. According to the spec [1], it seems like the request
should be forwarded when the Max-Forwards header is not present.

>
> 3. Is there a valid requirement to show the OPTIONS method in API Design
> phase, shouldn't it be enabled by default.
>
+1

[1] - http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html

>
> Thanks
> Jo
>
> Related
> https://wso2.org/jira/browse/APIMANAGER-1704
>
>
>
>
>
>
>
> --
> *Joseph Fonseka*
>  WSO2 Inc.; http://wso2.com
> lean.enterprise.middleware
>
> mobile: +94 772 512 430
> skype: jpfonseka
>
> * *
>
>


-- 
Nuwan Dias

Associate Tech Lead - WSO2, Inc. http://wso2.com
email : nuw...@wso2.com
Phone : +94 777 775 729
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Meeting Note : Implement a Pluggable Version Strategy feature in APIManager

[Dinesh J Weerakkody]

Hi All,

I have tested the above proposed method with a fresh APIM 1.8 pack. We can
successfully create and invoke APIs as follows.

[image: Inline image 1]

Here what we do is simply add the version number to context. Also I have
removed version from the resulting synapse api xml file (smiler to default
version) otherwise we have to send version number after the context (Ex:
http://localhost:8280/1.0.0/docman/*1.0.0*/add).

Now we can invoke this API as follows since context is matching to request
url and this is default version.
http://localhost:8280/1.0.0/docman/add

With the above solution system doesn't allow us to create a new version of
the API. Currently synapse doesn't allow to use deferent contexts with same
API name.
Ex:
Name : DOCMan
Context : */1.0.0/docman*
Version: 1.0.0

Name : DOCMan
Context :* /2.0.0/docman*
Version: 2.0.0

We used a template as a context to overcome this situation.

[image: Inline image 2]

With this solution context will be same among all versions. We resolve the
context template when API is initializing in synapse runtime. There is
another issue came up with this solution. Synapse doesn't allow to use
version attribute in API xml without a version-type (version strategy)
attribute. URL is the only value that we can use with the version-type
attribute and when we invoke api we must send version after the context if
we use URL as the version strategy. We can introduce a new version strategy
to overcome this situation. For the time been I have disabled the
validation.

After above mentioned changes we can successfully deploy multiple versions
of the same API. But when we try to invoke the APIs we faced another issue.
When the request comes to APIM side there is a problem in authentication
handler.

We use context to check whether access is granted to the given token or
not. In APIM side tables we have context template but in runtime we send
resolved context. So there will be no matching access grants and request
will be rejected.

We have two solutions to overcome this issue.

   - Have a separate column in APIM database to store the resolved context
   (Use in runtime)
   - Add a new attribute to RXT to store context template (This is used to
   display the context template in UI when editing the API)

I think that the second option is better than adding new columns to the
tables.

Please share your suggestions or thoughts on this.




Thanks

*Dinesh J. Weerakkody*
Software Engineer
WSO2 Inc.
lean | enterprise | middleware
M : +94 727 361788 | E : dine...@wso2.com | W : www.wso2.com

On Tue, Dec 23, 2014 at 5:43 PM, Dinesh J Weerakkody 
wrote:

> *Participants*
> APIM Team : Sumedha, Nuwan, Sanjeewa, Jo, Roshan, Lakshman, Dinesh
> ESB Team : Kasun
>
> *Redmine*
> https://redmine.wso2.com/issues/153
>
> We have discussed the following options as solutions to implement a
> pluggable version strategy for APIM manager product.
>
> *Option 1 : include version to context path*
> We allow users to define API context as a template in API create UI.
> Context = /api/{uri.var.version}/anytext
> Version = 1.0.0
>
> Then we generate the context path based on a the template when creating
> the synapse xml and use default version-type (empty).
>
> *Current version*
> http://ws.apache.org/ns/synapse";
> name="admin-API"
> context="/api”
> version="1.0.0"
> *version-type="url"*>
>
> *Proposed version*
> http://ws.apache.org/ns/synapse";
> name="admin-API"
> *context="/api/1.0.0"*
> version="1.0.0">
>
> Please note that version appended to context while version tag keeping as
> it is. If user does not include the {uri.var.version}, current process will
> be used.
>
> Ex:
> context="/1.0.0/api"
> context="/gov/1.0.0/api"
> context="/gov/1.0.0/api2"
>
> *Option 2 : Match APIs based on configured context pattern*
> http://ws.apache.org/ns/synapse";
> name="admin-API"
> context-pattern=”{uri.var.context}/{uri.var.version}”
> context="/api”
> version="1.0.0"
> version-type="url">
>
> Ex:
> {uri.var.version}/{uri.var.context}= http://localhost:8280/1.0.0/api
> {uri.var.context}/{uri.var.version}= http://localhost:8280/api/1.0.0/
>
> In synapse level we have to uniquely identify a API with the context and
> version. With second option, it will be difficult and can be lead to
> conflicts If user define an API with context=”1.0.0” version=”api” while
> having an API with context=”api” version=”1.0.0”. So we have decided to
> drop the second option and throughly test the first option.
>
> Please share your suggestions or thoughts on this.
>
>
> Thanks
>
> *Dinesh J. Weerakkody*
> Software Engineer
> WSO2 Inc.
> lean | enterprise | middleware
> M : +94 727 361788 | E : dine...@wso2.com | W : www.wso2.com
>
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Metrics Visualization: Using Kibana & Elasticsearch

[Chanaka Jayasena]

We have started working on new version of UES. It's still in the initial
stages. UES older version enables users to create custom dashboards by
importing gadgets  ( visualization panels) from Enterprise Store. The new
version is targeting to improve the older one. But we haven't consider the
possibility of extending Kibana and going forward with it. Our product will
be dealing with the raw data set rather than metrics. Right now we haven't
considered the implementation in detail. But we will take account the
possibility to use elastic search in our queries.

thanks,
Chanaka

On Mon, Jan 12, 2015 at 6:48 PM, Isuru Perera  wrote:

> Hi all,
>
> As part of Metrics [1] integration with WSO2 Platform [2], we are planning
> to have a separate UI for Metrics visualization.
>
> One of the main requirements is that the UI application should be
> standalone.There are Metrics reporters [3] for Ganglia & Graphite, which
> require to setup separate servers.
>
> Since we need a UI application, I looked at current work on such
> visualization applications.
>
> I mainly looked at Kibana [4] & Grafana [5]. Both applications are based
> on HTML, JavaScript, etc. I could run both in WSO2 Application Server
> without any issue.
>
> I chose Kibana, as it supports Elasticsearch and we can run Elasticsearch
> as an embedded server [6]. Elasticsearch also provides a reporter for
> Metrics [7].
>
> Grafana also looks good and as I found out, we need to write a custom data
> source to provide the Metrics data.
>
> Using mentioned software, I was able to run Kibana in a standalone
> application and visualize some sample Metrics.
>
> This was just a prototype and I will create a proper web application. With
> Kibana, we can easily create dashboards depending on our requirements. I'm
> going down this path, as we can quickly create dashboards and identify
> exact visualizations we need. Since we are in the initial stages of Metrics
> integration, we have to see how we can use the data provided by Metrics.
>
> With Metrics Reporters, each metric will be reported with the configured
> time period and there will be general statistical information, such as min,
> max, mean, percentiles etc for Timers. (See JSON format for metrics in
> README of [7]). We also need to figure out how we can visualize these data.
>
> I think we can use Kibana initially for our visualization purposes. All of
> the mentioned software and dependencies are based on Apache License.
>
> Only concern here would be the use of an embedded Elasticsearch server.
> IMO, Elasticsearch provides good features such as indexing data and smooth
> integration with Kibana.
>
> As I mentioned earlier, I would like to think of this as a temporary
> approach and we can create a separate dashboard once we have solid
> requirements. There is another work going on to create custom dashboards
> [8] and we can also make use of that later.
>
> We have not started the Metrics integration with products yet and we might
> have more concerns when we start to integrate.
>
> Please share your suggestions.
>
> Thanks!
>
> Best Regards,
>
> [1] https://dropwizard.github.io/metrics/
> [2] http://markmail.org/message/akqyreopmxhcd6hx
> [3]
> https://dropwizard.github.io/metrics/3.1.0/manual/core/#other-reporters
> [4] http://www.elasticsearch.org/overview/kibana/
> [5] http://grafana.org/
> [6]
> http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/client.html
> [7] https://github.com/elasticsearch/elasticsearch-metrics-reporter-java
> [8] http://markmail.org/message/74ogpnup5uqfqbps
>
>
> --
> Isuru Perera
> Senior Software Engineer | WSO2, Inc. | http://wso2.com/
> Lean . Enterprise . Middleware
>
> about.me/chrishantha
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
Chanaka Jayasena
Senior Software Engineer; WSO2, Inc.;  http://wso2.com/
email: chan...@wso2.com; cell: +94 77 785 5565
blog: http://chanaka3d.blogspot.com
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Metrics Visualization: Using Kibana & Elasticsearch

[Cyril Rognon]

Hi Isuru,

using ES+Kibana is a very good idea. We also are on the way to provide some
kibana dashboard to the devops of one of our client.
Why would you need some embedded ElasticSearch ? is it a security issue ?

For the architecture list, the elastic suite is very nice and could provide
a nice solution to provide rich dashboards. It would be very simple to
target ES as a stream store to benefit from kibana time-series dashboard.
We are thinking of using the complete ELK stack (adding logtash as well) to
provide easy admin  stats dashboard to multiple wso2 product, gaining the
fact is that it can also aggregate data from other sources than wos2 stat
(using logtash to gather other logs).

The apache licence of the whole products is something that makes it
possible.

Cheers,
Cyril



> -- Forwarded message --
> From: Isuru Perera 
> Date: Mon, Jan 12, 2015 at 2:18 PM
> Subject: [Architecture] Metrics Visualization: Using Kibana & Elasticsearch
> To: Srinath Perera , WSO2 Architecture <
> architecture@wso2.org>
>
>
> Hi all,
>
> As part of Metrics [1] integration with WSO2 Platform [2], we are planning
> to have a separate UI for Metrics visualization.
>
> One of the main requirements is that the UI application should be
> standalone.There are Metrics reporters [3] for Ganglia & Graphite, which
> require to setup separate servers.
>
> Since we need a UI application, I looked at current work on such
> visualization applications.
>
> I mainly looked at Kibana [4] & Grafana [5]. Both applications are based
> on HTML, JavaScript, etc. I could run both in WSO2 Application Server
> without any issue.
>
> I chose Kibana, as it supports Elasticsearch and we can run Elasticsearch
> as an embedded server [6]. Elasticsearch also provides a reporter for
> Metrics [7].
>
> Grafana also looks good and as I found out, we need to write a custom data
> source to provide the Metrics data.
>
> Using mentioned software, I was able to run Kibana in a standalone
> application and visualize some sample Metrics.
>
> This was just a prototype and I will create a proper web application. With
> Kibana, we can easily create dashboards depending on our requirements. I'm
> going down this path, as we can quickly create dashboards and identify
> exact visualizations we need. Since we are in the initial stages of Metrics
> integration, we have to see how we can use the data provided by Metrics.
>
> With Metrics Reporters, each metric will be reported with the configured
> time period and there will be general statistical information, such as min,
> max, mean, percentiles etc for Timers. (See JSON format for metrics in
> README of [7]). We also need to figure out how we can visualize these data.
>
> I think we can use Kibana initially for our visualization purposes. All of
> the mentioned software and dependencies are based on Apache License.
>
> Only concern here would be the use of an embedded Elasticsearch server.
> IMO, Elasticsearch provides good features such as indexing data and smooth
> integration with Kibana.
>
> As I mentioned earlier, I would like to think of this as a temporary
> approach and we can create a separate dashboard once we have solid
> requirements. There is another work going on to create custom dashboards
> [8] and we can also make use of that later.
>
> We have not started the Metrics integration with products yet and we might
> have more concerns when we start to integrate.
>
> Please share your suggestions.
>
> Thanks!
>
> Best Regards,
>
> [1] https://dropwizard.github.io/metrics/
> [2] http://markmail.org/message/akqyreopmxhcd6hx
> [3]
> https://dropwizard.github.io/metrics/3.1.0/manual/core/#other-reporters
> [4] http://www.elasticsearch.org/overview/kibana/
> [5] http://grafana.org/
> [6]
> http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/client.html
> [7] https://github.com/elasticsearch/elasticsearch-metrics-reporter-java
> [8] http://markmail.org/message/74ogpnup5uqfqbps
>
>
> --
> Isuru Perera
> Senior Software Engineer | WSO2, Inc. | http://wso2.com/
> Lean . Enterprise . Middleware
>
> about.me/chrishantha
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>
>
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

[Architecture] [BPS] Adding Form / Response generation functionality to HumanTask-explorer webapp

[Milinda Perera]

Hi,

We are planning to add form generation to HumanTask-explorer webapp [1].

*Current situation as follows:*
Currently HumanTask author create separate .jsp files and embed them in
Main > Human Tasks > List in management console to display task
information, and assignee work form

For Example, in ClaimsApprovalTask sample[2] ApproveClaim-input.jsp
ApproveClaim-input.jsp - Display inputs to the task
ApproveClaim-output.jsp - Generate output message

Since this method increase workload and inefficient from the side of the
humantask author, we decided to add form generation to the
HumanTask-explorer. This proposed method reduces effort needed creating
humantask.

*New Improvement :*

As suggested in [1] configuration need to done by the humantask author as
follows:

"The task author has to define which xml elements in the input message
should be displayed under which label. And which xml elements in the output
message should be filled with which form field. Task authors can provide
this information under the  element"





[Label to display for the input element]
[xpath to get the value from the input
message]

...



[Label to display for the form field]
[xpath of the element in the output message to be
filled with this form field]

...





Then the humantask webapp can generate the output message by mapping form
fields according to xpath expressions.

Following diagram shows architecture of this new improvement:


​

1 - retrieve renderings using HumanTaskClientAPIAdmin and generate from.
2 - Create response using ResponseCreator which create soap message when
user submit form.
3 - Send response

We are planning to use Membrane SOA Model[3] library to generate output
message (Response) using humantask wsdl

Suggestions are highly appreciated.

[1] mail : [Architecture] Jaggery based UI for WS-Human Tasks
[2]
https://github.com/milindaperera/product-bps/tree/master/modules/samples/product/src/main/resources/humantask/ClaimsApprovalTask
[3] http://www.membrane-soa.org/soa-model/

Thanks,
Milinda

-- 
Milinda Perera
Software Engineer;
WSO2 Inc. http://wso2.com ,
Mobile: (+94) 714 115 032
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

[Architecture] Metrics Visualization: Using Kibana & Elasticsearch

[Isuru Perera]

Hi all,

As part of Metrics [1] integration with WSO2 Platform [2], we are planning
to have a separate UI for Metrics visualization.

One of the main requirements is that the UI application should be
standalone.There are Metrics reporters [3] for Ganglia & Graphite, which
require to setup separate servers.

Since we need a UI application, I looked at current work on such
visualization applications.

I mainly looked at Kibana [4] & Grafana [5]. Both applications are based on
HTML, JavaScript, etc. I could run both in WSO2 Application Server without
any issue.

I chose Kibana, as it supports Elasticsearch and we can run Elasticsearch
as an embedded server [6]. Elasticsearch also provides a reporter for
Metrics [7].

Grafana also looks good and as I found out, we need to write a custom data
source to provide the Metrics data.

Using mentioned software, I was able to run Kibana in a standalone
application and visualize some sample Metrics.

This was just a prototype and I will create a proper web application. With
Kibana, we can easily create dashboards depending on our requirements. I'm
going down this path, as we can quickly create dashboards and identify
exact visualizations we need. Since we are in the initial stages of Metrics
integration, we have to see how we can use the data provided by Metrics.

With Metrics Reporters, each metric will be reported with the configured
time period and there will be general statistical information, such as min,
max, mean, percentiles etc for Timers. (See JSON format for metrics in
README of [7]). We also need to figure out how we can visualize these data.

I think we can use Kibana initially for our visualization purposes. All of
the mentioned software and dependencies are based on Apache License.

Only concern here would be the use of an embedded Elasticsearch server.
IMO, Elasticsearch provides good features such as indexing data and smooth
integration with Kibana.

As I mentioned earlier, I would like to think of this as a temporary
approach and we can create a separate dashboard once we have solid
requirements. There is another work going on to create custom dashboards
[8] and we can also make use of that later.

We have not started the Metrics integration with products yet and we might
have more concerns when we start to integrate.

Please share your suggestions.

Thanks!

Best Regards,

[1] https://dropwizard.github.io/metrics/
[2] http://markmail.org/message/akqyreopmxhcd6hx
[3] https://dropwizard.github.io/metrics/3.1.0/manual/core/#other-reporters
[4] http://www.elasticsearch.org/overview/kibana/
[5] http://grafana.org/
[6]
http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/client.html
[7] https://github.com/elasticsearch/elasticsearch-metrics-reporter-java
[8] http://markmail.org/message/74ogpnup5uqfqbps


-- 
Isuru Perera
Senior Software Engineer | WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware

about.me/chrishantha
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture