[Architecture] Generating a single Event per Message from our Products

As per meeting ( Paricipants: Sanjiva, Shankar, Sumedha, Anjana, Miyuru, Seshika, Suho, Nirmal, Nuwan) Currently we generate several events per message from our products. For example, when a message hits APIM, following events will be generated. 1. One from HTTP level 2. 1-2 from

Re: [Architecture] Maven project naming convention

> > Yes, this is an option but I don't see any big differences from this to > the approach we currently use. I agree there is no functional advantage, but it will make the pom looks nicer with less redundancy, subjectively speaking. At least shouldn't we keep the same convention across all poms

Re: [Architecture] [bpmn] Rewrite REST API with msf4j

Hi team, Please find my comments inline. On Fri, Mar 25, 2016 at 10:27 AM, Himasha Guruge wrote: > Hi All, > > I'm in the process of rewriting our existing BPMN REST API for C5, with > msf4j in place. While checking the used annotations in the code, I came > across the use

Re: [Architecture] Incremental Processing Support in DAS

Actually currently in ESB-analytics also we are using a similar mechanism. In there we keep 5 tables for each time unites(second, minute, hour, day, month) and we have windowOffset in correspondent to it time unite like in minute-60, hour- 3600 etc. We are only storing min, max, sum and count

Re: [Architecture] [C5] Artifact deployment coordination support for C5

Hi Nandika, On Thu, Mar 24, 2016 at 7:43 PM, Nandika Jayawardana wrote: > What about providing a REST API from each node of the cluster which can be > used to query deployed artifact information in addition to JMS publishing. > There will be admin services later on that will

Re: [Architecture] Defining UX Pasona for the Platform

Hi Srinath, We are planning to capture existing personas on each wso2 products, IMHO then we will be able to refine them easily. we'll start separate thread for that. Cheers, *Dakshika Jayathilaka* PMC Member & Committer of Apache Stratos Senior Software Engineer WSO2, Inc.

Re: [Architecture] [C5] Artifact deployment coordination support for C5

Hi, In JMS perspective we can use a topic. But if we use a topic message is only delivered to subscribers listening at that moment only. Say we have N1, N2 and N3 nodes. N1, N2 is up when the artifact is deployed to N1, thus N2 is a subscriber. It gets the artifact. Now N3 joins the cluster. In

Re: [Architecture] [ESB] Support break statement like functionality with iterate mediator

Hi All. In the existing implementation we have max and min number of message configuration. When the number of response messages received reaches this number, the aggregation will be completed but we don't have way to identified the request failure inside the aggregate mediator and break the

Re: [Architecture] [C5] Artifact deployment coordination support for C5

On Wed, Mar 23, 2016 at 10:45 AM, KasunG Gajasinghe wrote: > Hi, > > Given several issues we discovered with automatic artifact synchronization > with DepSync in C4, we have discussed how to approach this problem in C5. > > We are thinking of not doing the automated artifact

Re: [Architecture] Adding RNN to WSO2 Machine Learner

Hi all, One of the most important obstacles in machine learning and deep learning is getting data into a format that neural nets can understand. Neural nets understand vectors. Therefore, vectorization is an important part in building neural network algorithms. Canova is a Vectorization library

Re: [Architecture] RFC:Security Challenges in Analytics Story

As per meeting ( Sanjiva, Shankar, Sumedha, Anjana, Miyuru, Seshika, Suho, Nirmal, Nuwan) We need APIM and IOT Server to be able to publish events as "system user", but ask DAS to place data under Ann's ( related user) account. We need Devices to be able to *directly* send a event to DAS with an

Re: [Architecture] Incremental Processing Support in DAS

@sachith that approach requires us to parse the queries to see if it uses aggregate functions, doesn't it? On Fri, Mar 25, 2016 at 8:47 AM, Inosh Goonewardena wrote: > > > On Thu, Mar 24, 2016 at 9:56 PM, Sachith Withana wrote: > >> Hi Inosh, >> >> We wouldn't

Re: [Architecture] Log Analyzer Analytics for API Manager

As per meeting ( Sanjiva, Shankar, Sumedha, Anjana, Miyuru, Seshika, Suho, Nirmal, Nuwan, Malith, SajithP, SameeraR .. pls add) 1. Drop #3 2. Other than at above use cases are OK. 3. Need to integrate this with APIM Admin dashboard and for ESB analytics, it can run with

Re: [Architecture] APIM setup in k8s

Hi Lakmal, On Fri, Mar 25, 2016 at 8:33 AM, Lakmal Warusawithana wrote: > > Do we tested $subject? How we are going to update gateways when new apis > created? > We have $subject here [1]. However we still did not do any tests on synchronizing APIs generated by the publisher

Re: [Architecture] Maven project naming convention

On Fri, Mar 25, 2016 at 12:44 AM, Manuranga Perera wrote: > Our current naming convention [1] looks like: > org.wso2.carbon > org.wso2.carbon.core > > But I believe it should be like (and maven recommended [2]): > org.wso2.carbon > kernel > > (I see the second patten in some C5

Re: [Architecture] Defining UX Pasona for the Platform

Ping, any updates? On Fri, Feb 26, 2016 at 5:42 AM, Dakshika Jayathilaka wrote: > Sure, > > We'll try to come up with main personas that we can use. IMHO there may be > key personas + sub levels. I'll schedule a meeting once we complete first > draft. > > Thank you, > >

Re: [Architecture] Incremental Processing Support in DAS

On Thu, Mar 24, 2016 at 9:56 PM, Sachith Withana wrote: > Hi Inosh, > > We wouldn't have to do that IMO. > > We can persist the total aggregate value upto currentTimeWindow - > WindowOffset, along with the previous time window aggregation meta data as > well ( count, sum in the

[Architecture] APIM setup in k8s

Hi Imesh and team, Do we tested $subject? How we are going to update gateways when new apis created? thanks -- Lakmal Warusawithana Director - Cloud Architecture; WSO2 Inc. Mobile : +94714289692 Blog : http://lakmalsview.blogspot.com/ ___

Re: [Architecture] Incremental Processing Support in DAS

Hi Inosh, We wouldn't have to do that IMO. We can persist the total aggregate value upto currentTimeWindow - WindowOffset, along with the previous time window aggregation meta data as well ( count, sum in the average aggregation case). The previous total wouldn't be calculated again, it's the

Re: [Architecture] [IoTS] Profiling for IoTS

I'm also +1 to have fully fledged profile as the default one. Because other proposed profiles might only useful for specific cases such as in a deployment. But as we promoting IoT Server as a fully capable product to work with IoT, I believe having fully fledged profile as the default one will be

Re: [Architecture] Incremental Processing Support in DAS

Hi Sachith, *WindowOffset*: > > Events might arrive late that would belong to a previous processed time > window. To account to that, we have added an optional parameter that would > allow to > process immediately previous time windows as well ( acts like a buffer). > ex: If this is set to 1,

[Architecture] Fwd: [Update] C5 Permission model for resources

C5 Permission model for resources *When* Mon Mar 28, 2016 11am – 12pm Colombo *Where* LK 1st Floor Meeting Room Ext:1005150 (map ) *Video call* https://plus.google.com/hangouts/_/wso2.com/c5-permission

[Architecture] Maven project naming convention

Our current naming convention [1] looks like: org.wso2.carbon org.wso2.carbon.core But I believe it should be like (and maven recommended [2]): org.wso2.carbon kernel (I see the second patten in some C5 based code [3]. So not sure which one is the convention.) One issue with the second approach

Re: [Architecture] What's "id" in carbon.yml, which is in Carbon 5 conf directory

So, when deploying multiple products, this "id" should be changed and each server must specify a unique id, right? (Otherwise how can we uniquely identify a server?) How can I access this "id" from my component. Is there an API? On Thu, Mar 24, 2016 at 6:30 PM, Aruna Karunarathna

Re: [Architecture] What's "id" in carbon.yml, which is in Carbon 5 conf directory

On Thu, Mar 24, 2016 at 6:23 PM, Isuru Perera wrote: > Hi, > > $subject? > > What are the reasons for adding it? Can I use that to identify a carbon > server uniquely? > > For example, in Carbon Metrics, I want to use some value to identify a > server (The source of metrics).

[Architecture] What's "id" in carbon.yml, which is in Carbon 5 conf directory

Hi, $subject? What are the reasons for adding it? Can I use that to identify a carbon server uniquely? For example, in Carbon Metrics, I want to use some value to identify a server (The source of metrics). So, I have used a configuration named source and I use hostname as the default value. Can

Re: [Architecture] [IoTS] Is it the right thing to package "analytics" scripts with a plugin implementation?

Hi Srinath, On Thu, Mar 24, 2016 at 1:48 PM, Srinath Perera wrote: > Hi All, > > Prabath, if analytics is not in the device type, where should it go? > IMO, we can have it as a separate project structure with a separate packaging as part of the underlying tooling platform.

[Architecture] [DEV] [DS] Alternatives for Gadget-Dashboard communication

Hi all, WRT the discussion had in the PC, in order to persist gagdets' state we thought of providing a API for gadgets to save and retrieve their states from the dashboard URL (via hash), instead of using existing pubsub model. Therefore I did a research on alternatives to implement the API for

Re: [Architecture] [C5] Artifact deployment coordination support for C5

On Thu, Mar 24, 2016 at 3:23 PM, Sameera Jayasoma wrote: > Btw, who are the consumers of these messages? User can develop their own > tools to monitor artifact deployment. > > We also can come with CEP query to analyze whether the artifact deployment > happened successfully in

Re: [Architecture] [C5] Artifact deployment coordination support for C5

Btw, who are the consumers of these messages? User can develop their own tools to monitor artifact deployment. We also can come with CEP query to analyze whether the artifact deployment happened successfully in a cluster and display this information in a dashboard. We can come up with a DAS

Re: [Architecture] [IoTS] Profiling for IoTS

Hi Ayyoob, It should be other way around where default profile has everything. When a user start the standalone server with "sh wso2server.sh", it should have everything complete their use case (Including MB and DAS features). And in real deployment user will use IoT server with only IoT related

Re: [Architecture] [Dev] [REST APIs][Analytics] GET request with a payload

Hi all, POST for filterings is not an issue for special cases, as document also clearly confirms. However, I think the decision has to be made on practical use cases. This use case doesn't looks like a complex one. As Ayoma mention, it is a good idea to implement two filters to include and

Re: [Architecture] [Dev] [REST APIs][Analytics] GET request with a payload

Hi Gimantha, I think the point made by Udara is valid. Anyways if the user wants to get a selected number of columns, the chances are it won't exceed the url limit. ( due to the that number being low). Thanks, Sachith On Thu, Mar 24, 2016 at 2:21 PM, Gimantha Bandara wrote:

Re: [Architecture] [Dev] [REST APIs][Analytics] GET request with a payload

Hi Sanjeewa, Thank you for the guidelines doc. The exact problem is discussed in 10.2 in the above document. We will be filtering the record values in each records by providing the required columns, so only those column values will be returned with each record. According to the document the POST

Re: [Architecture] Incremental Processing Support in DAS

Hi Srinath, Please find the comments inline. On Thu, Mar 24, 2016 at 11:39 AM, Srinath Perera wrote: > Hi Sachith, Anjana, > > +1 for the backend model. > > Are we handling the case when last run was done, 25 minutes of data is > processed. Basically, next run has to re-run

[Architecture] [IoTS] Profiling for IoTS

Hi All, In IoTS we have included MB(MQTT), DAS features within the pack to avoid initial configuration and these features are tightly bounded with IoT scenarios. However in a real deployment scenario we should not have these features installed by default but should allow the deployment to have

Re: [Architecture] C5 Permission model for resources

On Thu, Mar 24, 2016 at 1:56 PM, Manuranga Perera wrote: > As far as I know, the current model supports for code level permission. > What is our plan for protecting a resource? > How can we say a particular resource (governance artifact/a web > page/message broker topic) is

Re: [Architecture] Log Analyzer Analytics for API Manager

Hi Srinath, Yes, all the above mentioned use cases can be implemented with the logs available from APIM and ESB. -- Thanks, Miyuru Dayarathna Senior Technical Lead Mobile: +94713527783 Blog: http://miyurublog.blogspot.com On Thu, Mar 17, 2016 at 10:56 AM, Srinath Perera

Re: [Architecture] [IoTS] Is it the right thing to package "analytics" scripts with a plugin implementation?

Hi All, Prabath, if analytics is not in the device type, where should it go? I do not agree with your observation that functional and non-functional parts should be separate. It helps to have it separate if non-functional parts can be reused with other devices. However, that is not the case

Re: [Architecture] DAS Lucene query AllowLeadingWildcard

Thanks Gimantha. On Thu, Mar 24, 2016 at 12:39 PM, Gimantha Bandara wrote: > Yes. It can be any special character*. *But we use : to match a field to > a value. Like, > > : > > On Thu, Mar 24, 2016 at 12:03 PM, Damith Wickramasinghe > wrote: > >> Hi

Re: [Architecture] [IoTS] Is it the right thing to package "analytics" scripts with a plugin implementation?

Hi Sumedha, IMO, the device management plug-in layer does show what features can be controlled, what device info to be captured, etc at a somewhat satisfactory level already. If this is not sufficient, we need to think further and see what needs to be added at a functional level. Also, I agree

Re: [Architecture] DAS Lucene query AllowLeadingWildcard

Yes. It can be any special character*. *But we use : to match a field to a value. Like, : On Thu, Mar 24, 2016 at 12:03 PM, Damith Wickramasinghe wrote: > Hi Gimantha, > > Great. One question. Does it can be any special character. ? > > Regards, > Damith. > > On Thu, Mar

Re: [Architecture] [WSO2 Docker Images] Creating Base Docker Images for WSO2 Products

On Thu, Mar 24, 2016 at 11:05 AM, Chamila De Alwis wrote: > >> The "base" images can be started and run in the standalone mode, although > not production readily. Usually base images are used to collect common > configurations, not runnable products. However it's naming issue,

Re: [Architecture] [IoTS] Is it the right thing to package "analytics" scripts with a plugin implementation?

Hi Suho, On Thu, Mar 24, 2016 at 12:21 AM, Sriskandarajah Suhothayan wrote: > We already have Dev Studio support for "analytics" artifacts. > > In this case it depends, If the plugin developer believes that he his > sensor need some specialised analytics and charts then we should

Re: [Architecture] [IoTS] Is it the right thing to package "analytics" scripts with a plugin implementation?

Hi Ruwan, Analytics is indeed a primary requirement in any IoT "eco-system". There's no doubt whatsoever about it. In fact, analytics is what inspires folks to achieve operational efficiency and all that stuff end-to-end with proper use of monitoring against organization-specific KPIs, as you

Re: [Architecture] Session Affinity Alternatives

Hi Frank, Sorry, maybe we have not thought this through. When I said "state transfer" I mean "full session state in the message (REST-style)" Q1: What can we do at the platform level to support "state transfer"? Q1. Case 1: for stateful Apps and Services written by end users - end user can

Re: [Architecture] [Dev] [REST APIs][Analytics] GET request with a payload

Hi, IMO by using POST to do a GET we are violating REST principals and may confuse users. You can see a query type API in [1] used by StackOverflow API. According to [2], it seems about 2000 characters are allowed most browsers which is a pretty large number where a normal GET request is

Re: [Architecture] DAS Lucene query AllowLeadingWildcard

Hi Gimantha, Great. One question. Does it can be any special character. ? Regards, Damith. On Thu, Mar 24, 2016 at 11:49 AM, Gimantha Bandara wrote: > Hi Damith, > > If the "roles" field contains comma separated values, you can simply > search for the specific role using

Re: [Architecture] DAS Lucene query AllowLeadingWildcard

Hi Damith, If the "roles" field contains comma separated values, you can simply search for the specific role using the following query "roles : role1" Lucene has an analysis process which takes place before indexing. So the field values will be tokenized into terms(Text fields are split

Re: [Architecture] Incremental Processing Support in DAS

Hi Sachith, Anjana, +1 for the backend model. Are we handling the case when last run was done, 25 minutes of data is processed. Basically, next run has to re-run last hour and update the value. When does one hour counting starts? is it from the moment server starts? That will be probabilistic

Re: [Architecture] [Dev] [REST APIs][Analytics] GET request with a payload

Thank you for your suggestions. We cannot exactly say that the number of columns/fields user will need. It depends on how many fields a table has and how many the user want to get from that table. So the url length might exceed. If so, we will have to go with a new method with POST as it seems the

[Architecture] DAS Lucene query AllowLeadingWildcard

Hi all, I have a column which contains roles as a comma separated string. eg:- role1,role2,role3 I need to find records which matches to specific role. As I checked theres no String contains function. But there is wildcard support[1]. To be able to work for my usecase wildcard should be of type