Re: [Architecture] Securing Stream Processor API through OAuth2

[Mohanadarshan Vivekanandalingam]

On Mon, Oct 30, 2017 at 12:50 PM, Damith Wickramasinghe 
wrote:

> Hi Niveathika,
>
> Are we securing event simulator apis as well ?
>

We have to secure that as well. IMO, all the core APIs need to be secured.

Thanks,
Mohan



>
> Regards,
> Damith
>
> On Mon, Oct 30, 2017 at 12:38 PM, Niveathika Rajendran <
> niveath...@wso2.com> wrote:
>
>> Hi all,
>>
>> The use case in accessing Stream Processor API's are as follows,
>>
>> 1. Dashboard front end APIs
>>
>> These are API's which the user users to access dashboards he/she will
>> create.
>>
>> These will be protected by using an Authentication API through which the
>> access token obtained by the login will be split into 2 and saved as
>> cookies. Authentication API will act as a proxy for the IdPClient OSGi
>> service.
>>
>> 2. Dashboard back end API's
>>
>> These will use the IdPClient OSGi service to get the access tokens using
>> client credential grant type which can be used to access other API's with
>> Bearer authorization headers.
>>
>>
>> 2. Databridge
>>
>> Here, the data bridge authentication is only done through basic
>> authentication. Oauth2 token validation is mocked through passing token
>> requests using password grant type. This is because the events will be sent
>> with Basic authorization headers and not with Bearer headers
>>
>>
>> For more info in SP IdP integration please refer[1].
>>
>> @Identity-Team, Could you provide feedback on the mechanisms used in
>> securing API's.
>>
>> [1] [Architecture] Securing Product Apis and Product artifacts in Stream
>> Processor
>>
>> --
>> Best Regards,
>> *Niveathika Rajendran,*
>> *Software Engineer.*
>> *Mobile : +94 077 903 7536 <+94%2077%20903%207536>*
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "WSO2 Engineering Group" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to engineering-group+unsubscr...@wso2.com.
>> For more options, visit https://groups.google.com/a/wso2.com/d/optout.
>>
>
>
>
> --
> Senior Software Engineer
> WSO2 Inc.; http://wso2.com
> 
> lean.enterprise.middleware
>
> mobile: *+94728671315 <+94%2072%20867%201315>*
>
>


-- 
*V. Mohanadarshan*
*Technical Lead,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Secure MQTT Receiver for DAS

[Mohanadarshan Vivekanandalingam]

On Sat, Dec 16, 2017 at 11:42 PM, Rasika Perera  wrote:

> Hi Kalai and All,
>
> As Sumedha mentioned you can refer, OAuth Protected MQTT extension in [1]
> for the IoT Server as well.
>
> If I understand you correctly, you are going to use DAS's carbon.xml
> values as the default trust store. If anyone interested, they can point a
> custom trust store.
>
> Generally, trust stores are used to store certificates from CAs which is
> used to verify certificate presented by the client in SSL Connection. With
> the current approach, having them in a central place(aka. carbon.xml) would
> ease the server config process. AFAIK we don't maintain multiple trust
> stores for a single server. On the other-hand, Introducing new
> configurations for additional trust stores would impact negatively on the
> support and maintainability aspects of the product. Thus, unless there's a
> huge use case for a custom trust store, I am -1 for introducing this new
> configuration.
>
> [1] https://github.com/wso2/carbon-device-mgt-plugins/tree/
> master/components/extensions/mb-extensions/org.wso2.carbon.
> andes.extensions.device.mgt.mqtt.authorization
>

+1. Let's use the trust store defined in carbon.xml..

Thanks,
Mohan



>
> On Fri, Dec 15, 2017 at 3:02 PM, Sumedha Rubasinghe 
> wrote:
>
>> There is an OAuth2 token based topic protector done for IoT scenarios.
>>
>> On Thu, Dec 14, 2017 at 5:25 PM, Kalaiyarasi Ganeshalingam <
>> kalaiyar...@wso2.com> wrote:
>>
>>> Hi all,
>>>
>>> DAS already has MQTT Receiver but It is not enabled for secure MQTT
>>> Communication. So, now I am going to work on this feature to enable secure
>>> MQTT. In the Secure connection, the broker and the client talk over the
>>> SSL. Here, SSL provide a secure communication channel between a client and
>>> a server. For this implementation, I am going to get the following optional
>>> parameters from the user:
>>> tlsTruststoreLocation : the trustStore file path .
>>> tlsTruststorePassword : the password of truststore.
>>> tlsTruststoreType :  the trustStore type.
>>> tlsVersion : the standard name of the requested protocol.
>>>
>>> Please let me know if you have any suggestions on this?
>>>
>>> Regards,
>>> Kalaiyarasi Ganeshalingam
>>> Associate Software Engineer| WSO2
>>> WSO2 Inc : http://wso2.org
>>> 
>>> Tel:+94 076 6792895 <076%20679%202895>
>>> LinkedIn :www.linkedin.com/in/kalaiyarasiganeshalingam
>>> Blogs : https://kalaiyarasig.blogspot.com/ 
>>>
>>
>>
>>
>> --
>> /sumedha
>> m: +94 773017743 <+94%2077%20301%207743>
>> b :  bit.ly/sumedha
>>
>
>
>
> --
> With Regards,
>
> *Rasika Perera*
> Senior Software Engineer
> LinkedIn: http://lk.linkedin.com/in/rasika90
>
> 
>
> WSO2 Inc. www.wso2.com
> lean.enterprise.middleware
>



-- 
*V. Mohanadarshan*
*Technical Lead,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Cassandra Table Extension implementations

[Mohanadarshan Vivekanandalingam]

On Tue, Jan 16, 2018 at 10:02 AM, Tharindu Jayathilake 
wrote:

> Hi All,
> We are implementing a Cassandra table extension that enables the Siddhi
> developers to persist events in Cassandra stores.
>
> Following operations are capable through this extension.
> 1. Define a Cassandra table
> 2. Insert events into Cassandra table
> 3. Read events from Cassandra table
> 4. Check if a Cassandra table contains a given row
> 5. Delete records from Cassandra table
> 6. Update records from Cassandra table
> 7. Update or insert records from Cassandra table
>
> High-level architecture of the extension siddhi-cassandra-store would look
> as follows.
>
> ​
> ​
> Appreciate any suggestions about the extension.
> Thanks & Regards,
>
> *Tharindu Jayathilake*
> Intern Software Engineer
> WSO2, Inc. *http://wso2.com *
>
> *Phone:+94773898282 <+94%2077%20389%208282>*
> *LinkedIn: **https://www.linkedin.com/in/tharindujayathilake/
> *
> 
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "WSO2 Engineering Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to engineering-group+unsubscr...@wso2.com.
> For more options, visit https://groups.google.com/a/wso2.com/d/optout.
>



-- 
*V. Mohanadarshan*
*Technical Lead,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

[Architecture] [VOTE] Release of WSO2 Stream Processor 4.4.0 RC2

[Mohanadarshan Vivekanandalingam]

Hi All,

WSO2 Stream Processor team is pleased to announce the second release
candidate of WSO2 Stream Processor 4.4.0.

WSO2 Stream Processor is an open source embodiment of the WSO2 Analytics
platform, of which the real-time, incremental & intelligent data processing
capabilities let digital businesses create actionable business insights and
data products.

Please find the improvements and fixes related to this release:

   - siddhi
   

   - carbon-analytics-common
   

   - carbon-analytics
   

   - carbon-dashboards
   

   - analytics-solutions
   

   - product-sp
   


You can download the product distribution from here


The tag to be voted upon:
https://github.com/wso2/product-sp/releases/edit/v4.4.0-RC2

Please download, test the product and vote.

[+] Stable - go ahead and release
[-] Broken - do not release (explain why)

You can find the official documentation in
https://docs.wso2.com/display/SP440  

Thanks,
WSO2 Stream Processor Team

-- 
*V. Mohanadarshan* | Senior Technical Lead | WSO2 Inc.
 |
(M) 94-771117673 | (E) mo...@wso2.com |
GET INTEGRATION AGILE
Integration Agility for Digitally Driven Business
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [VOTE] Release of WSO2 Stream Processor 4.4.0 RC3

[Mohanadarshan Vivekanandalingam]

HI All,

We have found some issues in our testing cycle hence closing off the vote
and will be doing another RC release.

Thanks,
Mohan


On Thu, Apr 25, 2019 at 12:02 PM Minudika Malshan  wrote:

> Hi all,
>
>
> WSO2 Stream Processor team is pleased to announce the third release candidate
> of *WSO2 Stream Processor 4.4.0*.
>
>
> WSO2 Stream Processor is an open source embodiment of the WSO2 Analytics
> platform, of which the real-time, incremental & intelligent data processing
> capabilities let digital businesses create actionable business insights and
> data products.
>
>
> Please find the improvements and fixes related to this release:
>
> - siddhi
> 
>
> - carbon-analytics-common
> 
>
> - carbon-analytics
> 
>
> - carbon-dashboards
> 
>
> - analytics-solutions
> 
>
> - product-sp
> 
>
>
> You can download the product distribution from here
> 
>
>
> The tag to be voted upon:
> https://github.com/wso2/product-sp/releases/tag/v4.4.0-RC3
>
>
> Please download, test the product and vote.
>
>
> *[+] Stable - go ahead and release*
>
> *[-] Broken - do not release *(explain why)
>
>
>
> Thanks!
>
> *- WSO2 Stream Processor Team -*
>
>
> --
> *Minudika Gammanpila*
> Software Engineer - WSO2
>
> Email   :  minud...@wso2.com
> Mobile :  +94715659887
> Web :  http://wso2.com
>
>  
>


-- 
*V. Mohanadarshan* | Senior Technical Lead | WSO2 Inc.
 |
(M) 94-771117673 | (E) mo...@wso2.com |
GET INTEGRATION AGILE
Integration Agility for Digitally Driven Business
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [VOTE] Release of WSO2 Stream Processor 4.4.0 RC5

[Mohanadarshan Vivekanandalingam]

Closing the vote since we have found some corrupted artifacts in the
distribution. RC6 vote is called due to this.

Thanks,
Mohan


On Tue, May 14, 2019 at 12:53 PM Ramindu De Silva  wrote:

> Hi all,
>
> WSO2 Stream Processor team is pleased to announce the fifth release
> candidate of WSO2 Stream Processor 4.4.0.
>
> WSO2 Stream Processor is an open source embodiment of the WSO2 Analytics
> platform, of which the real-time, incremental & intelligent data processing
> capabilities let digital businesses create actionable business insights and
> data products.
>
> Please find the improvements and fixes related to this release:
>
>- siddhi
>
> 
>- carbon-analytics-common
>
> 
>- carbon-analytics
>
> 
>- carbon-dashboards
>
> 
>- analytics-solutions
>
> 
>- product-sp
>
> 
>
> You can download the product distribution from here
> 
>
> The tag to be voted upon:
> https://github.com/wso2/product-sp/releases/tag/v4.4.0-RC5
>
> Please download, test the product and vote.
>
> [+] Stable - go ahead and release
> [-] Broken - do not release (explain why)
>
> You can find the official documentation in
> https://docs.wso2.com/display/SP440
>
> Best Regards,
> WSO2 Stream Processor Team
> --
> *Ramindu De Silva*
> Associate Technical Lead
> WSO2 Inc.: http://wso2.com
> lean.enterprise.middleware
>
> email: ramin...@wso2.com 
> mob: +94 719678895
>
> --
> You received this message because you are subscribed to the Google Groups
> "Analytics Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to analytics-group+unsubscr...@wso2.com.
> To view this discussion on the web visit
> https://groups.google.com/a/wso2.com/d/msgid/analytics-group/CANLowD5-SNPAAxthqMkweUzu1n%2B4nT%2BiDDw1iy5LBvRjJdQfcw%40mail.gmail.com
> 
> .
>


-- 
*V. Mohanadarshan* | Senior Technical Lead | WSO2 Inc.
 |
(M) 94-771117673 | (E) mo...@wso2.com |
GET INTEGRATION AGILE
Integration Agility for Digitally Driven Business
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Adding new throttling implementation related configurations to API Manager configuration file

[Mohanadarshan Vivekanandalingam]

On Fri, Apr 1, 2016 at 6:05 PM, Sanjeewa Malalgoda 
wrote:

> Hi All,
>

Hi Sanjeewa,


> While working on new throttling implementation for API Manager we wanted
> to add some additional configurations to product.
> The intention of this mail is discuss about them and come to conclusion.
>
> So far we had simple throttling design and we did not wanted to connect
> with other components while making throttling decision(as it was local
> counter).
> But with new implementation we need to make few connections while server
> startup and running. For them and to store other configurations we need to
> have following configuration parameters.
>
>- At server startup(if this is newly spawned instance) we need to get
>all throttled events from central policy server. For this we have already
>implemented JAX-RS based REST web service which we can deploy on central
>policy server. This web service need to be secured with basic auth and
>should only allowed admin calls. This server to server communication do not
>need any information related to tenants or user.For this we need to have
>following parameters at gateway side. Refer connection 01 in attached
>image[1]
>
> 01. throttle event service URL.
> 02. Admin user name.
> 03. Admin password.
> 04. If we fetch batched results then we need to pass batch size
> and offset(as of now haven't implemented this).
>

I think we should avoid getting all throttled events as we discussed
earlier and only get some portions (or fixed no of entries).. Then by
allowing to send throttle event trigger to topic  for each throttling
situation again and again, new node can get an update..

>
>- When server startup gateway need to subscribe to topic registered
>and central policy server (or other component communicate with central
>policy server). For this we need to have JMS connection details. Refer
>connection 02 in attached image[1]. To make that connection we need
>following configuration.
>
> 01 JMS connection URL+ Topic name.
> 02 User Name.
> 03 Password.
> 04. Some transport specific properties.
>
>- When we display tiers and let users to use tiers we need to limit
>visibility of unlimited tier. For this now we have
>configuration(EnableUnlimitedTier) in api-manager.xml configuration file.
>We need to have configuration for this as well.
>
>
>- Also we need to have configuration to enable new throttling
>implementation.
>
>
>- Important thing here is once users enabled new throttling feature,
>user interface and throttle handler in synapse configuration will be
>changed.
>- If users need to move current throttling data to new throttling
>engine there will be data migration.
>- Once users switched to new throttling implementation they cannot
>simply revert back to old throttling implementation.
>- And at a given time we cannot support both new and old throttling
>features from UI level(but runtime can handle it).
>- If users are still willing to use old throttling handlers(runtime)
>they can keep it(handlers in configuration) as it is.
>- But user interfaces will be show new throttling data and handlers
>remain unchanged.
>
>
>-
>
>If throttle data source maintained as separate data source then that
>data source name also should be in configuration.
>
> Other than that, we need to have some jms tuning parameters (Thread pool,
timeout values and etc) as well..


>
> So all together i would like to suggest to add following configuration to
> API Manager configuration file.
> This is not mandatory configuration and if this configuration was not
> there then system will run with old throttling implementation.
>
> 
> false
> true
> jdbc/WSO2AM_DB
> 
> http://127.0.0.1:9763/throttledata/service
> 
> admin
> admin
> 
> 
> tcp://127.0.0.1:52556
> admin
> admin
> 
> PropertyValue1
> PropertyValue2
> 
> 
> 
>
>
Thanks,
Mohan


>
> [01]Image
>
>
> ​
> Please review this and let us know your feedback.
>
>
> Thanks,
> sanjeewa.
>
>
> --
>
> *Sanjeewa Malalgoda*
> WSO2 Inc.
> Mobile : +94713068779
>
> blog
> :http://sanjeewamalalgoda.blogspot.com/
> 
>
>
>


-- 
*V. Mohanadarshan*
*Senior Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Adding arbitrary mapping in siddhi / execution plan validation

[Mohanadarshan Vivekanandalingam]

Yes.. +1..

On Tue, May 17, 2016 at 2:48 PM, Sajith Perera  wrote:

> +1,Yes this can be done, as we can read the annotation keys and values.
>
> On Tue, May 17, 2016 at 2:42 PM, Sriskandarajah Suhothayan 
> wrote:
>
>> How about
>>
>> @Import('loganalyzer:1.0.0', includeArbitraryData='true')
>> define stream loganalyzer (logstream string, arbitraryDataMap object);
>>
>> @Export('loganalyzer:1.0.0', includeArbitraryData ='true')
>> define stream loganalyzer (logstream string, arbitraryDataMap object);
>>
>>
>>
>>
>>
>>
>>
>> On Tue, May 17, 2016 at 1:26 PM, Sajith Perera  wrote:
>>
>>> Hi,
>>>
>>> While adding arbitrary mapping thing and doing execution plan
>>> validation, "EventProcessorHelper"  validate whether user given siddhi
>>> stream definition attribute count with the actual stream definition
>>> attribute count.
>>>
>>> Since if the user added new attribute for the siddhi stream definition
>>> "define stream loganalyzer (logstream string, mapData Object);"
>>> exception occurred.
>>>
>>> As a solution we can adding annotation "@Arbitrary" above the stream
>>> definition and check whether user allow to have more than attributes define
>>> in the stream.
>>>
>>>
>>> /* Enter a unique ExecutionPlan */
>>> @Plan:name('Loganalyzer')
>>>
>>> /* Enter a unique description for ExecutionPlan */
>>> -- @Plan:description('ExecutionPlan')
>>>
>>> /* define streams/tables and write queries here ... */
>>>
>>> @Plan:statistics('false')
>>>
>>> @Plan:trace('true')
>>>
>>> @Import('loganalyzer:1.0.0')@Arbitrary
>>> define stream loganalyzer (logstream string, mapData Object);
>>> @Export('outputStream:1.0.0')
>>> define stream cepOutput (outputValue string);
>>>
>>> from loganalyzer
>>> select cast(map:get(mapData , 'content') , 'string') as outputValue
>>> insert into cepOutput;
>>>
>>> WDYT?
>>>
>>>
>>> Thanks,
>>> SajithD
>>>
>>> --
>>> Sajith Dimal
>>> Software Engineer
>>> Mobile : +94783101496
>>> WSO2 Inc. | http://wso2.com
>>> lean.enterprise.middleware
>>>
>>
>>
>>
>> --
>>
>> *S. Suhothayan*
>> Technical Lead & Team Lead of WSO2 Complex Event Processor
>> *WSO2 Inc. *http://wso2.com
>> * *
>> lean . enterprise . middleware
>>
>>
>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog:
>> http://suhothayan.blogspot.com/ twitter:
>> http://twitter.com/suhothayan  | linked-in:
>> http://lk.linkedin.com/in/suhothayan *
>>
>
>
>
> --
> Sajith Dimal
> Software Engineer
> Mobile : +94783101496
> WSO2 Inc. | http://wso2.com
> lean.enterprise.middleware
>



-- 
*V. Mohanadarshan*
*Senior Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [Dev] Why APIM don't impose any permission for deleting an API

[Mohanadarshan Vivekanandalingam]

[adding dev]

On Fri, May 20, 2016 at 9:48 PM, Abimaran Kugathasan 
wrote:

> Hi All,
>
> API Manager has permissions for Create, Publish and Subscribe an API. Why
> API Manager don't check any permission for deleting an API?
>
> A user can create an API, another one varified the API and publish it, in
> the same way, we need to give permission for the user to delete the API.
>
> Was there any reason for allowing the user who created the API to delete
> it?
>
> --
> Thanks
> Abimaran Kugathasan
>
> Software Engineer | WSO2 Inc
> Data & APIs Technologies Team
> Mobile : +94 773922820
>
> 
> 
>   
> 
>
>


-- 
*V. Mohanadarshan*
*Senior Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [PET] CEP Extension for Unique window - Length, Time

[Mohanadarshan Vivekanandalingam]

On Mon, May 30, 2016 at 2:08 PM, Yashothara Shanmugarajah <
yashoth...@wso2.com> wrote:

> ​Hi All,
>
> I have planned to develop Custom Window CEP extension for Unique window
> with specifying length and time parameter for this. Already siddhi
> extension has feature for getting the latest events that are unique
> according to the unique attribute. It has only one parameter *'attribute*:
> The attribute that should be checked for uniqueness'.For example we can
> specify like this.
> ​*​*
>
>
> *  unique (attribute)*
> But now my task is add two parameters time and length * unique
> (attribute, length) and ** unique (attribute,
> time). *So the output should be the unique events within the given time
> window or length window according to the unique attribute value. So there
> should be two classes extending WindowProcessor.
>
> Please let me know if you have any suggestions on this.
>

Yes, here you have to write two window extensions (UniqueLength window and
UniqueTime window).. Please refer latest Siddhi window implementations for
more information..

Thanks,
Mohan



>
> Thanks.
> Best Regards,
> Yashothara.S
>
> Software Engineer
> WSO2
>
>


-- 
*V. Mohanadarshan*
*Senior Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [PET] CEP Extension for Unique window - Length, Time

[Mohanadarshan Vivekanandalingam]

On Mon, May 30, 2016 at 10:49 PM, Imesh Gunaratne  wrote:

> Hi Yashothara,
>
> On Mon, May 30, 2016 at 2:08 PM, Yashothara Shanmugarajah <
> yashoth...@wso2.com> wrote:
>>
>>
>> But now my task is add two parameters time and length * unique
>> (attribute, length) and ** unique (attribute,
>> time). *So the output should be the unique events within the given time
>> window or length window according to the unique attribute value. So there
>> should be two classes extending WindowProcessor.
>>
>
> Can't we add these attributes to the existing Unique window processor [1]
> (as optional) without implementing a new window processor?
> [1] https://docs.wso2.com/display/CEP310/Windows#Windows-UniqueWindow
>
>
Yes, your point is valid Imesh.. But IMHO, it is better to have separate
implementations.. Technically, all these window types can be implemented as
one window. But, I prefer to avoid it due to below reasons..

- To keep the source code simple and more specific for the requirement
- We are more concerned about performance and need to avoid unnecessary
condition checks and unrelated processing.. Per event latency and TPS are
very important benchmarks in CEP.

P.S: Anyway, if final implementations are very much identical (and no much
validations required) then we can merge them in to one..

Thanks,
Mohan


> Thanks
>
> --
> *Imesh Gunaratne*
> Software Architect
> WSO2 Inc: http://wso2.com
> T: +94 11 214 5345 M: +94 77 374 2057
> W: https://medium.com/@imesh TW: @imesh
> Lean . Enterprise . Middleware
>
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*V. Mohanadarshan*
*Senior Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [PET] [CEP] File Writer for CEP

[Mohanadarshan Vivekanandalingam]

On Tue, May 31, 2016 at 10:38 AM, Malaka Silva  wrote:

> Is the scope should only cover local file system? Then IMO this should be
> the optimum way.
>

+1 . Yes, covering local file system should be enough here..

Thanks,
Mohan


>
> On Tue, May 31, 2016 at 10:25 AM, Rajjaz Mohammed  wrote:
>
>> Hi All,
>>
>> I have planed to develop $Subject using NIO[1] . NIO was created to allow
>> Java programmers to implement high-speed I/O without having to write custom
>> native code. NIO moves the most time-consuming I/O activities (namely,
>> filling and draining buffers) back into the operating system, thus allowing
>> for a great increase in speed[2].
>>
>> Hope NIO was the best choice for CEP file writer. If you have any
>> suggestions please add in this thread.
>>
>> [1]
>> https://docs.oracle.com/javase/7/docs/api/java/nio/package-summary.html
>> [2] http://www.ibm.com/developerworks/java/tutorials/j-nio/j-nio.html
>>
>> --
>> Thank you
>> Best Regards
>>
>> *Rajjaz HM*
>> Associate Software Engineer
>> Platform Extension Team
>> WSO2 Inc. 
>> lean | enterprise | middleware
>> Mobile | +94752833834|+94777226874
>> Email   | raj...@wso2.com
>> LinkedIn  | Blogger
>>  | WSO2 Profile
>> 
>>
>
>
>
> --
>
> Best Regards,
>
> Malaka Silva
> Senior Tech Lead
> M: +94 777 219 791
> Tel : 94 11 214 5345
> Fax :94 11 2145300
> Skype : malaka.sampath.silva
> LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
> Blog : http://mrmalakasilva.blogspot.com/
>
> WSO2, Inc.
> lean . enterprise . middleware
> http://www.wso2.com/
> http://www.wso2.com/about/team/malaka-silva/
> 
> https://store.wso2.com/store/
>
> Save a tree -Conserve nature & Save the world for your future. Print this
> email only if it is absolutely necessary.
>



-- 
*V. Mohanadarshan*
*Senior Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [PET][CEP] Siddhi Extension for Unit Conversion

[Mohanadarshan Vivekanandalingam]

On Tue, May 31, 2016 at 11:15 AM, Hariprasath Thanarajah <
haripras...@wso2.com> wrote:

> Hi All,
>
> I have planned to develop $Subject using JSR-000363 Units of Measurement
> API [1]. Hope this is the best choice for creating the Unit conversion [2].
> If you have any suggestions please add in this thread.
>

OK, Have you done any evaluations on this? what are the other libraries
that you evaluated ?

Thanks,
Mohan

>
> [1] - https://jcp.org/aboutJava/communityprocess/pr/jsr363/index.html
> [2] -
> http://stackoverflow.com/questions/4224122/which-jsr-275-units-implementation-should-be-used
>
> --
>
>
> *Thank you and Regards**Hariprasath Thanarajah*
> Associate Software Engineer | WSO2
> E: haripras...@wso2.com
> M: +94752806528, 0777216903
>
>


-- 
*V. Mohanadarshan*
*Senior Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [PET][CEP] Siddhi Extension for Unit Conversion

[Mohanadarshan Vivekanandalingam]

On Wed, Jun 1, 2016 at 11:24 AM, Hariprasath Thanarajah <
haripras...@wso2.com> wrote:

> Hi Mohan,
>
> I am referring JUnitConv[1], Frink[2]. But JScience library is the best
> way to do the Unit conversion[3] which is an implementation of JSR 275.
> Since the JSR-275 has been rejected and the new Units of Measurement
> standard for Java is JSR 363 and it is pretty much good amongst all.
>
>
Great.. Then let's go with JSR 363..

Thanks,
Mohan



>
> [1] - http://www.tecnick.com/public/code/cp_dpage.php?aiocp_dp=junitconv
>
> [2] - http://futureboy.us/frinkdocs/
>
> [3] - http://jscience.org/api/javax/measure/unit/package-summary.html,
> http://jscience.org/
>
> [4] -
> http://stackoverflow.com/questions/6474658/unit-of-measurement-api-in-java
>
> On Tue, May 31, 2016 at 11:57 PM, Mohanadarshan Vivekanandalingam <
> mo...@wso2.com> wrote:
>
>>
>>
>> On Tue, May 31, 2016 at 11:15 AM, Hariprasath Thanarajah <
>> haripras...@wso2.com> wrote:
>>
>>> Hi All,
>>>
>>> I have planned to develop $Subject using JSR-000363 Units of Measurement
>>> API [1]. Hope this is the best choice for creating the Unit conversion [2].
>>> If you have any suggestions please add in this thread.
>>>
>>
>> OK, Have you done any evaluations on this? what are the other libraries
>> that you evaluated ?
>>
>> Thanks,
>> Mohan
>>
>>>
>>> [1] - https://jcp.org/aboutJava/communityprocess/pr/jsr363/index.html
>>> [2] -
>>> http://stackoverflow.com/questions/4224122/which-jsr-275-units-implementation-should-be-used
>>>
>>> --
>>>
>>>
>>> *Thank you and Regards**Hariprasath Thanarajah*
>>> Associate Software Engineer | WSO2
>>> E: haripras...@wso2.com
>>> M: +94752806528, 0777216903
>>>
>>>
>>
>>
>> --
>> *V. Mohanadarshan*
>> *Senior Software Engineer,*
>> *Data Technologies Team,*
>> *WSO2, Inc. http://wso2.com <http://wso2.com> *
>> *lean.enterprise.middleware.*
>>
>> email: mo...@wso2.com
>> phone:(+94) 771117673
>>
>
>
>
> --
>
>
> *Thank you and Regards**Hariprasath Thanarajah*
> Associate Software Engineer | WSO2
> E: haripras...@wso2.com
> M: +94752806528, 0777216903
>
>


-- 
*V. Mohanadarshan*
*Senior Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com <http://wso2.com> *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Data Bridge Agent Publisher for C5 products

[Mohanadarshan Vivekanandalingam]

On Thu, Jun 2, 2016 at 9:43 PM, Isuru Perera  wrote:

> Hi Suho,
>
> In Metrics, I have written the DAS Reporter using the data publisher. Do
> you think I should change the implementation to use an HTTP client?
>

We don't need to change from Thrift to HTTP, rather we need to have an
intermediate (high-level) layer which does not depend on underneath
transport protocol and relevant transport can be configured using a config
file (Like we have stream terminology for CEP publisher) .


>
> We also need to release Metrics soon. So, if we are going to change the
> implementation, we need to decide soon. However I think it'll have an
> impact on the release schedule for Metrics.
>
> In Metrics, the DAS reporter is used to store the historical metrics data
> in DAS. The reporter will send events periodically (1 minute, by default).
> So, do you think we will encounter any issue with current Data Publisher.
>

Theoretically, I don't think there are any issues but we need tryout..


>
> Please let me know.
>
> Thanks!
>
> On Thu, Jun 2, 2016 at 3:18 PM, Sriskandarajah Suhothayan 
> wrote:
>
>> Are we going to use data bridge in C5 ?
>> C5 have Netty based transports, cant we use one of them to publish events
>> to DAS, since DAS have the capability to receiving from any transport
>> protocol via extensions this will not be a problem.
>>
>> In my opinion we should not be depending on data publisher as it have
>> several issues like events can get out of ordered, dropped and its not
>> reliable. We should have a publishing framework which is independent of the
>> transport, so users can pic Thrift, HTTP or AMQP based on their use cases.
>>
>> WDYT?
>>
>> Suho
>>
>> On Thu, Jun 2, 2016 at 2:35 PM, Kishanthan Thangarajah <
>> kishant...@wso2.com> wrote:
>>
>>> A separate point to note.
>>>
>>> Thinking along the AS 6.0 and C5 aspect, what we need is a library,
>>> where we could use that in both OSGi env and non-OSGi environments. It
>>> should not have any direct dependency on carbon API's. Currently, with AS
>>> 6.0, we are using the data publisher from C4 without any code changes and
>>> removing unwanted dependencies. If we planning to write this again, we
>>> should come up with the minimum library that could be used in a OSGi and
>>> non-OSGi env.
>>>
>>>
>>> On Thu, Jun 2, 2016 at 1:40 PM, Isuru Perera  wrote:
>>>
 Hi,

 On Thu, Jun 2, 2016 at 1:10 PM, Sinthuja Ragendran 
 wrote:

> Hi IsuruP,
>
> Please find the comments inline.
>
> On Thu, Jun 2, 2016 at 12:18 PM, Isuru Perera  wrote:
>
>> Hi,
>>
>> This is regarding $subject and the main problem we have is that there
>> is no Carbon 5 compatible feature for data bridge agent.
>>
>
> Data bridge agent publisher is not depends on carbon, and it has some
> dependencies for carbon utils, and carbon base, which we can eliminate by
> passing proper configurations in data-agent-config.xml.
>
 Yes. This is what should be done.

 I think we should avoid carbon dependencies in data publisher. We can
 have some Carbon specific component to initialize data publisher in Carbon
 (OSGi) environment.

> In that case, what do you mean by it's not compatible by Carbon 5 as
> it's anyhow doesn't depend on carbon features?
>
 As I mentioned earlier, the publisher has Carbon 4.x dependencies. So,
 we need to workaround problems like NoClassDefFoundError for
 CarbonUtils etc.

>
>
>>
>> Since Data Bridge Agent is already an OSGi bundle, we can use it
>> within C5 products. But we have to include it with some feature.
>>
>> For example, Carbon Metrics needs to publish events to DAS. So, is it
>> okay if I keep data bridge agent in Metrics feature?
>>
>
> No, I don't think that is a good option. Because the publisher is a
> generic feature, and it doesn't have any relation ship to metrics feature
> other then metrics feature is using data publisher feature. In that case,
> you need to have just importFeature defn for the datapublisher feature 
> form
> the metrics feature.
>
 Yes. The correct way is to import data publisher feature. However there
 is no such feature available now. Since Metrics needs the publisher
 dependencies, I thought we can include those dependencies until we have a
 data publisher designed to work with C5. When someone wants to install
 metrics feature, it should work without any issue. Right now, I cannot do a
 release of Carbon Metrics till I have answers to the questions raised in
 this mail thread.

>
>
>>
>> Other problem is that the current Data Bridge Agent is written for
>> Carbon 4.x based products. For example it uses CarbonUtils to find the
>> location of data-agent-config.xml. The CarbonUtils class used by the 
>> agent
>> is only available in C4.
>>
>> We can avoid this by g

Re: [Architecture] [PET] [CEP] File Writer for CEP

[Mohanadarshan Vivekanandalingam]

On Sun, Jun 5, 2016 at 10:59 PM, Rajjaz Mohammed  wrote:

> Hi All,
>
> When I try FileWriter adapter I’m getting below output in file for both
> xml[1] and JSON[2]. Since it's coming through streaming each entries saving
> as  individual entries[3].
>
> Is this expected behaviour?
>

Yes, that is expected..


>
> [1] *XML*
>
> 199008131245false100temperature23.456567.12324100.3423.4545
> 199008131245true101temperature23.456567.12324100.3423.4545
>
> [2] *JSON*
>
> {"event":{"metaData":{"timestamp":199008131245,"isPowerSaverEnabled":false,"sensorId":100,"sensorName":"temperature"},"correlationData":{"longitude":23.45656,"latitude":7.12324},"payloadData":{"humidity":100.34,"sensorValue":23.4545}}}
> {"event":{"metaData":{"timestamp":199008131245,"isPowerSaverEnabled":true,"sensorId":101,"sensorName":"temperature"},"correlationData":{"longitude":23.45656,"latitude":7.12324},"payloadData":{"humidity":100.34,"sensorValue":23.4545}}}
>
> [3]
> data
>
>
> On Tue, May 31, 2016 at 11:54 PM, Mohanadarshan Vivekanandalingam <
> mo...@wso2.com> wrote:
>
>>
>>
>> On Tue, May 31, 2016 at 10:38 AM, Malaka Silva  wrote:
>>
>>> Is the scope should only cover local file system? Then IMO this should
>>> be the optimum way.
>>>
>>
>> +1 . Yes, covering local file system should be enough here..
>>
>> Thanks,
>> Mohan
>>
>>
>>>
>>> On Tue, May 31, 2016 at 10:25 AM, Rajjaz Mohammed 
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I have planed to develop $Subject using NIO[1] . NIO was created to
>>>> allow Java programmers to implement high-speed I/O without having to write
>>>> custom native code. NIO moves the most time-consuming I/O activities
>>>> (namely, filling and draining buffers) back into the operating system, thus
>>>> allowing for a great increase in speed[2].
>>>>
>>>> Hope NIO was the best choice for CEP file writer. If you have any
>>>> suggestions please add in this thread.
>>>>
>>>> [1]
>>>> https://docs.oracle.com/javase/7/docs/api/java/nio/package-summary.html
>>>> [2] http://www.ibm.com/developerworks/java/tutorials/j-nio/j-nio.html
>>>>
>>>> --
>>>> Thank you
>>>> Best Regards
>>>>
>>>> *Rajjaz HM*
>>>> Associate Software Engineer
>>>> Platform Extension Team
>>>> WSO2 Inc. <http://wso2.com/>
>>>> lean | enterprise | middleware
>>>> Mobile | +94752833834|+94777226874
>>>> Email   | raj...@wso2.com
>>>> LinkedIn <https://lk.linkedin.com/in/hmohammedrajjaz> | Blogger
>>>> <http://wso2experience.blogspot.com/> | WSO2 Profile
>>>> <http://wso2.com/about/team/mohammer-rajjaz/>
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Best Regards,
>>>
>>> Malaka Silva
>>> Senior Tech Lead
>>> M: +94 777 219 791
>>> Tel : 94 11 214 5345
>>> Fax :94 11 2145300
>>> Skype : malaka.sampath.silva
>>> LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
>>> Blog : http://mrmalakasilva.blogspot.com/
>>>
>>> WSO2, Inc.
>>> lean . enterprise . middleware
>>> http://www.wso2.com/
>>> http://www.wso2.com/about/team/malaka-silva/
>>> <http://wso2.com/about/team/malaka-silva/>
>>> https://store.wso2.com/store/
>>>
>>> Save a tree -Conserve nature & Save the world for your future. Print
>>> this email only if it is absolutely necessary.
>>>
>>
>>
>>
>> --
>> *V. Mohanadarshan*
>> *Senior Software Engineer,*
>> *Data Technologies Team,*
>> *WSO2, Inc. http://wso2.com <http://wso2.com> *
>> *lean.enterprise.middleware.*
>>
>> email: mo...@wso2.com
>> phone:(+94) 771117673
>>
>
>
>
> --
> Thank you
> Best Regards
>
> *Rajjaz HM*
> Associate Software Engineer
> Platform Extension Team
> WSO2 Inc. <http://wso2.com/>
> lean | enterprise | middleware
> Mobile | +94752833834|+94777226874
> Email   | raj...@wso2.com
> LinkedIn <https://lk.linkedin.com/in/hmohammedrajjaz> | Blogger
> <http://rajjazhm.blogspot.com/> | WSO2 Profile
> <http://wso2.com/about/team/mohammer-rajjaz/>
>



-- 
*V. Mohanadarshan*
*Associate Tech Lead,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com <http://wso2.com> *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [Analytics] Using UTC across all temporal analytics functions

[Mohanadarshan Vivekanandalingam]

+1, yes... We can extend Siddhi Time functions to allow timezone as a
parameter.. Will do that..



On Mon, Jun 13, 2016 at 7:13 PM, Ruwan Abeykoon  wrote:

> Hi All,
> +1 on storing time in UTC.
>
> However this will cause some minor (but annoying) discrepancies to occur
> when we do windowing on "per-hour" basis, when the time zone difference
> between UTC and the server is not full hour. e.g. +5:30 UTC.
> Is there any solution for that?
>
> Cheers,
> Ruwan
>
> On Mon, Jun 13, 2016 at 6:10 PM, Inosh Goonewardena 
> wrote:
>
>> Hi,
>>
>> In analytics use cases we use a couple of extensions to extract
>> attributes (year, month, day, hour, etc.) from the event timestamp. In
>> product anaytics implementations, we mostly use time:extract() Siddhi
>> extension [1] in real time scenarios and DateTimeUDF in batch scenarios
>> [2]. However, both those extensions give output time units based on the
>> local time zone of the server on which DAS is running.
>>
>> For example, if the DAS is running in a server which has the timezone set
>> as IST, for timestamp 1461135538669 (2016/04/20 06:58:58 GMT) output time
>> units are resolved as 2016, 04, 20, 12, 28.
>>
>> In most of the scenarios, analytics is implemented in per 
>> basis, i.e., we maintain summary tables for per minute, per hour, per day,
>> per month. These summary tables has columns for year, month, date, hour,
>> etc. Since aforementioned extensions are giving time units based on local
>> timezone what we store in there are are local time units.
>>
>> IMO, we should store UTC time units instead, since it is better to
>> maintain time units uniformly without depending on the time zone of the
>> server DAS is running. We have also found that this inconsistency is
>> capable of producing issues in incremental data processing.
>>
>> Shall we extend our analytics extensions to support UTC time units
>> throughout?
>>
>> [1]
>> https://github.com/wso2/siddhi/blob/master/modules/siddhi-extensions/time/src/main/java/org/wso2/siddhi/extension/time/ExtractAttributesFunctionExtension.java
>> [2]
>> https://github.com/wso2/shared-analytics/blob/master/components/spark-udf/org.wso2.carbon.analytics.shared.spark.common.udf/src/main/java/org/wso2/carbon/analytics/shared/common/udf/DateTimeUDF.java
>>
>> --
>> Thanks & Regards,
>>
>> Inosh Goonewardena
>> Associate Technical Lead- WSO2 Inc.
>> Mobile: +94779966317
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
>
> *Ruwan Abeykoon*
> *Associate Director/Architect**,*
> *WSO2, Inc. http://wso2.com  *
> *lean.enterprise.middleware.*
>
> email: ruw...@wso2.com
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*V. Mohanadarshan*
*Associate Tech Lead,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Incremental Processing Support in DAS

[Mohanadarshan Vivekanandalingam]

Hi Inosh,

Thanks for the fix.. We have incorporated this to analytics-is..

Regards,
Mohan


On Tue, Jun 14, 2016 at 7:33 PM, Inosh Goonewardena  wrote:

> Hi All,
>
> We made a modification to incremental data processing syntax and
> implementation in order to avoid inconsistencies occur with timeWindows.
>
> For example, with the previous approach if the timeWindow is specified as
> 3600 (or 86400) incremental read time is getting set to the hour starting
> time (or day starting) time of UTC time. It is due to the the fact that
> incremental start time is calculated by performing modulo operation on a
> timestamp. In most of the scenarios, analytics is implemented in per  unit> basis, i.e., we maintain summary tables for per minute, per hour, per
> day, per month. In these summary tables, we do windowing based on the local
> timezone of the server which DAS runs [1]. Since incremental time window
> start at UTC time (hour or day) inconsistencies happens in aggregate values
> of time windows.
>
> With the new introduced changes, incremental window start time is set
> based on local time zone to avoid aforementioned inconsistencies. Syntax is
> as follows.
>
> CREATE TEMPORARY TABLE T1 USING CarbonAnalytics options (tableName "test",
> incrementalProcessing "T1, *WindowUnit*, WindowOffset")*;*
>
>
> WindowUnit concept is similar to the previous concept of TimeWindow, but
> in here corresponding window time unit needs to be provided instead of time
> in millis. The units supported are SECOND, MINUTE, HOUR, DAY, MONTH, YEAR.
>
> For example, lets say server runs in IST time and last processed event
> time is 146538669 (2016/04/20 05:48:58 PM IST). If the WindowUnit is
> set as DAY and WindowOffset is set to 0 in the incremental table, next
> script execution will read data starting from 146109060 (2016/04/20
> 12:00:00 AM IST).
>
> [1] [Analytics] Using UTC across all temporal analytics functions
>
>
>
> On Fri, Mar 25, 2016 at 10:59 AM, Gihan Anuruddha  wrote:
>
>> Actually currently in ESB-analytics also we are using a similar
>> mechanism. In there we keep 5 tables for each time unites(second, minute,
>> hour, day, month) and we have windowOffset in correspondent to it time
>> unite like in minute-60, hour- 3600 etc. We are only storing min, max, sum
>> and count values only. So at a given time if we need the average, we divide
>> the sum with count and get that.
>>
>> On Fri, Mar 25, 2016 at 8:47 AM, Inosh Goonewardena 
>> wrote:
>>
>>>
>>>
>>> On Thu, Mar 24, 2016 at 9:56 PM, Sachith Withana 
>>> wrote:
>>>
 Hi Inosh,

 We wouldn't have to do that IMO.

 We can persist the total aggregate value upto currentTimeWindow -
 WindowOffset, along with the previous time window aggregation meta data as
 well ( count, sum in the average aggregation case).

>>>
>>> Yep. That will do.
>>>

 The previous total wouldn't be calculated again, it's the last two time
 windows ( including the current one) that we need to recalculate and add it
 to the previous total.

 It works almost the same way as the current incremental processing
 table, but keeping more meta_data on the aggregation related values.

 @Anjana WDYT?

 Thanks,
 Sachith


 On Fri, Mar 25, 2016 at 7:07 AM, Inosh Goonewardena 
 wrote:

> Hi Sachith,
>
>
> *WindowOffset*:
>>
>> Events might arrive late that would belong to a previous processed
>> time window. To account to that, we have added an optional parameter that
>> would allow to
>> process immediately previous time windows as well ( acts like a
>> buffer).
>> ex: If this is set to 1, apart from the to-be-processed data, data
>> related to the previously processed time window will also be taken for
>> processing.
>>
>
>
> This means, if window offset is set, already processed data will be
> processed again. How does the aggregate functions works in this case?
> Actually, what is the plan on supporting aggregate functions? Let's take
> average function as an example. Are we going to persist sum and count
> values per time windows and re-calculate whole average based on values of
> all time windows? Is so, I would guess we can update previously processed
> time windows sum and count values.
>
>
> On Thu, Mar 24, 2016 at 3:50 AM, Sachith Withana 
> wrote:
>
>> Hi Srinath,
>>
>> Please find the comments inline.
>>
>> On Thu, Mar 24, 2016 at 11:39 AM, Srinath Perera 
>> wrote:
>>
>>> Hi Sachith, Anjana,
>>>
>>> +1 for the backend model.
>>>
>>> Are we handling the case when last run was done, 25 minutes of data
>>> is processed. Basically, next run has to re-run last hour and update the
>>> value.
>>>
>>
>> Yes. It will recalculate for that hour and will update the value.
>>
>>
>>>
>>> When does one hour c

Re: [Architecture] Caching Support for Analytics Event Tables

[Mohanadarshan Vivekanandalingam]

On Mon, Jun 20, 2016 at 8:17 PM, Sriskandarajah Suhothayan 
wrote:

> Is this in line with the RDBMS implementation? Else it will be confusing
> to the users.
> Shall we have a chat and merge the caching code?
>

Yes, let's have a chat..

>
> @Mohan can you work with Anjana
>

sure...


>
> Regards
> Suho
>
> On Mon, Jun 20, 2016 at 12:49 PM, Anjana Fernando  wrote:
>
>> Hi,
>>
>> With a chat we had with Srinath, we've decided to set the default cache
>> timeout to 10 seconds, so from this moment, it is set to 10 seconds by
>> default in the code.
>>
>> Cheers,
>> Anjana.
>>
>> On Wed, Jun 15, 2016 at 1:57 PM, Nirmal Fernando  wrote:
>>
>>> Great! Thanks Anjana!
>>>
>>> On Wed, Jun 15, 2016 at 11:26 AM, Anjana Fernando 
>>> wrote:
>>>
 Hi,

 We've added the $subject. Basically, a local cache is now maintained in
 each event table, where it will store the most recently used data items in
 the cache, up to a certain given cache size, for a maximum given lifetime.
 The format is as follows:-

  @from(eventtable = 'analytics.table' , table.name = 'name', *caching*
 = 'true', *cache.timeout.seconds* = '10', *cache.size.bytes* =
 '10')

 The cache.timeout.seconds and cache.size.bytes values are optional,
 with default values of 60 (1 minute) and 1024 * 1024 * 10 (10 MB)
 respectively.

 Also, there are some debug logs available in the component, if you want
 to check for explicit cache hit/miss situations and record lookup timing,
 basically enable debug logs for the class
 "org.wso2.carbon.analytics.eventtable.AnalyticsEventTable".

 So basically, if you use analytics event tables in performance
 sensitive areas in your CEP execution plans, do consider using caching if
 it is possible to do so.

 The unit tests are updated with caching, and the updated docs can be
 found here [1].

 [1]
 https://docs.wso2.com/display/DAS310/Understanding+Event+Streams+and+Event+Tables#UnderstandingEventStreamsandEventTables-AnalyticseventtableAnalyticseventtable

 Cheers,
 Anjana.
 --
 *Anjana Fernando*
 Senior Technical Lead
 WSO2 Inc. | http://wso2.com
 lean . enterprise . middleware

>>>
>>>
>>>
>>> --
>>>
>>> Thanks & regards,
>>> Nirmal
>>>
>>> Team Lead - WSO2 Machine Learner
>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc.
>>> Mobile: +94715779733
>>> Blog: http://nirmalfdo.blogspot.com/
>>>
>>>
>>>
>>
>>
>> --
>> *Anjana Fernando*
>> Senior Technical Lead
>> WSO2 Inc. | http://wso2.com
>> lean . enterprise . middleware
>>
>
>
>
> --
>
> *S. Suhothayan*
> Technical Lead & Team Lead of WSO2 Complex Event Processor
> *WSO2 Inc. *http://wso2.com
> * *
> lean . enterprise . middleware
>
>
> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog:
> http://suhothayan.blogspot.com/ twitter:
> http://twitter.com/suhothayan  | linked-in:
> http://lk.linkedin.com/in/suhothayan *
>



-- 
*V. Mohanadarshan*
*Associate Tech Lead,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] IS-Analytics performance

[Mohanadarshan Vivekanandalingam]

On Mon, Jun 27, 2016 at 4:30 PM, Seshika Fernando  wrote:

> Hey saith,
>
> This is great. So when you removed the duplicate windows, there were no
> OOM issues?
>
Yes, we have introduced per-second windows and eventually per-minute event
count get reduced. Then, there will not be an OOM..
Latest number is also seems good.. SajithR, can share those numbers..

Thanks,
Mohan


> Seshi
> On 24 Jun 2016 14:28, "Sajith Ravindra"  wrote:
>
>> Hi Malith,
>>
>> Thanks for the explanation.
>>>
>>> I would expect some variation in the throughput. The aim should be
>>> minimize the variation in the throughput (while maintaining the throughput
>>> at its highest level).
>>>
>>  Agreed. Actually, our expectation is to minimize the fluctuation and to
>> increase the throughput.
>>
>>>
>>> It is possible to measure the latency as well?
>>>
>> In IS-Analytics we don't generate any output events or alerts currently.
>> Therefore, we don't calculate the latency. What we do is summarize events
>> using siddhi queries and persist them in a DB and further summarize
>> persisted data using Spark and then displayed through dashbaords.
>>
>>
>> Thanks
>> *,Sajith Ravindra*
>> Senior Software Engineer
>> WSO2 Inc.; http://wso2.com
>> lean.enterprise.middleware
>>
>> mobile: +94 77 2273550
>> blog: http://sajithr.blogspot.com/
>> 
>>
>> On Fri, Jun 24, 2016 at 1:57 PM, Malith Jayasinghe 
>> wrote:
>>
>>> Hi Sajith,
>>>
>>> Thanks for the explanation.
>>>
>>> I would expect some variation in the throughput. The aim should be
>>> minimize the variation in the throughput (while maintaining the throughput
>>> at its highest level).
>>>
>>> It is possible to measure the latency as well?
>>>
>>> regards
>>>
>>> Malith
>>>
>>>
>>>
>>> On Thu, Jun 23, 2016 at 11:13 PM, Sajith Ravindra 
>>> wrote:
>>>
 Hi Malith,

 The deployment is just a standalone DAS server, and we are planning to
 do a test for HA deployment in recent future.

 The workload is generated by a .csv data file which has 100K sample
 events, 10M events are generated by iterating through the same data set 100
 times. But we keep increasing the timestamp. A simple thrift client is used
 to publish data.



 Thanks
 *,Sajith Ravindra*
 Senior Software Engineer
 WSO2 Inc.; http://wso2.com
 lean.enterprise.middleware

 mobile: +94 77 2273550
 blog: http://sajithr.blogspot.com/
 

 On Fri, Jun 24, 2016 at 10:34 AM, Malith Jayasinghe 
 wrote:

> Hi Sajith,
>
> Could you please provide some details about how you are actually doing
> these performance tests. For example, what is deployment model?  How are
> you generating these workloads/events?
>
> Thanks
>
> Malith
>
> On Thu, Jun 23, 2016 at 11:28 AM, Sajith Ravindra 
> wrote:
>
>> Hi all,
>>
>> This is to give an update on the performance study we conducted on
>> is-analytics server on last. The idea of this test round was to evaluate
>> the performance of Siddhi queries used for is-analytics, therefore we
>> disabled event stream persistence and spark for this test.
>>
>> This test was conducted on a standalone DAS server with Xms2g and
>> Xmx4g.
>>
>> On the initial round when input TPS reaches ~20K, the server went OOM
>> after consuming around 1M events. The reason for this was the events
>> accumulated inside 7 1min time batch windows used inside.  To overcome 
>> this
>> we implemented an extension to siddhi which allows us to avoid 
>> duplicating
>> the window.
>>
>> After removing duplicate windows the server was able to consume
>> events at a rate of ~22K, but there were fluctuations (see the graph
>> bellow) of the throughput. With analysis, we found that intense GC causes
>> this. We suspect that this intense GC  is caused when expiring a large
>> number of events accumulated inside 1-minute window. To  overcome this we
>> are planning to batch events in 1-second windows and then accumulate
>> 1second batches in 1 min window in order to stop accumulating a large
>> number of events.
>>
>>
>> As the next steps, we are planning to test the performance with event
>> stream persistence and then move on to check the performance in DAS 
>> minimum
>> HA mode.
>>
>> We will keep updating this thread with our findings.
>>
>> Please share your thought, suggestions on this.
>>
>> Thanks
>> *,Sajith Ravindra*
>> Senior Software Engineer
>> WSO2 Inc.; http://wso2.com
>> lean.enterprise.middleware
>>
>> mobile: +94 77 2273550
>> blog: http://sajithr.blogspot.com/
>> 
>>
>> ___
>> Arc

Re: [Architecture] IS-Analytics performance

[Mohanadarshan Vivekanandalingam]

On Mon, Jun 27, 2016 at 8:03 PM, Seshika Fernando  wrote:

> Iranga,
> The optimization applies to siddhi queries that were written for IS
> analytics. Other product analytics would have different queries so this is
> not applicable there.
> This is not a siddhi level change, rather an optimization of the
> previously written queries.
>
Yes, above per-second and per-minute design is applied to some other
analytics product as well but it is not generic and too specific for the
usecase and query as mentioned by Seshi..

Thanks,
Mohan


> Seshi
> On 27 Jun 2016 16:48, "Iranga Muthuthanthri"  wrote:
>
>>
>>
>> On Mon, Jun 27, 2016 at 4:35 PM, Mohanadarshan Vivekanandalingam <
>> mo...@wso2.com> wrote:
>>
>>>
>>>
>>> On Mon, Jun 27, 2016 at 4:30 PM, Seshika Fernando 
>>> wrote:
>>>
>>>> Hey saith,
>>>>
>>>> This is great. So when you removed the duplicate windows, there were no
>>>> OOM issues?
>>>>
>>> Yes, we have introduced per-second windows and eventually per-minute
>>> event count get reduced. Then, there will not be an OOM..
>>>
>>
>> is this specific for IS-Analytics or common to all other 'product
>> analytics'.
>>
>>
>>> Latest number is also seems good.. SajithR, can share those numbers..
>>>
>>> Thanks,
>>> Mohan
>>>
>>>
>>>> Seshi
>>>> On 24 Jun 2016 14:28, "Sajith Ravindra"  wrote:
>>>>
>>>>> Hi Malith,
>>>>>
>>>>> Thanks for the explanation.
>>>>>>
>>>>>> I would expect some variation in the throughput. The aim should be
>>>>>> minimize the variation in the throughput (while maintaining the 
>>>>>> throughput
>>>>>> at its highest level).
>>>>>>
>>>>>  Agreed. Actually, our expectation is to minimize the fluctuation and
>>>>> to increase the throughput.
>>>>>
>>>>>>
>>>>>> It is possible to measure the latency as well?
>>>>>>
>>>>> In IS-Analytics we don't generate any output events or alerts
>>>>> currently. Therefore, we don't calculate the latency. What we do is
>>>>> summarize events using siddhi queries and persist them in a DB and further
>>>>> summarize persisted data using Spark and then displayed through
>>>>> dashbaords.
>>>>>
>>>>>
>>>>> Thanks
>>>>> *,Sajith Ravindra*
>>>>> Senior Software Engineer
>>>>> WSO2 Inc.; http://wso2.com
>>>>> lean.enterprise.middleware
>>>>>
>>>>> mobile: +94 77 2273550
>>>>> blog: http://sajithr.blogspot.com/
>>>>> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab>
>>>>>
>>>>> On Fri, Jun 24, 2016 at 1:57 PM, Malith Jayasinghe 
>>>>> wrote:
>>>>>
>>>>>> Hi Sajith,
>>>>>>
>>>>>> Thanks for the explanation.
>>>>>>
>>>>>> I would expect some variation in the throughput. The aim should be
>>>>>> minimize the variation in the throughput (while maintaining the 
>>>>>> throughput
>>>>>> at its highest level).
>>>>>>
>>>>>> It is possible to measure the latency as well?
>>>>>>
>>>>>> regards
>>>>>>
>>>>>> Malith
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 23, 2016 at 11:13 PM, Sajith Ravindra 
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Malith,
>>>>>>>
>>>>>>> The deployment is just a standalone DAS server, and we are planning
>>>>>>> to do a test for HA deployment in recent future.
>>>>>>>
>>>>>>> The workload is generated by a .csv data file which has 100K sample
>>>>>>> events, 10M events are generated by iterating through the same data set 
>>>>>>> 100
>>>>>>> times. But we keep increasing the timestamp. A simple thrift client is 
>>>>>>> used
>>>>>>> to publish data.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>&

Re: [Architecture] Monitor Logged In Users/Sessions

[Mohanadarshan Vivekanandalingam]

Hi Srinath,

Above mentioned scenarios are not handled in the initial version of session
related analytics (and not in our future list as well).. The scenarios that
we are covering are mentioned in [1].. We are looking at overall session
analytics and not focusing on user specific sessions or session specific
user.

But, as per Gayan's description it seems they are looking to do some
operations rather than just monitoring those sessions..  But anyway, we can
incorporate these session analytics for next version it it make sense..

[1] Analytics IS - Session Analytics View

Thanks,
Mohan


On Tue, Jun 28, 2016 at 8:26 AM, Srinath Perera  wrote:

> Is this done as part of IS analytics?
>
> --Srinath
>
> On Mon, Jun 27, 2016 at 3:53 PM, Gayan Gunawardana  wrote:
>
>> Hi All,
>>
>> This feature will provide capability to admin users to monitor other
>> logged in users sessions and kill those sessions if necessary. Currently
>> logged in users sessions persist inside IDN_AUTH_SESSION_STORE table and
>> there is no mapping to authenticated user. We are planning to introduce new
>> table to maintain mapping between user and session id.
>>
>> CREATE TABLE IDN_USER_SESSION_DATA (
>>
>>   SESSION_ID VARCHAR (100) DEFAULT NULL,
>>
>>   USER_NAME VARCHAR(100) DEFAULT NULL,
>>
>>   USER_DOMAIN_NAME VARCHAR(100) DEFAULT NULL,
>>
>>   TENANT_NAME VARCHAR(100) DEFAULT NULL,
>>
>>   FOREIGN KEY (SESSION_ID) REFERENCES
>> IDN_AUTH_SESSION_STORE(SESSION_ID) ON DELETE CASCADE,
>>
>>   PRIMARY KEY (SESSION_ID, USER_NAME)
>>
>> );
>>
>> According to latest implementation of session data persistence, we can
>> consider particular SESSION ID is active if last record (sorted by time
>> descending order) for given SESSION ID is "STORE" operation. If there are
>> two store operations to IDN_AUTH_SESSION_STORE table there is a problem of
>> duplicating data in IDN_USER_SESSION_DATA. We need to find a way to handle
>> this situation.
>>
>> 1. Listing active session list for given user
>>
>> In back-end distinguish sessions will be identified by using SESSION_ID
>> but in front-end we cannot display SESSION_ID. In front-end unique sessions
>> will be displayed according to User-Agent.
>>
>> 2. Listing users who have active sessions
>>
>> Listing users who have at least one active session will be challenging.
>> Since IDN_AUTH_SESSION_STORE table is HUGE in a production system, and
>> doing a JOIN operation with it would be costly.
>>
>> The username in the USER_SESSION_DATA is picked from the authenticated
>> user attribute available in the session context. This attribute is set
>> after all processing done in the SequenceHandler hence the authenticated
>> user here actually subject identifier, rather than a real username.
>>
>> Hence the username in the USER_SESSION_DATA table can be one of following,
>> i. A Local User : who use the actual username as the subject identifier
>> ii. A Local User : who use a claim as the subject identifier
>> iii. A Federated User : who use federated subject or a claim
>>
>> Only in first scenario, it can find proper user store domain from the
>> username. In the third scenario we can store federated IDP name for
>> USER_DOMAIN_NAME.
>>
>> Handling\Storing usernames is a common thing we need to decide (in OAuth
>> IDN_OAUTH2_ACCESS_TOKEN we have the same problem), so we should figure out
>> a general schema for IDN_USER_SESSION_DATA that can be used for all types
>> of users.
>>
>> Thanks,
>> Gayan
>> --
>> Gayan Gunawardana
>> Software Engineer; WSO2 Inc.; http://wso2.com/
>> Email: ga...@wso2.com
>> Mobile: +94 (71) 8020933
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> 
> Srinath Perera, Ph.D.
>http://people.apache.org/~hemapani/
>http://srinathsview.blogspot.com/
>



-- 
*V. Mohanadarshan*
*Associate Tech Lead,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Monitor Logged In Users/Sessions

[Mohanadarshan Vivekanandalingam]

On Tue, Jun 28, 2016 at 10:23 AM, Mohanadarshan Vivekanandalingam <
mo...@wso2.com> wrote:

> Hi Srinath,
>
> Above mentioned scenarios are not handled in the initial version of
> session related analytics (and not in our future list as well).. The
> scenarios that we are covering are mentioned in [1].. We are looking at
> overall session analytics and not focusing on user specific sessions or
> session specific user.
>
> But, as per Gayan's description it seems they are looking to do some
> operations rather than just monitoring those sessions..  But anyway, we can
> incorporate these session analytics for next version it it make sense..
>
> [1] Analytics IS - Session Analytics View
>
> Thanks,
> Mohan
>
>
> On Tue, Jun 28, 2016 at 8:26 AM, Srinath Perera  wrote:
>
>> Is this done as part of IS analytics?
>>
>> --Srinath
>>
>> On Mon, Jun 27, 2016 at 3:53 PM, Gayan Gunawardana 
>> wrote:
>>
>>> Hi All,
>>>
>>> This feature will provide capability to admin users to monitor other
>>> logged in users sessions and kill those sessions if necessary. Currently
>>> logged in users sessions persist inside IDN_AUTH_SESSION_STORE table and
>>> there is no mapping to authenticated user. We are planning to introduce new
>>> table to maintain mapping between user and session id.
>>>
>>> CREATE TABLE IDN_USER_SESSION_DATA (
>>>
>>>   SESSION_ID VARCHAR (100) DEFAULT NULL,
>>>
>>>   USER_NAME VARCHAR(100) DEFAULT NULL,
>>>
>>>   USER_DOMAIN_NAME VARCHAR(100) DEFAULT NULL,
>>>
>>>   TENANT_NAME VARCHAR(100) DEFAULT NULL,
>>>
>>>   FOREIGN KEY (SESSION_ID) REFERENCES
>>> IDN_AUTH_SESSION_STORE(SESSION_ID) ON DELETE CASCADE,
>>>
>>>   PRIMARY KEY (SESSION_ID, USER_NAME)
>>>
>>> );
>>>
>>> According to latest implementation of session data persistence, we can
>>> consider particular SESSION ID is active if last record (sorted by time
>>> descending order) for given SESSION ID is "STORE" operation. If there are
>>> two store operations to IDN_AUTH_SESSION_STORE table there is a problem of
>>> duplicating data in IDN_USER_SESSION_DATA. We need to find a way to handle
>>> this situation.
>>>
>>> 1. Listing active session list for given user
>>>
>>> In back-end distinguish sessions will be identified by using SESSION_ID
>>> but in front-end we cannot display SESSION_ID. In front-end unique sessions
>>> will be displayed according to User-Agent.
>>>
>>> 2. Listing users who have active sessions
>>>
>>> Listing users who have at least one active session will be challenging.
>>> Since IDN_AUTH_SESSION_STORE table is HUGE in a production system, and
>>> doing a JOIN operation with it would be costly.
>>>
>>> The username in the USER_SESSION_DATA is picked from the authenticated
>>> user attribute available in the session context. This attribute is set
>>> after all processing done in the SequenceHandler hence the authenticated
>>> user here actually subject identifier, rather than a real username.
>>>
>>> Hence the username in the USER_SESSION_DATA table can be one of
>>> following,
>>> i. A Local User : who use the actual username as the subject identifier
>>> ii. A Local User : who use a claim as the subject identifier
>>> iii. A Federated User : who use federated subject or a claim
>>>
>>> Only in first scenario, it can find proper user store domain from the
>>> username. In the third scenario we can store federated IDP name for
>>> USER_DOMAIN_NAME.
>>>
>>> Handling\Storing usernames is a common thing we need to decide (in OAuth
>>> IDN_OAUTH2_ACCESS_TOKEN we have the same problem), so we should figure out
>>> a general schema for IDN_USER_SESSION_DATA that can be used for all types
>>> of users.
>>>
>>> Thanks,
>>> Gayan
>>> --
>>> Gayan Gunawardana
>>> Software Engineer; WSO2 Inc.; http://wso2.com/
>>> Email: ga...@wso2.com
>>> Mobile: +94 (71) 8020933
>>>
>>> ___
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> --
>> 
>> Srinath Perera, Ph.D.
>>http://people.apache.org/~hemapani/
>>http://srinathsview.blogspot.com/
>>
>
>
>
> --
> *V. Mohanadarshan*
> *Associate Tech Lead,*
> *Data Technologies Team,*
> *WSO2, Inc. http://wso2.com <http://wso2.com> *
> *lean.enterprise.middleware.*
>
> email: mo...@wso2.com
> phone:(+94) 771117673
>



-- 
*V. Mohanadarshan*
*Associate Tech Lead,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com <http://wso2.com> *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Monitor Logged In Users/Sessions

[Mohanadarshan Vivekanandalingam]

On Tue, Jun 28, 2016 at 11:24 AM, Srinath Perera  wrote:

> Please make sure we do not do the work twice. If what this do is something
> IS analytics can use later, then OK. But if one is redundant at the end,
> that is a waste.
>

I don't think we can straight away use above implementation.. It seems,
above implementation is totally dealt with Identity tables and doing
manipulation with those data..

@Gayan, can we have a chat on this and see whether we can reuse any of
these work for Identity Session related analytics ?

Thanks,
Mohan



>
> --Srinath
>
> --Srinath
>
> On Tue, Jun 28, 2016 at 10:40 AM, Mohanadarshan Vivekanandalingam <
> mo...@wso2.com> wrote:
>
>>
>>
>> On Tue, Jun 28, 2016 at 10:23 AM, Mohanadarshan Vivekanandalingam <
>> mo...@wso2.com> wrote:
>>
>>> Hi Srinath,
>>>
>>> Above mentioned scenarios are not handled in the initial version of
>>> session related analytics (and not in our future list as well).. The
>>> scenarios that we are covering are mentioned in [1].. We are looking at
>>> overall session analytics and not focusing on user specific sessions or
>>> session specific user.
>>>
>>> But, as per Gayan's description it seems they are looking to do some
>>> operations rather than just monitoring those sessions..  But anyway, we can
>>> incorporate these session analytics for next version it it make sense..
>>>
>>> [1] Analytics IS - Session Analytics View
>>>
>>> Thanks,
>>> Mohan
>>>
>>>
>>> On Tue, Jun 28, 2016 at 8:26 AM, Srinath Perera 
>>> wrote:
>>>
>>>> Is this done as part of IS analytics?
>>>>
>>>> --Srinath
>>>>
>>>> On Mon, Jun 27, 2016 at 3:53 PM, Gayan Gunawardana 
>>>> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> This feature will provide capability to admin users to monitor other
>>>>> logged in users sessions and kill those sessions if necessary. Currently
>>>>> logged in users sessions persist inside IDN_AUTH_SESSION_STORE table and
>>>>> there is no mapping to authenticated user. We are planning to introduce 
>>>>> new
>>>>> table to maintain mapping between user and session id.
>>>>>
>>>>> CREATE TABLE IDN_USER_SESSION_DATA (
>>>>>
>>>>>   SESSION_ID VARCHAR (100) DEFAULT NULL,
>>>>>
>>>>>   USER_NAME VARCHAR(100) DEFAULT NULL,
>>>>>
>>>>>   USER_DOMAIN_NAME VARCHAR(100) DEFAULT NULL,
>>>>>
>>>>>   TENANT_NAME VARCHAR(100) DEFAULT NULL,
>>>>>
>>>>>   FOREIGN KEY (SESSION_ID) REFERENCES
>>>>> IDN_AUTH_SESSION_STORE(SESSION_ID) ON DELETE CASCADE,
>>>>>
>>>>>   PRIMARY KEY (SESSION_ID, USER_NAME)
>>>>>
>>>>> );
>>>>>
>>>>> According to latest implementation of session data persistence, we can
>>>>> consider particular SESSION ID is active if last record (sorted by time
>>>>> descending order) for given SESSION ID is "STORE" operation. If there are
>>>>> two store operations to IDN_AUTH_SESSION_STORE table there is a problem of
>>>>> duplicating data in IDN_USER_SESSION_DATA. We need to find a way to handle
>>>>> this situation.
>>>>>
>>>>> 1. Listing active session list for given user
>>>>>
>>>>> In back-end distinguish sessions will be identified by using
>>>>> SESSION_ID but in front-end we cannot display SESSION_ID. In front-end
>>>>> unique sessions will be displayed according to User-Agent.
>>>>>
>>>>> 2. Listing users who have active sessions
>>>>>
>>>>> Listing users who have at least one active session will be
>>>>> challenging. Since IDN_AUTH_SESSION_STORE table is HUGE in a production
>>>>> system, and doing a JOIN operation with it would be costly.
>>>>>
>>>>> The username in the USER_SESSION_DATA is picked from the authenticated
>>>>> user attribute available in the session context. This attribute is set
>>>>> after all processing done in the SequenceHandler hence the authenticated
>>>>> user here actually subject identifier, rather than a real username.
>>>>>
>>>>> Hence the username in the USER_SESSION_DATA table c

Re: [Architecture] IS-Analytics performance

[Mohanadarshan Vivekanandalingam]

On Wed, Jun 29, 2016 at 9:23 PM, Dulitha Wijewantha 
wrote:

> Hi Mohan/Sajith,
> Can you please explain what this particular scenario was in IS? Is this a
> counting scenario on a window? This would provide input for others who are
> writing real time analytics on CEP.
>

Yes, Here we have mainly focused on some aggregations like Sum with
external time batch windows.. Queries can be found in [1]

[1]
https://github.com/wso2/analytics-is/blob/master/features/org.wso2.carbon.analytics.is.feature/src/main/capp/AuthenticationAnalyticsExecutionPlan/IsAnalytics-ExecutionPlan-AuthenticationData.siddhiql

Thanks,
Mohan


>
> Cheers~
>
> On Mon, Jun 27, 2016 at 5:50 PM, Mohanadarshan Vivekanandalingam <
> mo...@wso2.com> wrote:
>
>>
>>
>> On Mon, Jun 27, 2016 at 8:03 PM, Seshika Fernando 
>> wrote:
>>
>>> Iranga,
>>> The optimization applies to siddhi queries that were written for IS
>>> analytics. Other product analytics would have different queries so this is
>>> not applicable there.
>>> This is not a siddhi level change, rather an optimization of the
>>> previously written queries.
>>>
>> Yes, above per-second and per-minute design is applied to some other
>> analytics product as well but it is not generic and too specific for the
>> usecase and query as mentioned by Seshi..
>>
>> Thanks,
>> Mohan
>>
>>
>>> Seshi
>>> On 27 Jun 2016 16:48, "Iranga Muthuthanthri"  wrote:
>>>
>>>>
>>>>
>>>> On Mon, Jun 27, 2016 at 4:35 PM, Mohanadarshan Vivekanandalingam <
>>>> mo...@wso2.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Mon, Jun 27, 2016 at 4:30 PM, Seshika Fernando 
>>>>> wrote:
>>>>>
>>>>>> Hey saith,
>>>>>>
>>>>>> This is great. So when you removed the duplicate windows, there were
>>>>>> no OOM issues?
>>>>>>
>>>>> Yes, we have introduced per-second windows and eventually per-minute
>>>>> event count get reduced. Then, there will not be an OOM..
>>>>>
>>>>
>>>> is this specific for IS-Analytics or common to all other 'product
>>>> analytics'.
>>>>
>>>>
>>>>> Latest number is also seems good.. SajithR, can share those numbers..
>>>>>
>>>>> Thanks,
>>>>> Mohan
>>>>>
>>>>>
>>>>>> Seshi
>>>>>> On 24 Jun 2016 14:28, "Sajith Ravindra"  wrote:
>>>>>>
>>>>>>> Hi Malith,
>>>>>>>
>>>>>>> Thanks for the explanation.
>>>>>>>>
>>>>>>>> I would expect some variation in the throughput. The aim should be
>>>>>>>> minimize the variation in the throughput (while maintaining the 
>>>>>>>> throughput
>>>>>>>> at its highest level).
>>>>>>>>
>>>>>>>  Agreed. Actually, our expectation is to minimize the fluctuation
>>>>>>> and to increase the throughput.
>>>>>>>
>>>>>>>>
>>>>>>>> It is possible to measure the latency as well?
>>>>>>>>
>>>>>>> In IS-Analytics we don't generate any output events or alerts
>>>>>>> currently. Therefore, we don't calculate the latency. What we do is
>>>>>>> summarize events using siddhi queries and persist them in a DB and 
>>>>>>> further
>>>>>>> summarize persisted data using Spark and then displayed through
>>>>>>> dashbaords.
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>> *,Sajith Ravindra*
>>>>>>> Senior Software Engineer
>>>>>>> WSO2 Inc.; http://wso2.com
>>>>>>> lean.enterprise.middleware
>>>>>>>
>>>>>>> mobile: +94 77 2273550
>>>>>>> blog: http://sajithr.blogspot.com/
>>>>>>> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab>
>>>>>>>
>>>>>>> On Fri, Jun 24, 2016 at 1:57 PM, Malith Jayasinghe >>>>>> > wrote:
>>>>>>>
>>>>>>>> Hi Sajith,
>>>>>>>>
>>>>>>>&

Re: [Architecture] Caching Real time analytics data

[Mohanadarshan Vivekanandalingam]

Hi Guys,

Sorry.. I don't think that we need to fix this in UI publisher level.. In
CEP perspective, what we already have is the correct approach.. IMO, we are
trying to solve this issue in a wrong place (which is UI publisher)..

Had a offline chat with Tharik on this, he is suggested that we can
implement a hybrid provider where we can get initial information from DB
then after sometime in realtime.. This seems like the correct approach for
me..

Thanks,
Mohan


On Thu, Jul 7, 2016 at 8:39 AM, Tharik Kanaka  wrote:

> Hi Suho,
>
> +1 for implementing in UI publisher level. We can send the cached events
> as a bulk once a web socket client get subscribed to the UI publisher. Then
> push real time events as before.
>
> This caching should be an optional feature where user could enable when
> creating UI publisher. Also we could let user to configure number of cached
> records to define limits if required.
>
> Regards,
>
> On Thu, Jul 7, 2016 at 1:39 AM, Sriskandarajah Suhothayan 
> wrote:
>
>> This is a valid case, I believe this can be added to the UI publisher.
>> @Thilini can you implement caching at UI publisher such that it when a
>> gadget connects during the connection time the gadget will get all the
>> cached events, and then update itself like how it does now ?
>>
>> For the proper fix is, events should be pushed to a persistent store via
>> DAS and the gadgets should be written to fetch the data from the store
>> during startup and then update itself via real-time events.
>>
>> We have plans to implement this in the next release on top of DAS.
>>
>> Regards
>> Suho
>>
>>
>>
>> On Wed, Jul 6, 2016 at 11:19 PM, Sachith Withana 
>> wrote:
>>
>>> Agreed Sinthuja.
>>>
>>> But what about for a smaller window size (5/10/15 mins)?
>>>
>>> The reason why I bought up this issue is, in my case, I use several real
>>> time gadgets.
>>> And at the startup, they are all empty for that user until the data gets
>>> pushed in.
>>>
>>> As an end user, I would like to see the last status of the real time
>>> analytics when I log in.
>>>
>>> Thanks,
>>> Sachith
>>>
>>> On Wed, Jul 6, 2016 at 12:09 PM, Sinthuja Ragendran 
>>> wrote:
>>>
 Hi Sachith,

 If the use-case is to display the 1 hour analytics data from CEP, then
 IMO the he/she need to simply store the CEP results into a persistence
 store (DAS or RDBMS via RDBMS event publisher), and then let the gadget
 read from the persistence store. I don't think caching is a good option in
 such cases because anyhow if the server crashes due to some reason the data
 is not going to be shown.

 Thanks,
 Sinthuja.

 On Wed, Jul 6, 2016 at 10:12 PM, Sachith Withana 
 wrote:

> Hi all,
>
> In the dashboard, the real time data is only shown if the user is
> logged into the dashboard at the time of the data is being pushed.
>
> If the data is being pushed every hour, a new user who logs in would
> potentially have to wait up to one hour to see the real time data, and if
> the user refreshes, then has to wait another hour to see the data, and
> would loose the current data completely.
>
> I understand from the CEP perspective it's similar to fire and forget,
> but can't we add some level of caching to prevent this?
>
> Regards,
> Sachith
> --
> Sachith Withana
> Software Engineer; WSO2 Inc.; http://wso2.com
> E-mail: sachith AT wso2.com
> M: +94715518127
> Linked-In: 
> https://lk.linkedin.com/in/sachithwithana
>



 --
 *Sinthuja Rajendran*
 Technical Lead
 WSO2, Inc.:http://wso2.com

 Blog: http://sinthu-rajan.blogspot.com/
 Mobile: +94774273955



>>>
>>>
>>> --
>>> Sachith Withana
>>> Software Engineer; WSO2 Inc.; http://wso2.com
>>> E-mail: sachith AT wso2.com
>>> M: +94715518127
>>> Linked-In: 
>>> https://lk.linkedin.com/in/sachithwithana
>>>
>>
>>
>>
>> --
>>
>> *S. Suhothayan*
>> Technical Lead & Team Lead of WSO2 Complex Event Processor
>> *WSO2 Inc. *http://wso2.com
>> * *
>> lean . enterprise . middleware
>>
>>
>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog:
>> http://suhothayan.blogspot.com/ twitter:
>> http://twitter.com/suhothayan  | linked-in:
>> http://lk.linkedin.com/in/suhothayan *
>>
>
>
>
> --
>
> *Tharik Kanaka*
>
> WSO2, Inc |#20, Palm Grove, Colombo 03, Sri Lanka
>
> Email: tha...@wso2.com | Web: www.wso2.com
>



-- 
*V. Mohanadarshan*
*Associate Tech Lead,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] RabbitMQ Input Adopter in CEP

[Mohanadarshan Vivekanandalingam]

Hi Yasothara,

- No need to implement an UI for receiver.. It is dynamically created based
on the properties that you defined in receiver core..
- Keep more time for testing (specially for perf test and long running test)

Other than above all looks good.

Thanks,
Mohan


On Thu, Jul 28, 2016 at 10:16 AM, Yashothara Shanmugarajah <
yashoth...@wso2.com> wrote:

> Hi,
>
> I have herewith attached the link [1] to my milestone plan for the RabbitMQ
> receiver. Please be kind enough to provide necessary feedback, at your
> convenience.
>
> Please let me know if there's anything else that I should work on or that
> you think I missed.
>
> [1]
> https://docs.google.com/a/wso2.com/spreadsheets/d/1g9vNrnJ_TbmZROd7SSo-lVKC-F9F9ruPrs0BlCUQCMU/edit?usp=sharing
>
> ​Thank you.​
>
> / <http://wso2.com/>Best Regards,
> Yashothara.S
> Software Engineer
> WSO2
> http://wso2.com
>
> On Wed, Jul 27, 2016 at 9:44 AM, Yashothara Shanmugarajah <
> yashoth...@wso2.com> wrote:
>
>> (adding architecture group)
>>
>> / <http://wso2.com/>Best Regards,
>> Yashothara.S
>> Software Engineer
>> WSO2
>> http://wso2.com
>>
>> On Wed, Jul 27, 2016 at 9:40 AM, Yashothara Shanmugarajah <
>> yashoth...@wso2.com> wrote:
>>
>>> Hi all,
>>>
>>> I have evaluated reusing the axis2 transport and ESB inbound transports.
>>> If we use Axis2 transport, we need to change configuration in axis2.xml. So
>>> after discussion with Suho and Mohan, I decided to create rabbitMQ from the
>>> scratch.
>>>
>>> Thanks.
>>>
>>> / <http://wso2.com/>Best Regards,
>>> Yashothara.S
>>> Software Engineer
>>> WSO2
>>> http://wso2.com
>>>
>>> On Thu, Jul 7, 2016 at 5:04 PM, Sriskandarajah Suhothayan >> > wrote:
>>>
>>>> @Yashothara
>>>>
>>>> Before deciding please evaluate reusing Axis2 and ESB inbound
>>>> transports, check for performance (like it always builds the message, etc )
>>>> & usability (like we need a restart if we need to modify the adaptor)
>>>>
>>>> Regards
>>>> Suho
>>>>
>>>> On Thu, Jul 7, 2016 at 3:56 PM, Mohanadarshan Vivekanandalingam <
>>>> mo...@wso2.com> wrote:
>>>>
>>>>> Anyway, if you guys have any issues on using axis2 transport for
>>>>> this.. Then please add them in strategy mail thread [1] as well.. We had a
>>>>> discussion on this already but haven't seen any concerns on using axis2
>>>>> transport..
>>>>>
>>>>> [1] RabbitMQ into CEP (was: Re: URGENT: Fw: Potenial use of WSO2 for
>>>>> Global Upstream Oil Industry)
>>>>>
>>>>> Thanks,
>>>>> Mohan
>>>>>
>>>>>
>>>>> On Thu, Jul 7, 2016 at 3:44 PM, Yashothara Shanmugarajah <
>>>>> yashoth...@wso2.com> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> Best Regards,
>>>>>> Yashothara.S
>>>>>>
>>>>>> Software Engineer
>>>>>> WSO2
>>>>>>
>>>>>>
>>>>>> On Thu, Jul 7, 2016 at 3:32 PM, Sriskandarajah Suhothayan <
>>>>>> s...@wso2.com> wrote:
>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> Regards
>>>>>>> Suho
>>>>>>>
>>>>>>> On Thu, Jul 7, 2016 at 3:31 PM, Malaka Silva 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I guess we can check for possibilities of reusing inbound endpoint
>>>>>>>> code here. This way we can identify what is missing to do this from 
>>>>>>>> both
>>>>>>>> sides. WDYT?
>>>>>>>>
>>>>>>>> On Thu, Jul 7, 2016 at 2:48 PM, Sriskandarajah Suhothayan <
>>>>>>>> s...@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> It's better if we can avoid using Axis2 Transport if we need to
>>>>>>>>> restart CEP/DAS for each and every changes.
>>>>>>>>> Please verify.
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Suho
>>>>>>>>>
>>>>>>>>&g

Re: [Architecture] [Arch] Adding CEP and ML samples to DAS distribution in a consistent way

[Mohanadarshan Vivekanandalingam]

On Mon, Aug 1, 2016 at 4:02 PM, Niranda Perera  wrote:

> Hi all,
>
> At the moment we are maintaining samples of DAS, CEP and ML in their own
> product repos. Since, DAS integrates both CEP and ML, we need to send these
> samples with DAS.
>
> Currently we do so for ML samples, but the approach we are using is to
> keep a local copy of the samples in the product-das repo. This approach is
> rather problematic, because when there are changes in the original samples,
> we would also have to reflect those changes manually in the product-das
> copy.
>
> Is there a more consistent way to add samples? May be like creating a
> separate samples feature?
>

IMO, best option is remove samples from product distribution and maintain
in a common location (svn or git). In this case, whoever like try the
samples can refer relevant locations..

This is also give us a freedom to add/update  samples time to time without
worrying about product releases..

Thanks,
Mohan


>
> Would like to hear from you regarding this.
>
> Best
>
> --
> *Niranda Perera*
> Software Engineer, WSO2 Inc.
> Mobile: +94-71-554-8430
> Twitter: @n1r44 
> https://pythagoreanscript.wordpress.com/
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*V. Mohanadarshan*
*Associate Tech Lead,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [Dev] [IS] [Analytics] Improvement to use Siddhi streams to send notifications

[Mohanadarshan Vivekanandalingam]

On Mon, Aug 1, 2016 at 8:38 PM, Indunil Upeksha Rathnayake  wrote:

> Hi Suhothayan,
>
> Hi Indunil,

I like to add some comments on this.. Please find them below..


> There was an issue in EventPublisherServiceDS where
> setConfigurationContextService() method get invoked after the bundle get
> activated. Due to that, when we are trying to invoke
> deployEventPublisherConfiguration() of EventPublisherService from the
> activate method of an osgi bundle in IS side, it's receiving a null
> pointer(Since it refers the ConfigurationContextService object in
> EventPublisherServiceValueHolder). I think you can resolve it by changing
> the osgi reference cardinality in [1] as "1..1"(Mandatory), if there is no
> specific reason for making it optional.
>

There is a valid reason for this..
I believe, as you know we cannot guarantee about OSGI bundle loading in
carbon environment.. In this case, there is a possibility where axis2
deployment can start before bundle activation of a OSGI component. To avoid
this we'll follow a similar approach like below,



   org.wso2.carbon.event.publisher.core.EventPublisherService



Here, we are adding the reference of the corresponding OSGI service which
is exposed by relevant OSGI module.. If you want to use above approach
(Axis2RequiredServices), we cannot have 1..1 mapping for
ConfigurationContextService since it causes cyclic dependency and affects
bundle loading..

In IS side we were able to get rid of the null pointer by adding an osgi
> reference for ConfigurationContextService in the service component and
> invoked the deployEventPublisherConfiguration() in activate() method.
>

No, above solution is not correct and will not work all the time.. There is
a possibility where you'll encounter same issue when
ConfigurationContextService is bind to you component first and takes
sometime to resolve for Event Publisher..

What is the usecase for creating an Event Publisher in server restart ? Can
you ship the pack with an Event Publisher or deploy an event publisher for
first event if it is not there..


> And also there was an issue in filling out dynamic properties of an output
> adapter from the arbitrary data values, and sent a PR for that. Please
> review and merge the PR in [2].
>

Thanks, Merged it..

Regards,
Mohan


>
> [1]
> https://github.com/wso2/carbon-analytics-common/blob/master/components/event-publisher/org.wso2.carbon.event.publisher.core/src/main/java/org/wso2/carbon/event/publisher/core/internal/ds/EventPublisherServiceDS.java#L56
> [2] https://github.com/wso2/carbon-analytics-common/pull/306/files
>
> Thanks and Regards
>
> On Mon, Aug 1, 2016 at 3:06 PM, Sriskandarajah Suhothayan 
> wrote:
>
>> HI Indunil
>>
>> Any update on this? Was the provided solution working?
>>
>> We released CEP 4.2-RC1. If we need new features/improvements for this
>> effort, we can incorporate them in the next component release.
>>
>> Regards
>> Suho
>>
>> On Fri, Jul 22, 2016 at 3:10 PM, Sriskandarajah Suhothayan > > wrote:
>>
>>>
>>>
>>> On Fri, Jul 22, 2016 at 3:00 PM, Johann Nallathamby 
>>> wrote:
>>>


 On Fri, Jul 22, 2016 at 8:33 AM, Indunil Upeksha Rathnayake <
 indu...@wso2.com> wrote:

> Hi,
>
> On Fri, Jul 22, 2016 at 12:28 PM, Sriskandarajah Suhothayan <
> s...@wso2.com> wrote:
>
>>
>>
>> On Fri, Jul 22, 2016 at 12:00 PM, Indunil Upeksha Rathnayake <
>> indu...@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> Please find the meeting notes in [1].  I have following
>>> considerations regarding the improvements we have discussed.
>>>
>>> (1) Even though we have configured to load the email template from
>>> EventPublisher(analytics side), the placeholder values has to be sent as
>>> meta data/correlation data/payload data/arbitrary data, since in 
>>> analytics
>>> side, the user claim values are not getting from the user store.
>>> In order to send the placeholder values from IS side, anyway we have
>>> to load the email template and retrieve the placeholders. So as I have
>>> understood, for email notifications, it's not needed to use the email
>>> template loading part in analytics, since it'll be a redundant task. 
>>> (Refer
>>> [2])
>>>
>>
>> Here we can set the claim values as arbitrary data, and the
>> notification specific details as the meta, correlation & payload data.
>> Then we can use the template loading only at the analytics side.
>>
> In this case, from IS side, without parsing only the user claims
> needed for a particular email template(i.e.user claim values for the
> placeholders in email template), we have to pass all the user claims as
> arbitrary data values. In that case there's no need for loading the
> template from the registry in IS side. So that in analytics side, all the
> values needed for filling out the template will be there. Will check on
> that.
>

Re: [Architecture] [Arch] Adding CEP and ML samples to DAS distribution in a consistent way

[Mohanadarshan Vivekanandalingam]

Hmm, I am not against for shipping samples with product distribution.. But
it is not a scalable solution anymore.. When platform story grows we are
tend to reuse features the shipping all the sample within a product
distribution is not scale..

IMHO, samples are used in development stages and installing svn or git in
that case does not cause any concerns.. If we want to debug and verify the
functionality then can't we simply copy the samples and tryout?..

Thanks,
Mohan


On Mon, Aug 1, 2016 at 5:00 PM, Gihan Anuruddha  wrote:

> I see two downsides from above approach.
>
> 1. Users have to install svn or git in their machine (This is a small
> issue, but unnecessary dependency).
> 2. Sometimes when we are facing some issues, easiest way for us is, run
> the sample and see things are working fine. Specially in a remote machine.
> We lose that freedom.
>
> Above are minor problems. But bit inconvenient for end users.
>
> Regards,
> Gihan
>
> On Mon, Aug 1, 2016 at 4:32 PM, Mohanadarshan Vivekanandalingam <
> mo...@wso2.com> wrote:
>
>>
>>
>> On Mon, Aug 1, 2016 at 4:02 PM, Niranda Perera  wrote:
>>
>>> Hi all,
>>>
>>> At the moment we are maintaining samples of DAS, CEP and ML in their own
>>> product repos. Since, DAS integrates both CEP and ML, we need to send these
>>> samples with DAS.
>>>
>>> Currently we do so for ML samples, but the approach we are using is to
>>> keep a local copy of the samples in the product-das repo. This approach is
>>> rather problematic, because when there are changes in the original samples,
>>> we would also have to reflect those changes manually in the product-das
>>> copy.
>>>
>>> Is there a more consistent way to add samples? May be like creating a
>>> separate samples feature?
>>>
>>
>> IMO, best option is remove samples from product distribution and maintain
>> in a common location (svn or git). In this case, whoever like try the
>> samples can refer relevant locations..
>>
>> This is also give us a freedom to add/update  samples time to time
>> without worrying about product releases..
>>
>> Thanks,
>> Mohan
>>
>>
>>>
>>> Would like to hear from you regarding this.
>>>
>>> Best
>>>
>>> --
>>> *Niranda Perera*
>>> Software Engineer, WSO2 Inc.
>>> Mobile: +94-71-554-8430
>>> Twitter: @n1r44 <https://twitter.com/N1R44>
>>> https://pythagoreanscript.wordpress.com/
>>>
>>> ___
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> --
>> *V. Mohanadarshan*
>> *Associate Tech Lead,*
>> *Data Technologies Team,*
>> *WSO2, Inc. http://wso2.com <http://wso2.com> *
>> *lean.enterprise.middleware.*
>>
>> email: mo...@wso2.com
>> phone:(+94) 771117673
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> W.G. Gihan Anuruddha
> Senior Software Engineer | WSO2, Inc.
> M: +94772272595
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*V. Mohanadarshan*
*Associate Tech Lead,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com <http://wso2.com> *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [Dev] [IS] [Analytics] Improvement to use Siddhi streams to send notifications

[Mohanadarshan Vivekanandalingam]

On Thu, Aug 11, 2016 at 11:32 AM, Indunil Upeksha Rathnayake <
indu...@wso2.com> wrote:

> Hi Suhothayan,
>
> You can refer [1] for the current approach we have taken in IS side when
> improving notification sending with siddhi streams. As per the discussion
> we had previously, this approach has been taken in order to avoid the
> performance degradation due to the redundant loading of email template in
> IS and analytics. The main reason for the redundant loading is that only in
> IS side, the user claims can be loaded which needs for filling out the
> placeholders in email template.
>
> As per the current implementation you are having, we can provide the
> registry path and let the email template get loaded in analytics side. For
> that there has to be some improvement in analytics side to get the user
> claims from user store and filling out the template with those claim
> values. So that without loading the email template from IS side, we can do
> it in analytics side.
>
> So the suggested improvements as follows.
> *IS side:*
>
>
>
>
>
>
> *1) Modified the publisher definition to include registry path of the
> email template, specifying the notification type and locale as
> placeholders2) When an email notification need to be send, an arbitrary map
> (including the data needs to load the email template from registry) will be
> published to the streamAnalytics side:1) Load the email template from the
> registry (use the arbitrary data values we have provided)2) Extract the
> placeholders in email template3) Get the user claims from user store and
> fill out the placeholders in the template with the necessary claim values*
>

Above [1] and [2] are already implemented in Event Publisher level and can
be usable for above usecase.. But [2] (which is mentioned as an improvement
for Analytics side) is not valid IMO.. That is not something that we need
to handle in analytics or Event Publisher side, it is architecturally
incorrect to handle in that level.. What need to be done is, we need to get
those claims from identity level and send those relevant claims in the
event itself..

What is the reason for defining [3] as an improvement for analytics ?

Thanks,
Mohan


>
>
> We have used two prefixes in placeholders of email templates as
> "user.claim.identity" and "user.claim", in order to specify that the
> placeholders has to be filled with an identity claim and other wso2 claim
> respectively. The claim URIs which we are using when retrieving necessary
> user claims for the email templates, will be generated appending necessary
> prefix to the "http://wso2.org/claims/";. As an example if the placeholder
> is "user.claim.givenname", the claim URI should be "
> http://wso2.org/claims/givenname";. So that placeholder has to be filled
> with the user claim value corresponding to the above mentioned claim URI.
> You can refer [2] for the implementation done in IS side, we can move that
> logic to analytics side.
>
> [1] https://github.com/wso2-extensions/identity-event-
> handler-notification/pull/26/files
> [2] https://github.com/wso2-extensions/identity-event-
> handler-notification/pull/26/files#diff-2200b351eeef81ebbb5ea7f0d1f1ec
> b7R119
>
> Thanks and Regards
>
> On Tue, Aug 9, 2016 at 9:50 PM, Sriskandarajah Suhothayan 
> wrote:
>
>> Based on the chat with Johann he suggested to support claims at event
>> publisher.
>> @Indunil, can you get the full requirements and update the thread.
>>
>> Regards
>> Suho
>>
>> On Mon, Aug 1, 2016 at 11:24 PM, Mohanadarshan Vivekanandalingam <
>> mo...@wso2.com> wrote:
>>
>>>
>>>
>>> On Mon, Aug 1, 2016 at 8:38 PM, Indunil Upeksha Rathnayake <
>>> indu...@wso2.com> wrote:
>>>
>>>> Hi Suhothayan,
>>>>
>>>> Hi Indunil,
>>>
>>> I like to add some comments on this.. Please find them below..
>>>
>>>
>>>> There was an issue in EventPublisherServiceDS where
>>>> setConfigurationContextService() method get invoked after the bundle
>>>> get activated. Due to that, when we are trying to invoke
>>>> deployEventPublisherConfiguration() of EventPublisherService from the
>>>> activate method of an osgi bundle in IS side, it's receiving a null
>>>> pointer(Since it refers the ConfigurationContextService object in
>>>> EventPublisherServiceValueHolder). I think you can resolve it by
>>>> changing the osgi reference cardinality in [1] as "1..1"(Mandatory), if
>>>> there is no specific reason for making it optional.
>>>>
>>>
>&

Re: [Architecture] [Dev] [IS] [Analytics] Improvement to use Siddhi streams to send notifications

[Mohanadarshan Vivekanandalingam]

On Thu, Aug 11, 2016 at 11:45 PM, Sriskandarajah Suhothayan 
wrote:

> I think getting the claims will improve the message formatting when
> sending the message, and based on the discussion with Johann they cannot
> determine what claims the message formatting will need. if IS need to send
> the claims it has to also read the message to understand the necessary
> claims.
>
> Any suggestions ?
>

Hmm.. Message is created/originated from IS side, Isn't it ? Then, why
can't we get whatever claims that required and send them.. Sorry, I don't
see any overall impact because of that (Only thing is there going to be
another registry access to get the template and load relevant claims). If
this is an huge overhead then can't we send all the claims with in the
event (Not sure whether it is acceptable)..

But IMO, this is a notification task isn't. Then, I don't see a need to
worry much about performance and throughput.. And I don't think it is
correct to change the generic notification event publisher component as
something which does some IS specific operation which violates its design
expectation and looks like a hack ..

Let's have a quick chat if that help..

Thanks,
Mohan



>
> Regards
> Suho
>
> On Thu, Aug 11, 2016 at 4:42 PM, Mohanadarshan Vivekanandalingam <
> mo...@wso2.com> wrote:
>
>>
>>
>> On Thu, Aug 11, 2016 at 11:32 AM, Indunil Upeksha Rathnayake <
>> indu...@wso2.com> wrote:
>>
>>> Hi Suhothayan,
>>>
>>> You can refer [1] for the current approach we have taken in IS side when
>>> improving notification sending with siddhi streams. As per the discussion
>>> we had previously, this approach has been taken in order to avoid the
>>> performance degradation due to the redundant loading of email template in
>>> IS and analytics. The main reason for the redundant loading is that only in
>>> IS side, the user claims can be loaded which needs for filling out the
>>> placeholders in email template.
>>>
>>> As per the current implementation you are having, we can provide the
>>> registry path and let the email template get loaded in analytics side. For
>>> that there has to be some improvement in analytics side to get the user
>>> claims from user store and filling out the template with those claim
>>> values. So that without loading the email template from IS side, we can do
>>> it in analytics side.
>>>
>>> So the suggested improvements as follows.
>>> *IS side:*
>>>
>>>
>>>
>>>
>>>
>>>
>>> *1) Modified the publisher definition to include registry path of the
>>> email template, specifying the notification type and locale as
>>> placeholders2) When an email notification need to be send, an arbitrary map
>>> (including the data needs to load the email template from registry) will be
>>> published to the streamAnalytics side:1) Load the email template from the
>>> registry (use the arbitrary data values we have provided)2) Extract the
>>> placeholders in email template3) Get the user claims from user store and
>>> fill out the placeholders in the template with the necessary claim values*
>>>
>>
>> Above [1] and [2] are already implemented in Event Publisher level and
>> can be usable for above usecase.. But [2] (which is mentioned as an
>> improvement for Analytics side) is not valid IMO.. That is not something
>> that we need to handle in analytics or Event Publisher side, it is
>> architecturally incorrect to handle in that level.. What need to be done
>> is, we need to get those claims from identity level and send those relevant
>> claims in the event itself..
>>
>> What is the reason for defining [3] as an improvement for analytics ?
>>
>> Thanks,
>> Mohan
>>
>>
>>>
>>>
>>> We have used two prefixes in placeholders of email templates as
>>> "user.claim.identity" and "user.claim", in order to specify that the
>>> placeholders has to be filled with an identity claim and other wso2 claim
>>> respectively. The claim URIs which we are using when retrieving necessary
>>> user claims for the email templates, will be generated appending necessary
>>> prefix to the "http://wso2.org/claims/";. As an example if the
>>> placeholder is "user.claim.givenname", the claim URI should be "
>>> http://wso2.org/claims/givenname";. So that placeholder has to be filled
>>> with the user claim value corresponding to the above mentioned claim URI.
>>> You ca

Re: [Architecture] [Dev] [IS] [Analytics] Improvement to use Siddhi streams to send notifications

[Mohanadarshan Vivekanandalingam]

On Mon, Aug 15, 2016 at 3:07 AM, Johann Nallathamby  wrote:

>
>
>>
>> But IMO, this is a notification task isn't. Then, I don't see a need to
>> worry much about performance and throughput..
>>
>
> Performance is not the issue here.
>
>
>> And I don't think it is correct to change the generic notification event
>> publisher component as something which does some IS specific operation
>> which violates its design expectation and looks like a hack ..
>>
>
> That is not quite correct Mohan. Adding claim values to email template is
> nothing IS specific. The CEP's output adaptor framework is used by many
> WSO2 products for notifications. This is a common and valid requirement for
> all the products IMHO. It is with that intension we implemented HTML
> templating support and multi language template support in CEP itself.
> Similarly this is also another requirement.
>
> And I also don't think it is violating the existing architecture.
>
> I would think the same if the adaptors are implemented like a library
> which don't have any other carbon runtime dependency. However fetching the
> email template from registry is also done within the adaptor. So there is
> no good explanation to say fetching user claims from user store violates
> the adaptor design.
>
> If there was a layer before the adaptor layer which handles fetching email
> templates from registry, then in the same layer we can enrich the event
> stream with user claims defined in that template. So by the time the event
> reaches the adaptor all necessary values are in the stream.
>
> What I am saying is whether this code is in the adaptor or in a higher
> layer, ultimately it must be in CEP for all the products to use it. That is
> why we reckon that it is an improvement for CEP.
>

Thanks Johann for the explanation.. I like to add some comments on above..
Event Publisher component consists of two layers, they are event enrichment
layer and transport layer.. Calling registry to get the template, multi
language support and HTML things are acceptable in event enrichment layer
since they used to enhance the event but retrieving claims to create/fill
the event cannot be considered as event enrichment IMHO..

Eventhough IS only sends event specific properties, it sends those
properties through stream then they are considered as event in the eventing
context.. In Event publisher layer, we can do enrichment for that event..

Anyway, Suho and myself had a quick offline chat last Friday.. After that,
I also have a mixed opinion like Suho have.. Anyway, we thought best
approach would be providing an extension point for event enrichment layer..
Then whenever need to perform any operation like above we can use that
extension.. Hope that solves above requirement..

Regards,
Mohan


>
> Regards,
> Johann.
>
>
>> Let's have a quick chat if that help..
>>
>> Thanks,
>> Mohan
>>
>>
>>
>>>
>>> Regards
>>> Suho
>>>
>>> On Thu, Aug 11, 2016 at 4:42 PM, Mohanadarshan Vivekanandalingam <
>>> mo...@wso2.com> wrote:
>>>
>>>>
>>>>
>>>> On Thu, Aug 11, 2016 at 11:32 AM, Indunil Upeksha Rathnayake <
>>>> indu...@wso2.com> wrote:
>>>>
>>>>> Hi Suhothayan,
>>>>>
>>>>> You can refer [1] for the current approach we have taken in IS side
>>>>> when improving notification sending with siddhi streams. As per the
>>>>> discussion we had previously, this approach has been taken in order to
>>>>> avoid the performance degradation due to the redundant loading of email
>>>>> template in IS and analytics. The main reason for the redundant loading is
>>>>> that only in IS side, the user claims can be loaded which needs for 
>>>>> filling
>>>>> out the placeholders in email template.
>>>>>
>>>>> As per the current implementation you are having, we can provide the
>>>>> registry path and let the email template get loaded in analytics side. For
>>>>> that there has to be some improvement in analytics side to get the user
>>>>> claims from user store and filling out the template with those claim
>>>>> values. So that without loading the email template from IS side, we can do
>>>>> it in analytics side.
>>>>>
>>>>> So the suggested improvements as follows.
>>>>> *IS side:*
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *1) Modified the publ

Re: [Architecture] [Dev] [APIM] Problem with integrating Event Stream OSGi Service

[Mohanadarshan Vivekanandalingam]

HI Malintha/Nuwan,

I have few queries and like to provide few clarifications on this..

Actually most of the points that you guys raised are discussed before and
changed based on some experiences..

- Defining stream from client side  : We had this in old datapublisher
component and removed due to various complications.. As Srinath mentioned
this was done after many rounds of discussions..

- Having single configuration file to add DAS endpoint : In past, we had
two level of configs (one for adapter which deals with transport and one
for event formatting which deals with stream). That means we can use one
event adapter for multiple streams.. But, there were some complaints saying
it is not user friendly and should have one config to publish events..

And also, event publisher is also another deployable artifact like other
synapse files.. In APIM, we are configuring key manager url in multiple api
configs (revoke api and etc.. ), event publisher is also similar to that..

But, I do understand that your points are valid and think we have two
options..

- Have a global property in output-adapter-config.xml and use that in
Thrift event publisher (if that property exists)..
- Allow to keep the DAS endpoint as a registry resource and use that..

Thanks,
Mohan


On Wed, Aug 24, 2016 at 10:05 AM, Nuwan Dias  wrote:

> Hi,
>
> I spoke with Srinath regarding this too. It would be ideal to have a
> global config which can override the per-stream configs so that we won't
> have to maintain the DAS configs in multiple places. In all cases we've
> come across so far we've only seen people use a single DAS (or cluster) and
> publish all streams to that DAS cluster. So at least as of now I don't see
> much value in having per stream config files.
>
> It would also be good if we can omit the necessity of having to define the
> streams on the client side (because they're already defined on the server).
> I see two main problems of having to define them on the clients.
>
> 1. When we have multiple event generators (a gateway cluster), the streams
> will have to be defined on all nodes. Do we use dep-sync to sync up the
> configs on the clients? Even if we do, I think its better to not depend on
> dep-sync for the new stuff we're developing because we're moving away from
> dep-sync soon.
>
> 2. Problems in upgrading to newer product versions. When upgrading the
> product version, if a schema change has occurred, currently we only have to
> worry about changing the schema on the server side. With the above
> limitation, we now have to worry about changing the clients too.
>
> Thanks,
> NuwanD.
>
> On Tue, Aug 23, 2016 at 10:16 PM, Malintha Amarasinghe  > wrote:
>
>> Hi All,
>>
>> Currently APIM is using an internal APIM specific configuration resides
>> in api-manager.xml which include DAS server URL, username, password etc.
>> Those configuration are used to instantiate an org.wso2.carbon.databridge.
>> agent.DataPublisher object, which is then used to publish events
>> directly to streams in DAS.
>>
>> As it is an APIM specific configuration, it is not reusable by other
>> non-APIM features. Also org.wso2.carbon.databridge.agent.DataPublisher is
>> intended to be used in non-carbon environments, for carbon environments,
>> the recommended way is to use the Event Stream OSGi Service.
>>
>> So we were trying to re-structure APIM data publishing code to use new
>> Event Stream OSGi service explained in [1].
>>
>> 1. Create all the streams defined in DAS in APIM.
>> 2. Create an *event publisher per each stream *which takes data from the
>> stream and publish to DAS.
>> 3. In each event publisher we need to configure DAS specific
>> configuration.
>>
>> Ex:
>> 
>> http://wso2.org/carbon/eventpublisher";>
>>   
>>   
>>   
>> admin
>> thrift
>> non-blocking
>> 0
>> tcp://localhost:7611
>> 
>>   
>> 
>>
>> But we are facing a small problem here; in APIM there are many event
>> streams being used. If there's a change happen to DAS configuration, we
>> need to change it in all the event publishers which makes it difficult to
>> maintain. As per the offline chat with DAS team, this is a current
>> limitation.
>>
>> Are we going to move forward with the existing implementation?
>>
>> Thanks,
>> Malintha
>>
>> [1] [Dev] Common configuration for publishing events from carbon servers
>> to DAS/CEP
>> --
>> Malintha Amarasinghe
>> Software Engineer
>> *WSO2, Inc. - lean | enterprise | middleware*
>> http://wso2.com/
>>
>> Mobile : +94 712383306
>>
>
>
>
> --
> Nuwan Dias
>
> Software Architect - WSO2, Inc. http://wso2.com
> email : nuw...@wso2.com
> Phone : +94 777 775 729
>
> ___
> Dev mailing list
> d...@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 
*V. Mohanadarshan*
*Associate Tech Lead,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673

Re: [Architecture] [Dev] [APIM] Problem with integrating Event Stream OSGi Service

[Mohanadarshan Vivekanandalingam]

On Thu, Aug 25, 2016 at 11:33 AM, Malintha Amarasinghe 
wrote:

> Hi,
>
> On Wed, Aug 24, 2016 at 11:02 AM, Mohanadarshan Vivekanandalingam <
> mo...@wso2.com> wrote:
>
>> HI Malintha/Nuwan,
>>
>> I have few queries and like to provide few clarifications on this..
>>
>> Actually most of the points that you guys raised are discussed before and
>> changed based on some experiences..
>>
>> - Defining stream from client side  : We had this in old datapublisher
>> component and removed due to various complications.. As Srinath mentioned
>> this was done after many rounds of discussions..
>>
>> - Having single configuration file to add DAS endpoint : In past, we had
>> two level of configs (one for adapter which deals with transport and one
>> for event formatting which deals with stream). That means we can use one
>> event adapter for multiple streams.. But, there were some complaints saying
>> it is not user friendly and should have one config to publish events..
>>
>> And also, event publisher is also another deployable artifact like other
>> synapse files.. In APIM, we are configuring key manager url in multiple api
>> configs (revoke api and etc.. ), event publisher is also similar to that..
>>
>> But, I do understand that your points are valid and think we have two
>> options..
>>
>> - Have a global property in output-adapter-config.xml and use that in
>> Thrift event publisher (if that property exists)..
>>
>
@All, I have incorporated above fix by introducing a global default
property in master branch

Thanks,
Mohan


>

> - Allow to keep the DAS endpoint as a registry resource and use that..
>>
>> Thanks for the response. It would be great to have above in order to
> mitigate the duplicated configuration issues.
>


> Thanks!
> Malintha
>
>> Thanks,
>> Mohan
>>
>>
>> On Wed, Aug 24, 2016 at 10:05 AM, Nuwan Dias  wrote:
>>
>>> Hi,
>>>
>>> I spoke with Srinath regarding this too. It would be ideal to have a
>>> global config which can override the per-stream configs so that we won't
>>> have to maintain the DAS configs in multiple places. In all cases we've
>>> come across so far we've only seen people use a single DAS (or cluster) and
>>> publish all streams to that DAS cluster. So at least as of now I don't see
>>> much value in having per stream config files.
>>>
>>> It would also be good if we can omit the necessity of having to define
>>> the streams on the client side (because they're already defined on the
>>> server). I see two main problems of having to define them on the clients.
>>>
>>> 1. When we have multiple event generators (a gateway cluster), the
>>> streams will have to be defined on all nodes. Do we use dep-sync to sync up
>>> the configs on the clients? Even if we do, I think its better to not depend
>>> on dep-sync for the new stuff we're developing because we're moving away
>>> from dep-sync soon.
>>>
>>> 2. Problems in upgrading to newer product versions. When upgrading the
>>> product version, if a schema change has occurred, currently we only have to
>>> worry about changing the schema on the server side. With the above
>>> limitation, we now have to worry about changing the clients too.
>>>
>>> Thanks,
>>> NuwanD.
>>>
>>> On Tue, Aug 23, 2016 at 10:16 PM, Malintha Amarasinghe <
>>> malint...@wso2.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> Currently APIM is using an internal APIM specific configuration resides
>>>> in api-manager.xml which include DAS server URL, username, password etc.
>>>> Those configuration are used to instantiate an org.wso2.carbon.databridge.
>>>> agent.DataPublisher object, which is then used to publish events
>>>> directly to streams in DAS.
>>>>
>>>> As it is an APIM specific configuration, it is not reusable by other
>>>> non-APIM features. Also org.wso2.carbon.databridge.agent.DataPublisher is
>>>> intended to be used in non-carbon environments, for carbon environments,
>>>> the recommended way is to use the Event Stream OSGi Service.
>>>>
>>>> So we were trying to re-structure APIM data publishing code to use new
>>>> Event Stream OSGi service explained in [1].
>>>>
>>>> 1. Create all the streams defined in DAS in APIM.
>>>> 2. Create an *event publisher per each stream *which 

Re: [Architecture] Adding reusable PDF table generator OSGI component to IS 6.0.0 Analytics

[Mohanadarshan Vivekanandalingam]

I believe it is a non-carbon component then it needs to go
shared-analytics. If it is a carbon component, then it needs to go
carbon-analytics-common.


Thanks,
Mohan

On Tue, Feb 7, 2017 at 4:02 PM, Danoja Dias  wrote:

> Hi All,
>
> We decided to write a reusable OSGI component to generate PDF with tables
> using PDFBOX library that is already used in IS pack for PDF generation
> in server side.
>
> We have implemented this in the following location [1] under PR [2]
>
> The usage of this component is described here [3] .
>
> Since this functionality seems common for all analytics, Would it be
> better to add this component in "shared-analytics" instead of within IS
> analytics ?
>
>
>
> [1] https://github.com/wso2/analytics-is/tree/master/components
> [2] https://github.com/wso2/analytics-is/pull/330/files
> [3] https://github.com/DanojaDias/ServersidePDFGenerator/
> blob/master/README.md
>
> --
> Best Regards,
> Danoja Dias
> Intern Software Engineering - WSO2
>
> Email : dan...@wso2.com
> Mobile : +94771160393 <+94%2077%20116%200393>
>
> [image: http://wso2.com/signature] 
>



-- 
*V. Mohanadarshan*
*Associate Tech Lead,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] tenant specific MQTT receivers in DAS , not listening to topics once tenant get unloaded

[Mohanadarshan Vivekanandalingam]

Yes, I had a chat with Jasintha on this as well.

In this case, we have a topic for each tenant since there are some
permission and authorizations involved with each tenant. Then we need to
connect to multiple topics to consume events from multiple tenants. This
means we need to maintain multiple subscriptions from DAS (1 subscription
for 1 topic).

Now the question is how to maintain tenant specific subscriptions in DAS.
Since we are using MQTT here, eager loading is the option that we suggest
to deploy tenant specific artifacts in server startup but since this is
cloud scenario, loading all the tenants in the startup is not an option as
suggested by Sinthuja. Then good option is maintaining the subscriptions of
tenants in super tenant level and push events by starting the tenant flow
based on the topic that we received the message. Here I believe, it is good
to have 1 receiver artifact for each tenant then if need to remove a
subscription of tenant then we can simply delete relevant artifact.

Thanks,
Mohan


On Mon, Mar 20, 2017 at 2:05 PM, Sinthuja Ragendran 
wrote:

> Hi Ayyoob,
>
> On Mon, Mar 20, 2017 at 1:54 PM, Ayyoob Hamza  wrote:
>
>> Agree that it's better to handle this in super tenant layer due to its
>> complexity, but I wonder whether this needs to be specific for
>> MQTT scenarios. Because we need the same requirement for HTTP and thrift
>> receivers. So for HTTP and Thrift can we have it through super tenant space
>> or do we have to think of a generic solution on how to deploy the artifacts
>> for the tenants.
>>
>
> Thrift case is different, because there is only one listener for all
> tenants and we don't start different endpoints/listeners per tenant. And
> the tenent flow is started, once the tenant is authenticated and sending
> the events. In the case of HTTP, we do have the same problem as MQTT. In
> HTTP case, there is seperate HTTP endpoints registered for different
> tenants, and AFAIR those will be removed once the tenant is unloaded. So
> far, we didn't have problems in the deployments as HTTP or other receivers
> than thrift is used in the multi tenant case.
>
>>
>> In addition to recievers, we are using publishers and
>> tenant-specific summarisation scripts, does these gets undeployed when the
>> tenant is unloaded ?.
>>
>
> Summarization scripts will work, AFAIR the tasks registered for the
> tenants will be still active though the tenant is unloaded, and once the
> trigger has been received for the task, it will be loading the tenant and
> executing the script. For the event publishers, the same issue exists, but
> before the publisher gets tiggered the event should be arrived into the
> core, and by that time the tenant is already loaded and hence I believe the
> publishers will not be having this problem.
>
> Thanks,
> Sinthuja.
>
>
>>
>> Thanks,
>> *Ayyoob Hamza*
>> *Software Engineer*
>> WSO2 Inc.; http://wso2.com
>> email: ayy...@wso2.com cell: +94 77 1681010 <%2B94%2077%207779495>
>>
>
>
>
> --
> *Sinthuja Rajendran*
> Technical Lead
> WSO2, Inc.:http://wso2.com
>
> Blog: http://sinthu-rajan.blogspot.com/
> Mobile: +94774273955 <+94%2077%20427%203955>
>
>
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*V. Mohanadarshan*
*Associate Tech Lead,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673 <+94%2077%20111%207673>
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] DevS CEP Plugin

[Mohanadarshan Vivekanandalingam]

Hi Suho,

Please find my comments below as per my understanding.

On Thu, Nov 14, 2013 at 11:43 AM, Sriskandarajah Suhothayan
wrote:

>
> Hi all,
>
> As we are planing to go for the CEP 3.0.0 plugin, I think we have to focus
> more on the its GUI and the usability aspects of it.
>
> CEP has two main concepts
> 1. Streams
> 2. Execution Plan.
>
> Execution Plan creation can look like the CEP 3.0.0 UI.
>
> But for Streams I think we have to do some Improvements. I'm not expecting
> all this to be done for the next release, but this kind of a long term
> vision, we have to find what should be and can be done now and execute
> them. Please give your comments and improvements.
>
> *1*.We need to have some sort of virtual Stream Store in DevS itself,
> this will allow us to select streams from drop down at the Execution Plan
> creation GUI.
>
> 1.1 This Stream Store will be populated by connecting DevS with CEP and/or
> by exporting Streams from CEP and importing to DevS  and/or through
> configs.
>
> (for now we'll go with configs)
>
>
At the moment, In CEP streams are stored in the registry(but there is a
config file also which used for samples but not used in runtime),  Then are
we connecting to registry to get stream info?? Is it possible to do in DevS.


> *2*. We can have a similar UI of CEP for Stream creation
>

+1.

>
> *3*. Event Builder and Formatters will be associated to the Streams.
>
> 3.1 Stream listing UI will list its associated Builders and Formatters
> under it.  Event Builder and Formatters won't have a separate listing
> page/GUI. Therefore Builder and Formatter can be only created after
> creating the Stream.
>
>
> 3.2 Need to figure out a proper way to export new/modified streams and
> apply that to CEP.
>
>
> 3.3 Event Formatter creation GUI can look like the current CEP 3.0.0 UI.
>
>
> 3.4 Event Builder GUI need to be fixed, the Event Builder GUI also need to
> use drop down to select the Stream. The mapping form need to be auto
> created based on the selected stream whereby only allowing the user to fill
> the incoming message related info.
>
>
I think, you saying a drop down for input streams in Event Builder, But
there will streams as input only for WSO2Event adaptor; for other adaptor
types it is different like topic and etc.. Isn't it ?.. Then agree to have
a dropdown for wso2Event adaptor related builders only.


>
> *4*. Input and Output Adapter types and their Message configurations
> fields for the Event Builder and Formatter need to be imported to the DevS.
>
> 4.1 The available Adapter types and their Message configurations fields
> will be imported by connecting DevS with CEP and/or by exporting from CEP
> and importing to DevS  and/or through configs.
>
> (for now we'll go with configs)
>
>
Input or output adaptors and Message configurations fields are not stored
as any config files, they needs to be retrieved from OSGI bundle. That
means DevS needs to communicate with admin service or something to get the
necessary info?? I am not sure whether these are possible..

@Harshana - needs to confirm this..


Regards,
Mohan


>
> Any suggestions appreciated!
>
> Regards
> Suho
>
> --
>
> *S. Suhothayan *
> Associate Technical Lead,
>  *WSO2 Inc. *http://wso2.com
> * *
> lean . enterprise . middleware
>
>
> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog:
> http://suhothayan.blogspot.com/ twitter:
> http://twitter.com/suhothayan  | linked-in:
> http://lk.linkedin.com/in/suhothayan *
>
>


-- 
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] DevS CEP Plugin

[Mohanadarshan Vivekanandalingam]

Hi Suho,


On Thu, Nov 14, 2013 at 8:40 PM, Sriskandarajah Suhothayan wrote:

>
>
>
> On Thu, Nov 14, 2013 at 2:07 AM, Harshana Martin wrote:
>
>> Hi All,
>>
>> Please see my comments inline.
>>
>>
>> I think we will also need to go for CEP 3.0.1 release with this plugin,
> because currently there is no CApp Deployer in CEP 3.0.0.
>
>
>> On Thu, Nov 14, 2013 at 11:43 AM, Sriskandarajah Suhothayan <
>> s...@wso2.com> wrote:
>>
>>>
>>> Hi all,
>>>
>>> As we are planing to go for the CEP 3.0.0 plugin, I think we have to
>>> focus more on the its GUI and the usability aspects of it.
>>>
>>> CEP has two main concepts
>>> 1. Streams
>>> 2. Execution Plan.
>>>
>>> Execution Plan creation can look like the CEP 3.0.0 UI.
>>>
>>> But for Streams I think we have to do some Improvements. I'm not
>>> expecting all this to be done for the next release, but this kind of a long
>>> term vision, we have to find what should be and can be done now and execute
>>> them. Please give your comments and improvements.
>>>
>>> *1*.We need to have some sort of virtual Stream Store in DevS itself,
>>> this will allow us to select streams from drop down at the Execution Plan
>>> creation GUI.
>>>
>>> 1.1 This Stream Store will be populated by connecting DevS with CEP
>>> and/or by exporting Streams from CEP and importing to DevS  and/or through
>>> configs.
>>>
>>> (for now we'll go with configs)
>>>
>>>
>> I believe these Streams are stored in the registry. In that we can
>> provide users following options to select a stream as in ESB Editor.
>>
>> 1. From Workspace - Locate and list the stream definitions in the Eclipse
>> Workspace
>> 2. From Registry - Allow user to browse registry of the CEP and select
>> from it.
>>
>> This approach is consistent across our other tools and users will feel
>> comfortable around this since it is the general practice in in DevS.
>>
> Great,
>
> The stream store for CEP can change to Registry, Cassandra, etc. I think
> we need to fix this in the CEP side because now we have issues when
> integrating CEP with BAM. All the stream related calls need to go via
> DataBridge Stream Definition Store. To my understanding Eclipse need to
> call to a Service of DataBridge Stream Definition Store and import the
> streams
> Is this possible ?
> else when we add a config file that need to override the streams in the
> DataBridge Stream Definition Store.
>
>
>>> *2*. We can have a similar UI of CEP for Stream creation
>>>
>>
>> +1
>>
>>>
>>> *3*. Event Builder and Formatters will be associated to the Streams.
>>>
>>> 3.1 Stream listing UI will list its associated Builders and Formatters
>>> under it.  Event Builder and Formatters won't have a separate listing
>>> page/GUI. Therefore Builder and Formatter can be only created after
>>> creating the Stream.
>>>
>>>
>>> 3.2 Need to figure out a proper way to export new/modified streams and
>>> apply that to CEP.
>>>
>>>
>> Correct. Previously we used to have just one file. Now that we have
>> multiple files, we may have to introduce a packaging mechanism for them
>> with a new deployer. Need to discuss this further whether we can reuse the
>> existing Registry Resource artifacts, etc for this and avoid introduction
>> of new packaging mechanism.
>>
> I thinks Capp is good enough, let see.
>
>>
>>> 3.3 Event Formatter creation GUI can look like the current CEP 3.0.0 UI.
>>>
>>>
>>> 3.4 Event Builder GUI need to be fixed, the Event Builder GUI also need
>>> to use drop down to select the Stream. The mapping form need to be auto
>>> created based on the selected stream whereby only allowing the user to fill
>>> the incoming message related info.
>>>
>>>
>> Aslong as Stream has the necessary information to do this, we can do it.
>>
> @Mohan, we need this to all, and not only for WSO2Event,
> E.g in XML/JMS case we add the topic and then we add the xml mapping and
> finally create an output stream.
> My recommendation is we'll add the topic and then select the expected
> output stream from the drop down which will given an easy way to fill the
> xml mapping.
> Does this make sense?
>

Hmm.. IMHO, I'm not feel it makes much sense; it increases the time on
configuration little more and might confuse also..

>
>
>>> *4*. Input and Output Adapter types and their Message configurations
>>> fields for the Event Builder and Formatter need to be imported to the DevS.
>>>
>>> 4.1 The available Adapter types and their Message configurations fields
>>> will be imported by connecting DevS with CEP and/or by exporting from CEP
>>> and importing to DevS  and/or through configs.
>>>
>>> (for now we'll go with configs)
>>>
>>>
>> This is again have to consider how they are persisted in the CEP side at
>> the moment and decide how we should do it.
>>
>
> There is no configs files for this in the CEP side, my suggestion is to
> let the user write one and add that to Eclipse for now, If we can get that
> info by connecting Eclipse to CEP that's great.
>

Important thing is, As you mentioned

Re: [Architecture] Due Nov. 19 CEP 3.0/BAM 2.4 Release Answers

[Mohanadarshan Vivekanandalingam]

Hi Daya,

Great work.. :) Happy to hear that CEP 3.0.0 improved than 2.1.0.. We can
do better analysis in future..

Thanks & Regards,
Mohan

-- 
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [CEP] Adding AMQP Properties to Output Messages using an AMQP Output Adaptor and Event Formatters

[Mohanadarshan Vivekanandalingam]

Hi Imesh,

Based on the information that you given, there are two ways to implement
this which discussed in this thread.

1) Patching the old JMS adaptor to support user defined AMQP properties

2) Writing a new output event adaptor which support for your usecase.

IMHO, as you said better to write a new adaptor (option 2) which matches
for usecase since it is clean and more flexible rather than patching the
jms adaptor. As you mentioned you can re-use jms implementation to achieve
your task.

You can follow the link [1] to write a custom event adaptor.

[1]
http://wso2.com/library/articles/2013/08/writing-custom-event-adaptors-for-cep-3.0.0/

Thanks & Regards,
Mohan



On Sun, Dec 22, 2013 at 5:00 PM, Imesh Gunaratne  wrote:

> Hi,
>
> Thanks Lasantha for your feedback.
>
> No, there won't be any processing happening in CEP based on the AMQP
> properties. The idea is to support the AMQP protocol when writing from CEP
> to the message consumer. As a result AMQP properties could be included in
> messages using event formatters.
>
> Nirmal, the reason why I proposed to implement a new adapter is that JMS
> and AMQP would mean two different things (as you have mentioned) in the
> context of messaging:
>
>- JMS is a standard* messaging API for Java platform*
>- AMQP is a specification which provides a *standard messaging
>protocol across all platforms*.
>
> Therefore I think an AMQP Output Adapter would provide a much clear view
> and add more value to CEP when communicating with AMQP based, non Java
> message consumers.
>
> Yes as I have pointed out earlier the idea is to implement the AMQP Output
> Adapter by re-using the features in JMS Output Adapter.
>
> Thanks
>
>
>
> On Sun, Dec 22, 2013 at 1:18 PM, Lasantha Fernando wrote:
>
>> Hi Imesh,
>>
>> +1 for implementing an AMQP output adapter. I think it will be very
>> useful to add it as a supported adapter type for CEP.
>>
>> Will there be any processing happening based on the AMQP properties
>> within the CEP? Or is it simply that an AMQP message needs to be sent with
>> properties to the AMQP based broker?
>>
>> Thanks,
>> Lasantha
>>
>>
>> On 22 December 2013 10:34, Imesh Gunaratne  wrote:
>>
>>> Hi All,
>>>
>>> In Apache Stratos (incubating) 4.0.0 we are using WSO2 CEP for
>>> aggregating statistics events sent by load balancers and cartridge agents
>>> (agents in VM instances). The aggregated values are used by the Autoscaler
>>> for taking autoscaling decisions.
>>>
>>> Here the events are sent to CEP using the default input adapter (via
>>> Thrift) and the aggregated events are sent to an AMQP based message broker
>>> using the JMS Output Adapter. Event formatters have been used for defining
>>> the formats of the output messages.
>>>
>>> *A Sample Event Formatter:*
>>> 
>>> >>   statistics="disable" trace="enable" xmlns="
>>> http://wso2.org/carbon/eventformatter";>
>>>   
>>>   
>>>
>>> {"average_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>>   
>>>   
>>> >> name="transport.jms.Destination">summarized-health-stats
>>>   
>>> 
>>>
>>> The above is a sample event formatter which defines an aggregated event.
>>> Similarly there is a collection of events sent by CEP to the message
>>> broker. For each event there is an event formatter defined.
>>>
>>>
>>> *The Requirement:*
>>> Here there is a requirement to include a set of AMQP Properties [1] in
>>> the output message. These properties may include name value pairs specific
>>> for each event:
>>>
>>> *A Sample AMQP Message*
>>> Message Properties:
>>>
>>> event_class_name: 
>>> org.apache.stratos.messaging.event.health.stat.AverageInFlightRequests
>>>
>>> Message Body:
>>>
>>> {"average_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>>
>>>
>>> *The Proposal:*
>>> As I understood this functionality is not supported by the JMS Output
>>> Adapter at the moment and I think it's logical since AMQP features should
>>> not be included in JMS Output Adapter.
>>>
>>> As a solution to this requirement I would like to implement an AMQP
>>> Output Adapter on top of the JMS Output Adapter to support this
>>> functionality [2]. I will send more implementation design details in
>>> another mail as I progress.
>>>
>>> Really appreciate your thoughts on this.
>>>
>>> [1]
>>> http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol#Message_format
>>> [2] http://docs.wso2.org/display/CEP300/Output+Event+Adaptors
>>>
>>>
>>> Many Thanks
>>> --
>>> *Imesh Gunaratne*
>>> Technical Lead
>>> WSO2 Inc: http://wso2.com
>>> T: +94 11 214 5345 M: +94 77 374 2057
>>> W: http://imesh.gunaratne.org
>>> Lean . Enterprise . Middleware
>>>
>>>
>>> ___
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> --
>> *Lasantha Fernando*
>

[Architecture] HTTP Input Event Adaptor For CEP

[Mohanadarshan Vivekanandalingam]

Hi All,

We have started working on implementing HTTP input event adaptor for CEP.
Using this adaptor, we can send any type of message (No need to be a
defined format) to CEP for processing.
HTTP event adaptor will have the ability to forward the incoming messages
to a topic, based on the user configuration. Here, we can follow two
approaches on developing the adaptor. We are looking
for a best option based on below,

1) We can have a single http endpoint (eg :
*https://localhost:9443/message_endpoint
*) and all the users can send
events to this specific endpoint. Here user need to set a custom header
which specifying the corresponding topic where events needs to be forwarded.

2) We can create dynamic endpoints based on the configuration given by the
user. For example if the topic is stockQuote then event adaptor can
register a dynamic endpoint like  *https://localhost:9443/endpoint/stockQuote
 *. then users can send events
to corresponding dynamic endpoint.

What would be the best option that we can follow on [1] & [2]. Appreciate
your ideas..

Thanks & Regards,
Mohan


-- 
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] HTTP Input Event Adaptor For CEP

[Mohanadarshan Vivekanandalingam]

Hi All,

I have followed the second approach and implemented the HTTP input event
adaptor since it would be more clear..

Thanks,
Mohan



On Fri, Jan 17, 2014 at 1:31 PM, Sriskandarajah Suhothayan wrote:

> I believe the 2nd approach will be clear because from the URL itself we
> know to which topic/service we are sending the event. And this is also the
> approach used in the old WS-Eventing approach as well.
>
> Suho
>
>
> On Thu, Jan 16, 2014 at 8:09 PM, Mohanadarshan Vivekanandalingam <
> mo...@wso2.com> wrote:
>
>> Hi All,
>>
>> We have started working on implementing HTTP input event adaptor for CEP.
>> Using this adaptor, we can send any type of message (No need to be a
>> defined format) to CEP for processing.
>> HTTP event adaptor will have the ability to forward the incoming messages
>> to a topic, based on the user configuration. Here, we can follow two
>> approaches on developing the adaptor. We are looking
>> for a best option based on below,
>>
>> 1) We can have a single http endpoint (eg : 
>> *https://localhost:9443/message_endpoint
>> <https://localhost:9443/message_endpoint>*) and all the users can send
>> events to this specific endpoint. Here user need to set a custom header
>> which specifying the corresponding topic where events needs to be forwarded.
>>
>> 2) We can create dynamic endpoints based on the configuration given by
>> the user. For example if the topic is stockQuote then event adaptor can
>> register a dynamic endpoint like  *https://localhost:9443/endpoint/stockQuote
>> <https://localhost:9443/endpoint/stockQuote> *. then users can send
>> events to corresponding dynamic endpoint.
>>
>> What would be the best option that we can follow on [1] & [2]. Appreciate
>> your ideas..
>>
>> Thanks & Regards,
>> Mohan
>>
>>
>> --
>> *V. Mohanadarshan*
>> *Software Engineer,*
>> *Data Technologies Team,*
>> *WSO2, Inc. http://wso2.com <http://wso2.com> *
>> *lean.enterprise.middleware.*
>>
>> email: mo...@wso2.com
>> phone:(+94) 771117673
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
>
> *S. Suhothayan*
> Associate Technical Lead,
>  *WSO2 Inc. *http://wso2.com
> * <http://wso2.com/>*
> lean . enterprise . middleware
>
>
> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog:
> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/>twitter:
> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | linked-in:
> http://lk.linkedin.com/in/suhothayan <http://lk.linkedin.com/in/suhothayan>*
>
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com <http://wso2.com> *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] HTTP Input Event Adaptor For CEP

[Mohanadarshan Vivekanandalingam]

On Fri, Jan 17, 2014 at 5:31 PM, Kasun Indrasiri  wrote:
Hi Kasun

>
> You may try to reuse the non-blocking transport implementation[1] (based
> on ESB PTT architecture) done on top of HTTPCore NIO. It doesn't have any
> direct relationship between Siddhi as its a generic non-blocking transport
> implementation.
>
> [1] https://github.com/kasun04/siddhi-esb/tree/master/modules/transports
>

Thanks.. I'll look into this code and see whether we can re-use it for our
implementation and get back to you if we encountered any issues..

Regards,
Mohan




>
>
> On Fri, Jan 17, 2014 at 2:00 PM, Paul Fremantle  wrote:
>
>> How does this relate to the work that Kasun did with his non-blocking
>> HTTP input adapter for CEP?
>>
>> Paul
>>
>>
>> On 16 January 2014 14:39, Mohanadarshan Vivekanandalingam > > wrote:
>>
>>> Hi All,
>>>
>>> We have started working on implementing HTTP input event adaptor for
>>> CEP. Using this adaptor, we can send any type of message (No need to be a
>>> defined format) to CEP for processing.
>>> HTTP event adaptor will have the ability to forward the incoming
>>> messages to a topic, based on the user configuration. Here, we can follow
>>> two approaches on developing the adaptor. We are looking
>>> for a best option based on below,
>>>
>>> 1) We can have a single http endpoint (eg : 
>>> *https://localhost:9443/message_endpoint
>>> <https://localhost:9443/message_endpoint>*) and all the users can send
>>> events to this specific endpoint. Here user need to set a custom header
>>> which specifying the corresponding topic where events needs to be forwarded.
>>>
>>> 2) We can create dynamic endpoints based on the configuration given by
>>> the user. For example if the topic is stockQuote then event adaptor can
>>> register a dynamic endpoint like  
>>> *https://localhost:9443/endpoint/stockQuote
>>> <https://localhost:9443/endpoint/stockQuote> *. then users can send
>>> events to corresponding dynamic endpoint.
>>>
>>> What would be the best option that we can follow on [1] & [2].
>>> Appreciate your ideas..
>>>
>>> Thanks & Regards,
>>> Mohan
>>>
>>>
>>> --
>>> *V. Mohanadarshan*
>>> *Software Engineer,*
>>> *Data Technologies Team,*
>>> *WSO2, Inc. http://wso2.com <http://wso2.com> *
>>> *lean.enterprise.middleware.*
>>>
>>> email: mo...@wso2.com
>>> phone:(+94) 771117673
>>>
>>> ___
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> --
>> Paul Fremantle
>> CTO and Co-Founder, WSO2
>> OASIS WS-RX TC Co-chair, Apache Member
>>
>> UK: +44 207 096 0336
>> US: +1 646 595 7614
>>
>> blog: http://pzf.fremantle.org
>> twitter.com/pzfreo
>> p...@wso2.com
>>
>> wso2.com Lean Enterprise Middleware
>>
>> Disclaimer: This communication may contain privileged or other
>> confidential information and is intended exclusively for the addressee/s.
>> If you are not the intended recipient/s, or believe that you may have
>> received this communication in error, please reply to the sender indicating
>> that fact and delete the copy you received and in addition, you should not
>> print, copy, retransmit, disseminate, or otherwise use the information
>> contained in this communication. Internet communications cannot be
>> guaranteed to be timely, secure, error or virus-free. The sender does not
>> accept liability for any errors or omissions.
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Kasun Indrasiri
> Software Architect
> WSO2, Inc.; http://wso2.com
> lean.enterprise.middleware
>
> cell: +94 77 556 5206
> Blog : http://kasunpanorama.blogspot.com/
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com <http://wso2.com> *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Should we have one place to define event streams?

[Mohanadarshan Vivekanandalingam]

Hi All,

I like to add some more information in to this..

On Mon, Jan 20, 2014 at 6:12 PM, Sinthuja Ragendran wrote:

> Hi,
>
>
> On Mon, Jan 20, 2014 at 5:36 PM, Maninda Edirisooriya wrote:
>
>> Already BAM can create streams with REST API. To facilitate further for a
>> person configuring BAM manually without a toolbox, we should provide a UI
>> to create streams.
>>
>
Yes Maninda, But there is a limitation where event needs to have a strict
format.. But we are implementing a HTTP event adaptor which can be used to
send any type of event (like json, xml and text) and do the processing at
the CEP end (EventBuilder helps to convert these events into common format).


>
> CEP already has UI to create streams [1], I think it's for registry stream
> defintion store. I'm not sure whether it'll support cassandra stream
> definition store at the moment. We need to test and make it compatible with
> Cassandra stream definition store as well for BAM.
>

No, At the moment CEP does not support cassandra as a stream store (only
registry based), because we are directly storing the stream in registry
using stream manger component. But we should use the databridge stream
store component when dealing with stream. But we had some limitations in
stream store component in CEP perspective but Now i think with the current
re-factoring of databridge we can achieve this.

And also we are going to improve the stream store UI to give more
functionality (like editing, sample event generation and etc..).. I'll
share more details in an another thread.

>
>
> (and better if possible to facilitate edit and delete streams) And that UI
>> should support defining indexes when required as current implementation of
>> defining indexes is not nice.
>>
>
> The event streams are common for BAM and CEP hence we should be careful
> when adding indexes, etc to that common UI where it's not relevant to CEP
> usecase. Here we have to load the attributes dynamically depending on what
> stream definition store (cassandra, registry, etc), so that we can add the
> relevant configuration like indexes, etc.
>

+1..


Thanks & Regards,
Mohan


>
> [1] http://docs.wso2.org/display/CEP300/Working+with+Event+Streams
>
> Thanks,
> Sinthuja.
>
>>
>> As REST API fulfils light weight requirements of BAM, we can remove the
>> Thrift based stream creation feature from the data publisher side. In high
>> performance requirement of BAM, creating a stream from publisher side is
>> not valid. They should deploy a toolbox at the beginning instead. Anyway
>> this will make it backward incompatible for older BAM versions.
>>
>> And BTW what about keeping all the toolboxes in a WSO2 Store as discussed
>> earlier in the thread?
>>
>>
>> * Maninda Edirisooriya*
>> Software Engineer
>>
>> *WSO2, Inc.*lean.enterprise.middleware.
>>
>> *Blog* : http://maninda.blogspot.com/
>> *Phone* : +94 777603226
>>
>>
>> On Mon, Jan 20, 2014 at 4:58 PM, Paul Fremantle  wrote:
>>
>>> I'd like to add some requirements in if possible.
>>>
>>> 1) I'd like some really easy dynamic stuff. For example, if I'm using
>>> REST, can I just send a JSON? We could create a stream definition
>>> dynamically with the name based on some part of the URL and compare to
>>> previous definitions for a version number. This isn't going to be
>>> efficient, but in many cases I just want to get on with things.
>>>
>>> I think we have to be aware we have three potentially different usecases
>>> here:
>>> a) High performance BAM and CEP - need the stream defined in advance for
>>> efficiency
>>> b) Ad hoc BAM and logging - maybe we don't want to be so prescriptive /
>>> statically defined
>>> c) Lightweight CEP / prototyping - being able to use a stream like a
>>> dynamic language and do duck typing would be useful.
>>>
>>> 2) I'd like to support both a Registry model but also being able to
>>> point to a stream definition JSON via a URL. This would make this nice and
>>> "webby"/ REST-friendly.
>>>
>>> 3) We need to make it super easy to define a stream, edit stream
>>> definitions, find them, etc.
>>>
>>> Paul
>>>
>>>
>>> On 20 January 2014 11:10, Srinath Perera  wrote:
>>>
 Hi All,

 I am reviving this thread as now we have support to add remove streams
 though CEP UI (and BAM?) and via toolboxes, but we still define the event
 streams with the data publisher each time. (this thread discusses about
 stop doing that).

 Typical CEP/ BAM usecase would go like following

 1) first you identify and define streams
 2) then you write queries
 3) then you write/configure the agents and point to streams they should
 publish to.

 However, we redefine streams from agents every time we publish. This
 has several complications.

 1) If an agent choose to define stream different from queries,
 everything get messed up. (problem was in samples in BAM).
 2) If there are multiple event receivers, we need a way to sync them
 with db everytime w

[Architecture] Changing the representation of streamId in BAM & CEP

[Mohanadarshan Vivekanandalingam]

Hi All,

As you already knew that CEP & BAM works based on stream concept. We are
defining the streams in a strict format. Here streams are uniquely
identified by combination of StreamName and Stream Version (Simply
StreamId). If an event comes to CEP or BAM, we check the streamId of the
event and do processing accordingly. But here we found that, there are some
issues occurred when dispatching events in multiple tenant environment.

*Issue*

For example -  If says Tenant-A created a stream called testStream with the
version is 1.0.0 (streamId:1.0.0), again Tenant-B also creating a stream
called testStream with version 1.0.0. Then there are two streams defined in
CEP with the same streamId in different tenants since it is possible to do.
Then if says there is an event coming to CEP, then CEP only checks the
streamId of the event and redirect to specific stream. In this case since
two tenants are registered a stream with same streamId, it will process the
execution plans of both tenants. Then It will leads to un-acceptable
situation.

If Tenant-A send events for its' execution, then these events leads to
execution of both Tenant-A & Tenant-B. We can detect the tenant id for some
input transports (Thrift, Http and etc) but not for all (jms can send
events without authentication). Then it is not possible for CEP to derive
the tenant specific information from events at all the time.

[image: Inline image 4]


*Solution*

If we make the streamId unique across multiple tenants then we can overcome
this issue. That means prefixing the tenant domain with streamId.

For example - If Tenant-A  (tenant domain is tenantA) created a stream
called testStream with version 1.0.0 then streamId is defined as
*tenantA:testStream:1.0.0
*, If a super tenant created a stream then it defined as
*testStream:1.0.0* without
any prefix. By this approach we can make streamId unique across multiple
tenants.


WDYT ?? Appreciate any ideas or feedback on this..



Thanks & Regards,
Mohan

-- 
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
<>___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Changing the representation of streamId in BAM & CEP

[Mohanadarshan Vivekanandalingam]

Hi All,

Thanks for your ideas...

Look OK as long as we do not make non-tenant case complicated.
>

Yes, I think it will not make much complicate..

>
> When you list streams, would you see the tenantA part? what happen when
> people debug things?
> What would happen with super tenant?
>

Yes.. Since streams are stored in registry, It will only display the tenant
specific streams.. Yes debug will happen with super tenant..

But We have thought of another approach which is more good..

*Solution*

We can check the credentials of an event (for adaptors which could
authenticate) and route event to tenant specific stream at adaptor level
itself.. But this can be only done for adaptors like http, WSO2Event
(thrift) and other adaptors which can authenticate..

If events send through jms (adaptor which cannot authenticate) then event
will goes to other tenants as well..

WDYT??

@Eranda - Yes, I don't think publishing events in cross tenant manner will
not be a valid usecase (but not sure).. But adding tenant id when
publishing events makes things complex..

@Maninda - Yes that also a good approach but we need to refactor complete
databridge component which affect many products..


Thanks,
Mohan



> --Srinath
>
>
>
> On Wed, Jan 22, 2014 at 6:10 PM, Mohanadarshan Vivekanandalingam <
> mo...@wso2.com> wrote:
>
>> Hi All,
>>
>> As you already knew that CEP & BAM works based on stream concept. We are
>> defining the streams in a strict format. Here streams are uniquely
>> identified by combination of StreamName and Stream Version (Simply
>> StreamId). If an event comes to CEP or BAM, we check the streamId of the
>> event and do processing accordingly. But here we found that, there are some
>> issues occurred when dispatching events in multiple tenant environment.
>>
>> *Issue*
>>
>> For example -  If says Tenant-A created a stream called testStream with
>> the version is 1.0.0 (streamId:1.0.0), again Tenant-B also creating a
>> stream called testStream with version 1.0.0. Then there are two streams
>> defined in CEP with the same streamId in different tenants since it is
>> possible to do. Then if says there is an event coming to CEP, then CEP only
>> checks the streamId of the event and redirect to specific stream. In this
>> case since two tenants are registered a stream with same streamId, it will
>> process the execution plans of both tenants. Then It will leads to
>> un-acceptable situation.
>>
>> If Tenant-A send events for its' execution, then these events leads to
>> execution of both Tenant-A & Tenant-B. We can detect the tenant id for some
>> input transports (Thrift, Http and etc) but not for all (jms can send
>> events without authentication). Then it is not possible for CEP to derive
>> the tenant specific information from events at all the time.
>>
>> [image: Inline image 4]
>>
>>
>> *Solution*
>>
>> If we make the streamId unique across multiple tenants then we can
>> overcome this issue. That means prefixing the tenant domain with streamId.
>>
>> For example - If Tenant-A  (tenant domain is tenantA) created a stream
>> called testStream with version 1.0.0 then streamId is defined as  
>> *tenantA:testStream:1.0.0
>> *, If a super tenant created a stream then it defined as
>> *testStream:1.0.0* without any prefix. By this approach we can make
>> streamId unique across multiple tenants.
>>
>>
>> WDYT ?? Appreciate any ideas or feedback on this..
>>
>>
>>
>> Thanks & Regards,
>> Mohan
>>
>> --
>> *V. Mohanadarshan*
>> *Software Engineer,*
>> *Data Technologies Team,*
>> *WSO2, Inc. http://wso2.com <http://wso2.com> *
>> *lean.enterprise.middleware.*
>>
>> email: mo...@wso2.com
>> phone:(+94) 771117673
>>
>
>
>
> --
> 
> Srinath Perera, Ph.D.
>   Director, Research, WSO2 Inc.
>   Visiting Faculty, University of Moratuwa
>   Member, Apache Software Foundation
>   Research Scientist, Lanka Software Foundation
>   Blog: http://srinathsview.blogspot.com/
>   Photos: http://www.flickr.com/photos/hemapani/
>Phone: 0772360902
>



-- 
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com <http://wso2.com> *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
<>___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] CEP UI re-factoring and adding much more functionality

[Mohanadarshan Vivekanandalingam]

On Thu, Jan 30, 2014 at 3:26 PM, Srinath Perera  wrote:

> Mohan, for listing and editing Streams, could you look at integrating WSO2
> Store?
>

Yes sure.. Will look in to that.. Better, if we can have a small chat
regarding this when it is possible...

Thanks,
Mohan





>
> This component MUST be use by both BAM and CEP to list and edit streams.
> Anjana please note also.
>
> --Srinath
>
>
> On Wed, Jan 22, 2014 at 11:32 AM, Sriskandarajah Suhothayan  > wrote:
>
>>
>>
>> On Wed, Jan 22, 2014 at 11:18 AM, Lasantha Fernando wrote:
>>
>>> Hi Mohan,
>>>
>>> +1 for the design. IMO, the in-flow and out-flow UI will be very useful
>>> to get an idea about how the events are flowing, which is currently a bit
>>> lacking in CEP, I think. Great addition!
>>>
>>> Will the user be able to sample events generated in the stream UI to
>>> test a flow, or will that part come under a separate component?
>>>
>>
>> Based on the current plan the Try-it for streams will become a separate
>> component. In future when we have this we can integrate that with the
>> sample event generation UI.
>>
>> Currently the use of Sample event generation UI is, allowing users to
>> create sample events, edit them, and finally copy and send them via
>> curl,JMS, etc..
>>
>> Suho
>>
>>
>>> Thanks,
>>> Lasantha
>>>
>>>
>>>
>>> On 21 January 2014 19:43, Mohanadarshan Vivekanandalingam <
>>> mo...@wso2.com> wrote:
>>>
>>>>
>>>> Hi All,
>>>>
>>>> As you already knew that we have done major improvements and changes in
>>>> CEP 3.0.0 (which is a complete re-write) specially in UI aspect. But we
>>>> found, there are some gaps that we can fix and improve the usability
>>>> experience further. These changes are targeted for next CEP release which
>>>> is version 3.1.0. And below UI improvements also targeted on CEP tooling
>>>> aspect.
>>>>
>>>> Please see the below figures which are mock-up design flow of the event
>>>> stream UI and execution plan UI. Based on the below design we are trying to
>>>> achieve the default-event concepts and also giving opportunity to advanced
>>>> event configurations also. Appreciate any ideas and suggestions on this...
>>>>
>>>> Thanks & Regards,
>>>> Mohan
>>>>
>>>>
>>>> --
>>>> *V. Mohanadarshan*
>>>> *Software Engineer,*
>>>> *Data Technologies Team,*
>>>> *WSO2, Inc. http://wso2.com <http://wso2.com> *
>>>> *lean.enterprise.middleware.*
>>>>
>>>> email: mo...@wso2.com
>>>> phone:(+94) 771117673
>>>>
>>>
>>>
>>>
>>> --
>>> *Lasantha Fernando*
>>> Software Engineer - Data Technologies Team
>>> WSO2 Inc. http://wso2.com
>>>
>>> email: lasan...@wso2.com
>>> mobile: (+94) 71 5247551
>>>
>>
>>
>>
>> --
>>
>> *S. Suhothayan *
>> Associate Technical Lead,
>>  *WSO2 Inc. *http://wso2.com
>> * <http://wso2.com/>*
>> lean . enterprise . middleware
>>
>>
>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog:
>> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/> twitter:
>> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | linked-in:
>> http://lk.linkedin.com/in/suhothayan <http://lk.linkedin.com/in/suhothayan>*
>>
>>
>
>
> --
> 
> Srinath Perera, Ph.D.
>   Director, Research, WSO2 Inc.
>   Visiting Faculty, University of Moratuwa
>   Member, Apache Software Foundation
>   Research Scientist, Lanka Software Foundation
>   Blog: http://srinathsview.blogspot.com/
>   Photos: http://www.flickr.com/photos/hemapani/
>Phone: 0772360902
>



-- 
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com <http://wso2.com> *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [C5] Solution for Patching C5 Servers

[Mohanadarshan Vivekanandalingam]

>
>
>>
>> In the new Patching model. we are hoping to provide, to patch any type of
>> file. This patch folder is going to be used to backup such files. for
>> e.g config files
>>
>> Patching a config file is bit tricky, because based on the deployment
> user might have already tweaked the config file. In this case how our
> patching model works?
> Will it replace the config file removing user's changes? or do we have a
> mechanism of merging the changes(i.e like how we apply patches to svn/git
> )?
>

+1, Good point Suho.. Ya how we are going to handle it ??
Removing user's changes not going to be a solution anyway since it might
leads to many issues.. If we are going to merge the changes, what is the
design level design that you made, but this process cannot slow down
server-start up too much..

 Can you please elaborate the solution please..

Thanks,
Mohan



>
> Thanks
> Suho
>
>
>>
>>>
>>> Thanks,
>>> Lasantha
>>>

>>
>> I will continue the POC for patching c5 servers with the above
>> changes.
>>
>>
> Thanks,
> Manoj
>



 --
 *Afkham Azeez*
 Director of Architecture; WSO2, Inc.; http://wso2.com
 Member; Apache Software Foundation; http://www.apache.org/
 * *
 *email: **az...@wso2.com* 
 * cell: +94 77 3320919 <%2B94%2077%203320919> blog: *
 *http://blog.afkham.org* 
 *twitter: 
 **http://twitter.com/afkham_azeez*
 * linked-in: **http://lk.linkedin.com/in/afkhamazeez
 *

 *Lean . Enterprise . Middleware*

 ___
 Architecture mailing list
 Architecture@wso2.org
 https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


>>>
>>>
>>> --
>>> *Lasantha Fernando*
>>> Software Engineer - Data Technologies Team
>>> WSO2 Inc. http://wso2.com
>>>
>>> email: lasan...@wso2.com
>>> mobile: (+94) 71 5247551
>>>
>>> ___
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> --
>>
>> * Aruna Sujith Karunarathna* | Software Engineer
>> WSO2, Inc | lean. enterprise. middleware.
>> #20, Palm Grove, Colombo 03, Sri Lanka
>> Mobile: +94 71 9040362 | Work: +94 112145345
>> Email: ar...@wso2.com | Web: www.wso2.com
>>
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
>
> *S. Suhothayan*
> Technical Lead & Team Lead of WSO2 Complex Event Processor
>  *WSO2 Inc. *http://wso2.com
> * *
> lean . enterprise . middleware
>
>
>
> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog:
> http://suhothayan.blogspot.com/  twitter:
> http://twitter.com/suhothayan  | linked-in:
> http://lk.linkedin.com/in/suhothayan *
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Using Siddhi Event processor to implement/evaluate some clustering algorithms

[Mohanadarshan Vivekanandalingam]

On Fri, Jun 27, 2014 at 9:05 AM, Srinath Perera  wrote:

> Hi All,
>
> Lahiru and myself had a call today morning.
>
> Plan is to
> 1) Lahiru to look at hoeffding tree and other classification algorithms
> and select one to implement. He will compare the performance against MOA or
> some other implementation.
> 2) then we will use it for a  Fraud analysis scenario as a proof of its
> validity.
>
> Then we will decide how to continue from that point.
>
> Mohan could you look at the patch? Lahiru will write test cases that you
> can use to verify.
>

Sure.. Will do..

Thanks,
Mohan


>
> --Srinath
>
>
> On Tue, Jun 24, 2014 at 4:56 AM, Lahiru Gunathilake 
> wrote:
>
>> Hi Srinath,
>>
>> Thanks for the response.
>> On Jun 23, 2014, at 10:48 PM, Srinath Perera wrote:
>>
>> Hi Lahiru,
>>
>> Sorry for not responding earlier. I was traveling last week.
>>
>> I guess you know frequent item set !=  clustering algorithms.
>>
>> +1
>>
>>
>> Can we have a chat sometime to discuss details about the implementation.
>>
>> Yes sure, that would be very useful.
>>
>> Thanks
>> Lahiru
>>
>>
>> --Srinath
>>
>>
>> On Sun, Jun 22, 2014 at 6:25 PM, Lahiru Gunathilake > > wrote:
>>
>>> Hi All,
>>>
>>> I have implemented another frequency counting algorithm[1] which is a
>>> classic algorithm for mining frequent items in a data stream. Basically
>>> users can specify a minimum average value of frequent items with an error
>>> value.
>>>
>>> This algorithm will accept two user-specified parameters: a support
>>> threshold s [0-1] and an error parameter e [0-1] such that e << s and
>>> recommended number for e is normally s/10 or s/20. Let N denote the
>>> current length
>>> of the stream, i.e., the number of tuples seen so far. At any point of
>>> time, this algorithm can be asked to produce a list of events along
>>> with their estimated frequencies. The answers produced by this
>>> algorithm will have the
>>> following guarantees:
>>> 1. All item(set)s whose true frequency exceeds sN are outputs.
>>>  There are no false negatives .
>>> 2. No events whose true frequency is less than (s - e) N
>>> is output.
>>>  3. Estimated frequencies are less  than the true frequencies
>>> by at most eN.
>>>  .
>>> The incoming stream is conceptually divided in to buckets of width w =
>>> ceiling(1/e) transactions each. Buckets are labeled with bucket ids ,
>>> starting from 1.We denote the current bucket id  by bcurrent whose
>>> value is ceiling(N/w). For an element , we denote its true frequency in the
>>> stream seen so far by "fe"  . Note that e and w are fixed for a data
>>> stream while N, bcurrent and fe are the variables whose value changes when
>>> the stream progress.
>>> Here the data structure ,  is a set of entries are tuples with the form
>>> of (event, f, delta), where event is the actual event and "f" is the  is an
>>> integer representing its estimated frequency, and delta is the maximum
>>> possible error in "fe".
>>>
>>>  In this algorithm every new event will anyways added in to the
>>> data-structure optimistically and there is no initial condition to enter in
>>> to the data-structure but iteratively if certain events are not match to
>>> the condition provided
>>> by the user those events will be removed when N%windowSize = 0, during
>>> this I output those events as expired-events in the window. For the
>>> incoming events if event is already exists I output those as current-events
>>> and if its a new
>>> event I just add it to the data-structure and iterate through all the
>>> events available and find the matching events based on s and e values and
>>> only those events will be output.
>>>
>>> I am not sure based on the window definition this is the correct
>>> approach or may be windows aren't the best way to associate this algorithm
>>> in to siddhi. I have attached my patch to jira[2] and if you can look that
>>> would be great.
>>>
>>> Siddhi Query will looks like below,
>>> from  cseEventStream#window.lossyFrequent(0.1,0.01) " +
>>>"select symbol,
>>> price " +
>>>"insert into
>>> StockQuote;
>>>
>>>
>>> [1]Gurmeet Singh Manku and Rajeev Motwani. 2002. Approximate frequency
>>> counts over data streams. In Proceedings of the 28th international
>>> conference on Very Large Data Bases (VLDB '02). VLDB Endowment 346-357.
>>>
>>> Regards
>>> Lahiru
>>> On Jun 19, 2014, at 2:36 AM, Lahiru Gunathilake wrote:
>>>
>>> Hi All,
>>> On Jun 17, 2014, at 1:49 AM, Lahiru Gunathilake wrote:
>>>
>>> Hi All,
>>>
>>> I am planning to evaluate different event stream clustering algorithms
>>> as part of my studies(I am a graduate student at indiana University). I
>>> think Siddhi is a good place to experiment this, As per my understanding
>>> based on the docs Siddhi doesn't have a stream clustering interface I can
>>> use directly to plug my own algorithm. So I am thinking of first come up an
>>> interface for different 

Re: [Architecture] Using Siddhi Event processor to implement/evaluate some clustering algorithms

[Mohanadarshan Vivekanandalingam]

On Fri, Jun 27, 2014 at 10:00 AM, Lahiru Gunathilake 
wrote:

> Hi Mohan,
>

Hi Lahiru,


>
> I wrote some samples but I can write more test-cases and provide another
> patch. Please feel free to change the naming of the windows as you like.
>

Really appreciate your contribution.. Sure, I'll start look into this..

Thanks,
Mohan


> Regards
> Lahiru
>
> On Jun 26, 2014, at 11:35 PM, Srinath Perera wrote:
>
> Hi All,
>
> Lahiru and myself had a call today morning.
>
> Plan is to
> 1) Lahiru to look at hoeffding tree and other classification algorithms
> and select one to implement. He will compare the performance against MOA or
> some other implementation.
> 2) then we will use it for a  Fraud analysis scenario as a proof of its
> validity.
>
> Then we will decide how to continue from that point.
>
> Mohan could you look at the patch? Lahiru will write test cases that you
> can use to verify.
>
> --Srinath
>
>
> On Tue, Jun 24, 2014 at 4:56 AM, Lahiru Gunathilake 
> wrote:
>
>> Hi Srinath,
>>
>> Thanks for the response.
>> On Jun 23, 2014, at 10:48 PM, Srinath Perera wrote:
>>
>> Hi Lahiru,
>>
>> Sorry for not responding earlier. I was traveling last week.
>>
>> I guess you know frequent item set !=  clustering algorithms.
>>
>> +1
>>
>>
>> Can we have a chat sometime to discuss details about the implementation.
>>
>> Yes sure, that would be very useful.
>>
>> Thanks
>> Lahiru
>>
>>
>> --Srinath
>>
>>
>> On Sun, Jun 22, 2014 at 6:25 PM, Lahiru Gunathilake > > wrote:
>>
>>> Hi All,
>>>
>>> I have implemented another frequency counting algorithm[1] which is a
>>> classic algorithm for mining frequent items in a data stream. Basically
>>> users can specify a minimum average value of frequent items with an error
>>> value.
>>>
>>> This algorithm will accept two user-specified parameters: a support
>>> threshold s [0-1] and an error parameter e [0-1] such that e << s and
>>> recommended number for e is normally s/10 or s/20. Let N denote the
>>> current length
>>> of the stream, i.e., the number of tuples seen so far. At any point of
>>> time, this algorithm can be asked to produce a list of events along
>>> with their estimated frequencies. The answers produced by this
>>> algorithm will have the
>>> following guarantees:
>>> 1. All item(set)s whose true frequency exceeds sN are outputs.
>>>  There are no false negatives .
>>> 2. No events whose true frequency is less than (s - e) N
>>> is output.
>>>  3. Estimated frequencies are less  than the true frequencies
>>> by at most eN.
>>>  .
>>> The incoming stream is conceptually divided in to buckets of width w =
>>> ceiling(1/e) transactions each. Buckets are labeled with bucket ids ,
>>> starting from 1.We denote the current bucket id  by bcurrent whose
>>> value is ceiling(N/w). For an element , we denote its true frequency in the
>>> stream seen so far by "fe"  . Note that e and w are fixed for a data
>>> stream while N, bcurrent and fe are the variables whose value changes when
>>> the stream progress.
>>> Here the data structure ,  is a set of entries are tuples with the form
>>> of (event, f, delta), where event is the actual event and "f" is the  is an
>>> integer representing its estimated frequency, and delta is the maximum
>>> possible error in "fe".
>>>
>>>  In this algorithm every new event will anyways added in to the
>>> data-structure optimistically and there is no initial condition to enter in
>>> to the data-structure but iteratively if certain events are not match to
>>> the condition provided
>>> by the user those events will be removed when N%windowSize = 0, during
>>> this I output those events as expired-events in the window. For the
>>> incoming events if event is already exists I output those as current-events
>>> and if its a new
>>> event I just add it to the data-structure and iterate through all the
>>> events available and find the matching events based on s and e values and
>>> only those events will be output.
>>>
>>> I am not sure based on the window definition this is the correct
>>> approach or may be windows aren't the best way to associate this algorithm
>>> in to siddhi. I have attached my patch to jira[2] and if you can look that
>>> would be great.
>>>
>>> Siddhi Query will looks like below,
>>> from  cseEventStream#window.lossyFrequent(0.1,0.01) " +
>>>"select symbol,
>>> price " +
>>>"insert into
>>> StockQuote;
>>>
>>>
>>> [1]Gurmeet Singh Manku and Rajeev Motwani. 2002. Approximate frequency
>>> counts over data streams. In Proceedings of the 28th international
>>> conference on Very Large Data Bases (VLDB '02). VLDB Endowment 346-357.
>>>
>>> Regards
>>> Lahiru
>>> On Jun 19, 2014, at 2:36 AM, Lahiru Gunathilake wrote:
>>>
>>> Hi All,
>>> On Jun 17, 2014, at 1:49 AM, Lahiru Gunathilake wrote:
>>>
>>> Hi All,
>>>
>>> I am planning to evaluate different event stream clustering algorithms
>>> as part of my stud

Re: [Architecture] Using Siddhi Event processor to implement/evaluate some clustering algorithms

[Mohanadarshan Vivekanandalingam]

On Sat, Jun 28, 2014 at 2:22 PM, Lahiru Gunathilake 
wrote:

> Hi Mohan,
>
> I have attached the latest patch for cep-877(I removed the old patch from
> the jira). But the very first patch I have attached in CEP-873 is required
> by this patch.
>
> I made few improvements to both the algorithms where you can give the
> attributes you want to count. Initial version I did was to count distinct
> tuples but practically I think counting distinct attributes is going to be
> useful. If user doesn't give any attribute I simply count the distinct
> tuple with all the attributes.
>
> I have added two test cases but I can see in the build cluster test cases
> are removed, I did test locally only with my test cases and worked fine.
>
> If you have any issues with the two patches please let me know.
>
>
OK Lahiru, I'll go through that Today...

Thanks,
Mohan


> Thanks
> Lahiru
> On Jun 27, 2014, at 2:03 AM, Seshika Fernando wrote:
>
> Hi Lahiru,
>
> As Srinath has mentioned as well, frequency counting algorithms are very
> useful in financial market scenarios as well (especially fraud detection
> and surveillance).
> Thanks for doing this and I will take a look too.
>
> seshika
>
>
> On Fri, Jun 27, 2014 at 10:52 AM, Mohanadarshan Vivekanandalingam <
> mo...@wso2.com> wrote:
>
>>
>>
>>
>> On Fri, Jun 27, 2014 at 10:00 AM, Lahiru Gunathilake <
>> lginn...@indiana.edu> wrote:
>>
>>> Hi Mohan,
>>>
>>
>> Hi Lahiru,
>>
>>
>>>
>>> I wrote some samples but I can write more test-cases and provide another
>>> patch. Please feel free to change the naming of the windows as you like.
>>>
>>
>> Really appreciate your contribution.. Sure, I'll start look into this..
>>
>> Thanks,
>> Mohan
>>
>>
>>> Regards
>>> Lahiru
>>>
>>> On Jun 26, 2014, at 11:35 PM, Srinath Perera wrote:
>>>
>>> Hi All,
>>>
>>> Lahiru and myself had a call today morning.
>>>
>>> Plan is to
>>> 1) Lahiru to look at hoeffding tree and other classification algorithms
>>> and select one to implement. He will compare the performance against MOA or
>>> some other implementation.
>>> 2) then we will use it for a  Fraud analysis scenario as a proof of its
>>> validity.
>>>
>>> Then we will decide how to continue from that point.
>>>
>>> Mohan could you look at the patch? Lahiru will write test cases that you
>>> can use to verify.
>>>
>>> --Srinath
>>>
>>>
>>> On Tue, Jun 24, 2014 at 4:56 AM, Lahiru Gunathilake <
>>> lginn...@indiana.edu> wrote:
>>>
>>>> Hi Srinath,
>>>>
>>>> Thanks for the response.
>>>> On Jun 23, 2014, at 10:48 PM, Srinath Perera wrote:
>>>>
>>>> Hi Lahiru,
>>>>
>>>> Sorry for not responding earlier. I was traveling last week.
>>>>
>>>> I guess you know frequent item set !=  clustering algorithms.
>>>>
>>>> +1
>>>>
>>>>
>>>> Can we have a chat sometime to discuss details about the
>>>> implementation.
>>>>
>>>> Yes sure, that would be very useful.
>>>>
>>>> Thanks
>>>> Lahiru
>>>>
>>>>
>>>> --Srinath
>>>>
>>>>
>>>> On Sun, Jun 22, 2014 at 6:25 PM, Lahiru Gunathilake <
>>>> lginn...@indiana.edu> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> I have implemented another frequency counting algorithm[1] which is a
>>>>> classic algorithm for mining frequent items in a data stream. Basically
>>>>> users can specify a minimum average value of frequent items with an error
>>>>> value.
>>>>>
>>>>> This algorithm will accept two user-specified parameters: a support
>>>>> threshold s [0-1] and an error parameter e [0-1] such that e << s and
>>>>> recommended number for e is normally s/10 or s/20. Let N denote the
>>>>> current length
>>>>> of the stream, i.e., the number of tuples seen so far. At any point
>>>>> of time, this algorithm can be asked to produce a list of events
>>>>> along with their estimated frequencies. The answers produced by this
>>>>> algorithm will have the
>>>>> following guarantees:
>>>>> 1. All item(set)s wh

Re: [Architecture] Using Siddhi Event processor to implement/evaluate some clustering algorithms

[Mohanadarshan Vivekanandalingam]

Hi Lahiru,

I have gone through above patches and play around with it little bit..
CEP873.patch seems fine and i have done some basic testing  But i have got
some issues when applying CEP877.patch. Even-though i am able to get rid
off that after making some changes, I think it is better if you send proper
patch for that.

Will commit these improvements once i got the patch for CEP877.

Thanks,
Mohan



On Mon, Jun 30, 2014 at 10:27 AM, Mohanadarshan Vivekanandalingam <
mo...@wso2.com> wrote:

>
>
> On Sat, Jun 28, 2014 at 2:22 PM, Lahiru Gunathilake 
> wrote:
>
>> Hi Mohan,
>>
>> I have attached the latest patch for cep-877(I removed the old patch from
>> the jira). But the very first patch I have attached in CEP-873 is required
>> by this patch.
>>
>> I made few improvements to both the algorithms where you can give the
>> attributes you want to count. Initial version I did was to count distinct
>> tuples but practically I think counting distinct attributes is going to be
>> useful. If user doesn't give any attribute I simply count the distinct
>> tuple with all the attributes.
>>
>> I have added two test cases but I can see in the build cluster test cases
>> are removed, I did test locally only with my test cases and worked fine.
>>
>> If you have any issues with the two patches please let me know.
>>
>>
> OK Lahiru, I'll go through that Today...
>
> Thanks,
> Mohan
>
>
>> Thanks
>> Lahiru
>> On Jun 27, 2014, at 2:03 AM, Seshika Fernando wrote:
>>
>> Hi Lahiru,
>>
>> As Srinath has mentioned as well, frequency counting algorithms are very
>> useful in financial market scenarios as well (especially fraud detection
>> and surveillance).
>> Thanks for doing this and I will take a look too.
>>
>> seshika
>>
>>
>> On Fri, Jun 27, 2014 at 10:52 AM, Mohanadarshan Vivekanandalingam <
>> mo...@wso2.com> wrote:
>>
>>>
>>>
>>>
>>> On Fri, Jun 27, 2014 at 10:00 AM, Lahiru Gunathilake <
>>> lginn...@indiana.edu> wrote:
>>>
>>>> Hi Mohan,
>>>>
>>>
>>> Hi Lahiru,
>>>
>>>
>>>>
>>>> I wrote some samples but I can write more test-cases and provide
>>>> another patch. Please feel free to change the naming of the windows as you
>>>> like.
>>>>
>>>
>>> Really appreciate your contribution.. Sure, I'll start look into this..
>>>
>>> Thanks,
>>> Mohan
>>>
>>>
>>>> Regards
>>>> Lahiru
>>>>
>>>> On Jun 26, 2014, at 11:35 PM, Srinath Perera wrote:
>>>>
>>>> Hi All,
>>>>
>>>> Lahiru and myself had a call today morning.
>>>>
>>>> Plan is to
>>>> 1) Lahiru to look at hoeffding tree and other classification algorithms
>>>> and select one to implement. He will compare the performance against MOA or
>>>> some other implementation.
>>>> 2) then we will use it for a  Fraud analysis scenario as a proof of its
>>>> validity.
>>>>
>>>> Then we will decide how to continue from that point.
>>>>
>>>> Mohan could you look at the patch? Lahiru will write test cases that
>>>> you can use to verify.
>>>>
>>>> --Srinath
>>>>
>>>>
>>>> On Tue, Jun 24, 2014 at 4:56 AM, Lahiru Gunathilake <
>>>> lginn...@indiana.edu> wrote:
>>>>
>>>>> Hi Srinath,
>>>>>
>>>>> Thanks for the response.
>>>>> On Jun 23, 2014, at 10:48 PM, Srinath Perera wrote:
>>>>>
>>>>> Hi Lahiru,
>>>>>
>>>>> Sorry for not responding earlier. I was traveling last week.
>>>>>
>>>>> I guess you know frequent item set !=  clustering algorithms.
>>>>>
>>>>> +1
>>>>>
>>>>>
>>>>> Can we have a chat sometime to discuss details about the
>>>>> implementation.
>>>>>
>>>>> Yes sure, that would be very useful.
>>>>>
>>>>> Thanks
>>>>> Lahiru
>>>>>
>>>>>
>>>>> --Srinath
>>>>>
>>>>>
>>>>> On Sun, Jun 22, 2014 at 6:25 PM, Lahiru Gunathilake <
>>>>> lginn...@indiana.edu> wrote:
>>>>>
>>>>>> Hi All,
>>>>>&g

[Architecture] Java 1.7 support for github repo

[Mohanadarshan Vivekanandalingam]

Hi All,

Are we planning to do $subject.. ?
In CEP, we have developed an adptor which supports for websocket. Here we
are using latest jetty version which only supports Java 1.7 according to
[1]..  But since we have configured jenkins only with java 1.6, this
components is getting failed. Is it possible to configure Java 1.7 for CEP
related repo or do we have any other option on this ?

[1]
http://www.eclipse.org/jetty/documentation/current/what-jetty-version.html#d0e75


Thanks,
Mohan

-- 
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

[Architecture] [Dev] WSO2 Complex Event Processor 4.0.0-M1 Released

[Mohanadarshan Vivekanandalingam]

*WSO2 Complex Event Processor 4.0.0-M1 Released!*


WSO2 CEP Team is pleased to announce the release of WSO2 Complex Event
Processor (Version - 4.0.0 Milestone 1 )

The Milestone-1 pack is available at [1]

Please follow the documentation for more info (which is working in
progress)  in [2]

Following are the bug fixes, improvements and the new features.

Bug

   - [CEP-521 ] - Input Event Adaptor
   creation UI should disable the field 'Durable subscriber name' when durable
   subscription is disabled
   - [CEP-734 ] -
   
"org.wso2.carbon.event.output.adaptor.core.exception.OutputEventAdaptorEventProcessingException:
   not-available" exception seen on "Test Connection" for email output adapters
   - [CEP-793 ] - event builder help
   page on ui is on event streams
   - [CEP-851 ] - Bool and Boolean
   issues when creating event stream from event processor
   - [CEP-859 ] - NPE thrown when
   wrong configs are given to event tables
   - [CEP-864 ] - Siddhi code
   contains unnecessary system.out statements.. Needs to remove them
   - [CEP-865 ] - IsMatch not
   properly working for regex when using some meta Characters
   - [CEP-866 ] - Event Table is not
   properly passing the SQL predicates
   - [CEP-867 ] - pizza-shop
   pizzaPublisherClient cant change the port
   - [CEP-869 ] - JSON builder
   mapping fails when the mapping order is not same as the event stream
   - [CEP-871 ] - Siddhi output null
   events when using having clause


Improvement

   - [CEP-872 ] - Implementing
   different stream clustering logics to siddhi event processor
   - [CEP-889 ] - Blooms Filter for
   siddhi event tables


New Feature

   - [CEP-852 ] - Event Simulator
   with sending multiple events using uploaded files
   - [CEP-879 ] - MQTT input Event
   Adaptor for CEP
   - [CEP-880 ] - MQTT output Event
   Adaptor for CEP
   - [CEP-885 ] - Time Series
   Regression Extension to Siddhi
   - [CEP-886 ] - Time Series
   Forecaster for Siddhi
   - [CEP-887 ] - Outlier Detection
   Extension for Siddhi
   - [CEP-888 ] - Initial storm
   solution for CEP


Patch

   - [CEP-850 ] - Problem with JMS
   output adapter on CEP 3.1.0 ( User name [null] or password is invalid.)
   - [CEP-883 ] - Fix for the issue
   of throwing exception when sending null field to a JSON, XML & Text event
   formatter.


Task

   - [CEP-695 ] - Provide more
   specific messages for adaptors which does not contain any test connection
   feature


Sub-task

   - [CEP-873 ] - Implementing simple
   counting algorithm to keep most frequent events in a window
   - [CEP-877 ] - Implementing
   LossyCountng algorithm



We welcome your feedback on this release of CEP.

[1] 
*http://svn.wso2.org/repos/wso2/people/mohan/CEP4.0.0-M1/wso2cep-4.0.0-M1.zip
*

[2]
https://docs.wso2.com/display/CEP400/WSO2+Complex+Event+Processor+Documentation


Thanks

*- WSO2 CEP Team -*

-- 
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Fwd: Kafka transport for WSO2 ESB

[Mohanadarshan Vivekanandalingam]

Hi Ishara,

I believe you have gone through the Kafka documentation. AFAIK, to consume
events from kafka we communicate with zookeeper and to publish events we
dealt with brokers. Here ESB connector does both consuming and publishing
right ? .Then kafka connector needs to connect with zookeeper to consume
events isn't it ? Then it should reflect in the diagram right. Please
correct me if i am wrong..

Thanks,
Mohan




On Sun, Jul 6, 2014 at 12:57 PM, Ishara Cooray  wrote:

>
> Sending to architecture group
>
> Ishara Cooray
> Senior Software Engineer
>
> +9477 262 9510
>
>
> -- Forwarded message --
> From: Ishara Cooray 
> Date: Sun, Jul 6, 2014 at 12:39 PM
> Subject: [Architecture] Kafka transport for WSO2 ESB
> To:
> Cc: Kasun Indrasiri , Srinath Perera ,
> Samisa Abeysinghe , Kathees Rajendram 
>
>
> Hi,
>
> we have come up with a high level architecture design for the kafka
> transport, which is attached here with for your kind review.
>
> Architecture - kafka transport for ESB
> 
>
>
> Your feedback is highly appreciated.
>
>
> Thanks & Regards,
> Ishara Cooray
> Senior Software Engineer
>
> +9477 262 9510
>
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [Dev] Java 1.7 support for github repo

[Mohanadarshan Vivekanandalingam]

Hi Kasun,

I believe you have some idea about our usecase. Can you please point any
references  which uses web socket feature of the Tomcat server (if any).
This will help Dilini to get some idea and start working on that..

Thanks,
Mohan



On Fri, Jul 4, 2014 at 10:05 PM, Sriskandarajah Suhothayan 
wrote:

> Thats good we'll verify that.
>
> Suho
>
>
> On Fri, Jul 4, 2014 at 5:19 PM, Kasun Gajasinghe  wrote:
>
>>
>> AFAIK, you do not need JDK 1.7 to compile websocket based webapps.
>>
>>
>> On Fri, Jul 4, 2014 at 4:21 PM, Sriskandarajah Suhothayan 
>> wrote:
>>
>>> At runtime that will help. But this is not possible when compiling and
>>> releasing the product :(
>>>
>>> Suho
>>>
>>>
>>> On Fri, Jul 4, 2014 at 4:17 PM, Kasun Gajasinghe 
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> How Tomcat Websocket works is that if the JDK version is less than 7,
>>>> it disables Websocket feature and the container starts up. It prints a
>>>> warning log pointing out that user is not running in JDK 7. You might want
>>>> to follow the same procedure.
>>>>
>>>> Regards,
>>>> KasunG
>>>>
>>>>
>>>> On Fri, Jul 4, 2014 at 3:32 PM, Sriskandarajah Suhothayan <
>>>> s...@wso2.com> wrote:
>>>>
>>>>> Yes, thats an issue.
>>>>> Because of this we wont be able to build the full platform in one java
>>>>> version.
>>>>>
>>>>> So what will be the recommended approach here ?
>>>>>
>>>>> Regards
>>>>> Suho
>>>>>
>>>>>
>>>>> On Fri, Jul 4, 2014 at 3:26 PM, Sagara Gunathunga 
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 4, 2014 at 3:07 PM, Sriskandarajah Suhothayan <
>>>>>> s...@wso2.com> wrote:
>>>>>>
>>>>>>> Our idea is to use Tomcat this is jetty is just a quick PoC we did
>>>>>>> for CEP. But even to use Tomcat Web sockets we have to have to build 
>>>>>>> using
>>>>>>> Java 7[1].
>>>>>>> So anyway we have to move to Java 7 to build the CEP repo so that it
>>>>>>> can compile the WebSocket servlet
>>>>>>>
>>>>>>> What is our plan on moving the build to Java 7?
>>>>>>>
>>>>>>
>>>>>> You can configure Jenkins job to run with Java 7 easily, I don't
>>>>>> remember whether we have installed Java7 on Jenkins but we can easily
>>>>>> install and configure CEP job to run with Java7.   BTW this means Java7
>>>>>> will be a mandatory to run CEP ?
>>>>>>
>>>>>>
>>>>>> Thanks !
>>>>>>
>>>>>>>
>>>>>>> Regards
>>>>>>> Suho
>>>>>>>
>>>>>>> [1]http://tomcat.apache.org/tomcat-7.0-doc/web-socket-howto.html
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jul 4, 2014 at 3:06 PM, Afkham Azeez  wrote:
>>>>>>>
>>>>>>>> +1. Don't unnecessarily bring in Jetty.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jul 4, 2014 at 2:49 PM, Sagara Gunathunga 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Jul 4, 2014 at 2:40 PM, Mohanadarshan Vivekanandalingam <
>>>>>>>>> mo...@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi All,
>>>>>>>>>>
>>>>>>>>>> Are we planning to do $subject.. ?
>>>>>>>>>> In CEP, we have developed an adptor which supports for websocket.
>>>>>>>>>> Here we are using latest jetty version which only supports Java 1.7
>>>>>>>>>> according to [1]..  But since we have configured jenkins only with 
>>>>>>>>>> java
>>>>>>>>>> 1.6, this components is getting failed. Is it possible to configure 
>>>>>>>>>> Java
&

Re: [Architecture] ESB Connector for BAM & CEP

[Mohanadarshan Vivekanandalingam]

Hi All,

We had a discussion sometime back (last year) on this.. Please check the
mail thread [1] for more info..

[1] [Architecture] Improving Data Publisher Mediator to replace BAM Mediator

Thanks,
Mohan


On Mon, Oct 20, 2014 at 12:55 PM, Anjana Fernando  wrote:

> Hi Sanjiva,
>
> On Sat, Oct 18, 2014 at 9:44 PM, Sanjiva Weerawarana 
> wrote:
>
>> On Fri, Oct 17, 2014 at 6:47 PM, Lahiru Chandima 
>> wrote:
>>
>>> Hi all,
>>>
>>> We (Vijitha and myself) are going to develop an ESB
>>> Connector which can publish events to BAM and CEP.
>>>
>>
>> We're currently using the words "ESB Connector" to mean something that
>> becomes a set of mediators to access (primarily) HTTP APIs. This is not
>> that right?? If so lets not use the same words please.
>>
>
> Yeah, I guess, this is simply a mediator, just like what we have now. It
> was just Kasun has mentioned to Suho, to create this as an ESB Connector,
> where we haven't actually gone into what it means to create a connector.
> @Kasun, maybe you can shed some light on this, on what you meant earlier.
>
>
>>
>> Currently, BAM mediator is used for sending events from ESB to BAM. BAM
>>> Mediator needs to be configured with server details and streams need to be
>>> defined prior to using them in sequences of ESB Synapse configuration.
>>>
>>> With new ESB connector, it will be possible to add all configurations
>>> needed to publish events to BAM or CEP in the ESB Synapse configuration
>>> itself. This brings the configuration to a single place unlike when using
>>> the BAM mediator.
>>>
>>
>> Is this a real requirement? Do we expect different ESB configs (running
>> on the same server) to want to send data to different BAM/CEP endpoints? I
>> understand the multitenancy usecase - is it only for that?
>>
>
> Actually, the idea here is, what we have now for defining the target for
> events, "BAM Profiles", to replace that. Basically in a BAM Profile, we put
> all the information such as the event receiver details, and the stream
> definition, and how the properties are mapped using properties from a
> sequence. We want to clean this up by, simply having something equivalent
> to a message store we have now, where we named is "data sink" for now, to
> be the target for events, and the actual stream mapping will be done by the
> mediator itself, in the sequence. And the motivation for having this as an
> artifact in the synapse configuration is, where from a place like DevStudio
> or any development environment, we can create these deployment artifacts
> can directly deploy them, without going to the admin console and adding
> them through the UI.
>
>
>>
>> *High level architecture - components.*
>>>
>>> 1) New Synapse construct will be introduced to configure details related
>>> to the data sink (ie. CEP or BAM).
>>>
>>> Eg.
>>>
>>> 
>>> tcp://localhost:7611
>>> admin
>>> admin
>>> 
>>>
>>
>> What do we call this now? The term "dataSink" doesn't work for me but
>> that could be just me.
>>
>
> Yeah, we had a meeting sometime back to discuss this, and that day also,
> we couldn't decide on the name :) .. basically something related to a
> dataSink, any suggestions? :) ..
>
>
>>
>> 2) An ESB connector will be developed which can be used to publish events
>>> to a data sink. Connector will use data-bridge API (Thrift) for publishing
>>> events (It will be possible to introduce new transports other than Thrift
>>> later).
>>>
>>
>> Once again, what are the operations? Why do we want to write a connector
>> instead of using / improving the current "BAM mediator"?
>>
>
> So yeah, it will basically be improving/re-writing the current mediator,
> with a name like "dataPublish" or something, without any "BAM" part, and
> with the stream property mapping part added there itself, rather than
> having it in another place.
>
>
>>
>> Thrift stream related data (stream name, version and stream properties)
>>> will be configured in the ESB connector configuration (in Synapse config),
>>> which is more logical compared to configuring it separately as done in the
>>> BAM mediator.
>>>
>>
>> Can you give a specific example to illustrate the difference please?
>>
>
> So basically in the current BAM mediator, we do not give the direct
> mapping to the stream properties there, we simply populate some message
> context properties and call the mediator. And the mediator simply looks up
> the information provided by the BAM Profile to do the actual mapping, where
> in our profile, we have given like Xpath expressions to say, what are the
> values in a specific target stream event.
>
> Cheers,
> Anjana.
>
>
>>
>> Thanks,
>>
>> Sanjiva.
>> --
>> Sanjiva Weerawarana, Ph.D.
>> Founder, Chairman & CEO; WSO2, Inc.;  http://wso2.com/
>> email: sanj...@wso2.com; office: (+1 650 745 4499 | +94  11 214 5345)
>> x5700; cell: +94 77 787 6880 | +1 408 466 5099; voip: +1 650 265 8311
>> blog: http://sanjiva.weerawarana.org/; twitter: @sanjiva
>> Lean . Enterprise . Middleware
>>
>> ___

Re: [Architecture] Folder structure of Analytics Toolbox (BAM & CEP)

[Mohanadarshan Vivekanandalingam]

On Thu, Oct 30, 2014 at 8:30 PM, Sriskandarajah Suhothayan 
wrote:

> Hi
>
> The current folder structure[1] does not extend clearly for CEP
> integration and hence we have to decide on the new folder structure of the
> Analytics Toolbox. shall we have
>
> inputs
> - adaptors
> - builders
> outputs
> - adaptors
> - formatters
> streams
> uis
> - gadgets
> - jaggery-apps
> reports
> - jesper
> analytics
> - executionplans
> - hive-scrips
>
> Thoughts?
>

+1.. Above structure covers all the artifacts which are available at the
moment. I think analyzers.properties file is missed in above, what is the
best place for that (anaytics or new folder called scheduler) ??

Thanks,
Mohan

>
> Regards
> Suho
>
> [1]https://docs.wso2.com/display/BAM200/Creating+a+Custom+Toolbox
>
> --
>
> *S. Suhothayan*
> Technical Lead & Team Lead of WSO2 Complex Event Processor
>  *WSO2 Inc. *http://wso2.com
> * *
> lean . enterprise . middleware
>
>
> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog:
> http://suhothayan.blogspot.com/ twitter:
> http://twitter.com/suhothayan  | linked-in:
> http://lk.linkedin.com/in/suhothayan *
>



-- 
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Nominal attributes in Siddhi

[Mohanadarshan Vivekanandalingam]

On Fri, Feb 27, 2015 at 7:26 AM, Srinath Perera  wrote:

> Mohan, could you check when you can?
>

Sure, will do..

Thanks,
Mohan


>
> On Fri, Feb 27, 2015 at 6:46 AM, Ginnaliya Gamathige, Lahiru Manananda <
> lginn...@indiana.edu> wrote:
>
>>  Hi Srinath,
>>
>>  I have wrapped up the implementation and sent a pull request. Please
>> let me know if there are any issues with merging the pull request.
>>
>>  I have attached the draft version of the report I wrote based on the
>> current implementation, this contains more improvements we should do to
>> support fully for the Hoeffding algorithm.
>>
>>  As per our discussion happened in Bloomington, I had a look in to
>> regression analysis for data streams and there are lots of interesting
>> things we can do with Siddhi to do multi-dimensional regressions.
>> Mostly people tried implementing one parse algorithms  to find patterns
>> with sensor data and dynamically modify the model. I hope to send a detail
>> email on this.
>>
>> Thanks
>> Lahiru
>>
>>
>>  On Sep 23, 2014, at 3:39 AM, Ginnaliya Gamathige, Lahiru Manananda <
>> lginn...@indiana.edu> wrote:
>>
>>  Hi Suho,
>>
>>  I have modified Attribute class which will keep all the possible values
>> in attribute and can create the numeric value 2 for “rainy”. In the
>> incoming event we can have the possible values for each attribute. I have
>> the code committed to my local github[1] and once I have a working copy
>> with some good tests I can send a pull request.
>>
>>  Once we have the tree build for these events, Siddhi should be able to
>> take a given event and do a prediction and determine the value of the last
>> attribute and based on the last attribute values we
>> can move event in to different event streams.
>> Ex: We can send 10 events with proper value for play (yes,no) after you
>> have the tree built you can just send the rest of the attributes in the
>> event and Decision tree will predict the value for play (yes or no), so the
>> query can be written to the play event. In the Window implementation we can
>> specify minimum events required to build the tree and always keep the last
>> attribute as the class attribute (During the initial implementation).
>>
>>  So query will be, from
>> weather#siddhiht(minimum_events_for_tree_build,give_desired_class_values)
>>   select * insert into play for all-events;
>>
>>  For a simple scenario we can assume there is only a single class we
>> need to check in this case we can say play=“yes”, so query can look like
>> below for 10 minimum events.
>>
>>  weather#siddhiht(10,play="yes")
>>   select * insert into play for all-events;
>>
>>  If events come empty play value before the tree is built we can throw
>> an exception or just emit the decision but it will be with low accuracy.
>>
>>  Regards
>> Lahiru
>>
>>
>>  [1]https://github.com/glahiru/siddhi
>>  On Sep 23, 2014, at 1:56 AM, Sriskandarajah Suhothayan 
>> wrote:
>>
>>  Looks OK.
>>
>>  Can you please clarify how would you write queries in this case? (e.g
>> If its "rainy" how would you check?)
>> and internally how all of these will be processed?
>>
>>  Regards
>> Suho
>>
>>
>> On Tue, Sep 23, 2014 at 10:30 AM, Srinath Perera 
>> wrote:
>>
>> I think this is OK. Suho, WDYT?
>>
>>  On Mon, Sep 22, 2014 at 12:15 AM, Ginnaliya Gamathige, Lahiru Manananda
>>  wrote:
>>
>>  Hi Devs,
>>
>>  I am developing a classification algorithm for siddhi and I have a
>> requirement of defining attributes as a nominal attribute.
>>
>>  Nominal attribute is similar to a string attribute but during the
>> definition we have to give the possible values for that attribute.
>> One of the value from that set can come as the actual value in a
>> particular event. I can see in the code Siddhi doesn’t support this (Please
>> correct me if I am wrong), so
>> I am thinking of changing the Siddhi language grammar to parse nominal
>> attribute as below.
>>
>>  Example query: define stream weather(outlook nominal{sunny, overcast,
>> rainy}, temperature nominal{hot, mild, cool}, humidity nominal{high,
>> normal},windy nominal{TRUE, FALSE},play nominal{yes, no}).
>>
>>  If there is a better way to do this, please provide me some feedback.
>>
>>  During the classification algorithm it will use numeric values which
>> maps to these possible values(Ex: sunny=0,overcase=1,rainy=2) and do the
>> calculation using numeric values. Initial phase of the algorithm
>> I am implementing[1], I hope to process nominal values (which is easy to
>> implement), then algorithm should support processing numeric data too.
>>
>>  If you have any feedback please let me know.
>>
>>  [1]http://homes.cs.washington.edu/~pedrod/papers/kdd00.pdf
>>
>>  Regards
>>  Lahiru
>>
>>  ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>>
>>
>>  --
>> 
>> Srinath Perera, Ph.D.
>>http://people.apache.org/~hemapani/

Re: [Architecture] ESB Feedback: Improve Payload Factory and Improving Enrich

[Mohanadarshan Vivekanandalingam]

On Fri, May 8, 2015 at 8:48 AM, Srinath Perera  wrote:

> https://wso2.org/jira/browse/ML-16
>

@Srinath, I think we need to move this jira to ESB Project right.


> On Wed, May 6, 2015 at 2:34 PM, Srinath Perera  wrote:
>
>> Thanks Maheeka.
>>
>> Kasun +1, but remove other attributes from docs at the point. Shall we
>> create a Jira?
>>
>> On Wed, May 6, 2015 at 12:34 AM, Kasun Indrasiri  wrote:
>>
>>> Hi Srinath,
>>>
>>> Regarding enrich mediator, I think we can get rid of most of the
>>> redundant attributes and support them through xpath only. For instance most
>>> of the operations that you do with enrich can be done using xpath attribute
>>> in both source and target. So, we will fully verify and fix these use cases
>>> while keeping support for existing syntax for backward compatibility.
>>>
>>> 
>>> 
>>> >> xmlns:m0="http://services.samples"/>
>>> 
>>> 
>>>
>>>
>>> 
>>> 
>>> >> xmlns:m0="http://services.samples"/>
>>> 
>>> 
>>>
>>>
>>>
>>> On Wed, May 6, 2015 at 9:46 AM, Srinath Perera  wrote:
>>>
 Hi Kasun,

 I have been working on ESB script last week, and two comments.

 1. If we use payload factory mediator, and if Xpath matches more than
 one results, what do we do? It is a valid usecase if someone want to copy N
 elements from request to response via payload factory.

 2. Enrich has too many parameters, and some combinations does not work.
 ( see below).

 
 >>> [type=custom|envelope|body|property|inline] xpath="" property="" />
 >>> [type=custom|envelope|body|property|inline] xpath="" property="" />
 

 I think we can remove "type" and "property" and support both via Xpath
 only. We can break Xpath to /foo/bar where source can be $envelope,
 $body $header $getProperty("bar") etc ( $envelope etc is already there).

 If we do this correct, we can refer to anything in ESB environment via
 Xpath of the format /foo/bar. Then we can make this consistent
 across the language.

 WDYT?

 --Srinath

 --
 
 Blog: http://srinathsview.blogspot.com twitter:@srinath_perera
 Site: http://people.apache.org/~hemapani/
 Photos: http://www.flickr.com/photos/hemapani/
 Phone: 0772360902

>>>
>>>
>>>
>>> --
>>> Kasun Indrasiri
>>> Software Architect
>>> WSO2, Inc.; http://wso2.com
>>> lean.enterprise.middleware
>>>
>>> cell: +94 77 556 5206
>>> Blog : http://kasunpanorama.blogspot.com/
>>>
>>
>>
>>
>> --
>> 
>> Srinath Perera, Ph.D.
>>http://people.apache.org/~hemapani/
>>http://srinathsview.blogspot.com/
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> 
> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera
> Site: http://people.apache.org/~hemapani/
> Photos: http://www.flickr.com/photos/hemapani/
> Phone: 0772360902
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

[Architecture] [DEV] WSO2 Complex Event Processor 4.1.0 Alpha Released

[Mohanadarshan Vivekanandalingam]

Hi All,

WSO2 CEP Team is pleased to announce the Alpha release of WSO2 Complex
Event Processor 4.1.0.

WSO2 Complex Event Processor (CEP) can be used to identify the most
meaningful events within an event cloud, analyzes their impacts, and acts
on them in real time. Built to be extremely high performing it offers
significant time saving and affordable acquisition. WSO2 CEP Manager is
released under the Apache Software License 2.0.

The distribution is available at [1]. The documentation for CEP 4.1.0 Alpha
can be found at [2] (which is WIP). Your feedback is most welcome, and any
issues can be reported to the project at [3].

Following bug fixes, improvements and new features have been added along
with this release .

*Improvements*

   - [CEP-644 ] - add insertOrUpdate
   to event table
   - [CEP-1419 ] - Provide "insert
   or update" functionality to event tables
   - [CEP-1426 ] - When passing
   parameters to function allow the syntax func(*) if all the attributes of
   the inputstream have to be passed
   - [CEP-1439 ] - Map extension
   support
   - [CEP-1445 ] - Improve
   user-friendliness by sorting drop-down lists
   - [CEP-1448 ] - Hazelcast Event
   Table for Siddhi
   - [CEP-1450 ] - Adding Arbitrary
   Map support for Receiver Components
   - [CEP-1453 ] - Pause, Resume and
   Stop support for Event Simulator
   - [CEP-1458 ] - Support for more
   then 2 nodes in HA deployment


*New Features*

   - [CEP-101 ] - Indexing for
   Hazelcast table
   - [CEP-105 ] - Time length window
   implementation for Siddhi
   - [CEP-1365 ] - Add a function to
   extract 'DAY OF WEEK' from time/timestamp
   - [CEP-1423 ] - Support
   outer/left joins in Siddhi
   - [CEP-1440 ] - Geo Closest
   function
   - [CEP-1441 ] - External Batch
   window for siddhi
   - [CEP-1457 ] - Minima and Maxima
   detection on Siddhi
   - [CEP-1459 ] - Instrumentation &
   Monitoring support for CEP and Siddhi


*Bugs*

   - [CEP-1117 ] - No default value
   support for Map mapping
   - [CEP-1338 ] - [Intermittent]
   Error observed in edit tenant UI
   - [CEP-1352 ] - 'The [action]
   cannot be processed at the receiver' error when clicked on 'Streams' option
   - [CEP-1361 ] - Index Out of
   Bounds Exception after a new field is added to Stream
   - [CEP-1363 ] - Execution plan
   does not display "from" when we type @ symbol as a result this needs to be
   added
   - [CEP-1375 ] -
   org.wso2.carbon.event.processor.admin bundle activation is getting delayed
   because it get wires with wrong dependency version
   - [CEP-1393 ] - NPE thrown for
   Websocket Local Input Receiver if the same receiver is redeployed
   - [CEP-1400 ] - Error when
   integrating ML extension to CEP storm cluster
   - [CEP-1401 ] - Issue in email
   event adapter
   - [CEP-1405 ] - "No extension
   exist for StreamFunctionExtension{namespace='ml'}" exception when started
   worker node with -DworkerNode=true property
   - [CEP-1407 ] - In conditions
   cannot be used in select statement
   - [CEP-1408 ] - Concurrent
   Modification exception
   - [CEP-1409 ] - When modifying an
   event stream, information does not get removed from in-memory cache at
   databridge
   - [CEP-1410 ] - Validate Input
   Stream for WSO2Event custom mapping
   - [CEP-1411 ] - Siddhi thread
   dead on invalid inputs
   - [CEP-1415 ] - Event Table
   Update results in duplicate inserts
   - [CEP-1418 ] - Test cases are
   failing in SnapshotOutputRateLimitTestCase due to possible thread
   contention issue
   - [CEP-1422 ] -
   ArrayIndexOutOfBoundsException when adding wso2-event publisher with
   invalid Receiver URL
   - [CEP-1427 

[Architecture] [DEV] WSO2 Complex Event Processor 4.1.0 Beta Released

[Mohanadarshan Vivekanandalingam]

Hi All,

WSO2 CEP Team is pleased to announce the Beta release of WSO2 Complex Event
Processor 4.1.0.

WSO2 Complex Event Processor (CEP) can be used to identify the most
meaningful events within an event cloud, analyzes their impacts, and acts
on them in real time. Built to be extremely high performing it offers
significant time saving and affordable acquisition. WSO2 CEP Manager is
released under the Apache Software License 2.0.

The distribution is available at [1]. The documentation for CEP 4.1.0 Beta
can be found at [2] (which is WIP). Your feedback is most welcome, and any
issues can be reported to the project at [3].

Following bug fixes and improvements have been added along with this
release .

*Bug*

   - [CEP-1347 ] - Refreshing Arc
   and Map charts is visible when throughput is high
   - [CEP-1367 ] - Cannot add a
   datasource, with name having character 's'
   - [CEP-1383 ] - Intermittently an
   NPE is thrown on worker logs when updating execution planner with Output
   Rate Limiting patterns
   - [CEP-1387 ] - Analytics
   Dashboard - Multiline Y axis Limitations
   - [CEP-1388 ] - "Error while
   indexing" issue observed in worker node when starting a fresh pack in CEP
   distributed cluster
   - [CEP-1389 ] - Analytics
   Dashboard - Line and Area charts miss first event when plotting
   - [CEP-1391 ] - OOM when nodes
   are disconnected from hazelcast cluster in distributed mode
   - [CEP-1392 ] - Restrict setting
   up incorrect configurations in event simulator configuration files for
   sample mode
   - [CEP-1395 ] - Analytics
   Dashboard - Login redirection does not work in Sample mode
   - [CEP-1431 ] - Cannot put
   "((true) in SomeEventTable)" as an attribute in the select statement
   - [CEP-1438 ] - Siddhi PMML
   extension is always expecting an output field
   - [CEP-1460 ] - MQTT Event
   receiver unsubscribes from the topic on shutdown, even cleanSession is set
   to false.
   - [CEP-1461 ] - Issue on viewing
   system logs in management console
   - [CEP-1466 ] - two get calls
   executes when datasources getting populated for Batch and realtime


*Improvement*

   - [CEP-1406 ] - Need to have tool
   tips for icons in the Dashboard
   - [CEP-1467 ] - When deleting a
   gadget need a configurable option to either delete whole layout with block
   or only the block(gadget content))



References:

​[1] https://github.com/wso2/product-cep/releases/tag/v4.1.0-beta
[2]
https://docs.wso2.com/display/CEP410/WSO2+Complex+Event+Processor+Documentation
[3] https://wso2.org/jira/browse/CEP


Thanks,
-WSO2 CEP Team-

-- 
*V. Mohanadarshan*
*Senior Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Extending execution manager to support Batch analytics scripts

[Mohanadarshan Vivekanandalingam]

I think we need to consider few things here to find a better approach..

- Since we are planning to use same Execution manager for CEP then we need
to maintain clear separation between realtime and batch part of it.. That
means, we need two different components for this..

- We have an single UI component (Jaggary UI) for both Realtime and batch
configuration..  Then have to decide whether to have different admin and
core components..

- IMO,  for the time being let's implement as separate components (since we
already have realtime components) and decide on whether to merge or not
later based on the outcome..

Does this make sense ?


Thanks,
Mohan


On Fri, Feb 26, 2016 at 2:31 PM, Sriskandarajah Suhothayan 
wrote:

> We will implement the batch & realtime part of the execution manager as
> separate backend components.
> And move the common aspects in the execution.manager.core.
>
> WDYT?
>
> Regards
> Suho
>
> On Fri, Feb 26, 2016 at 2:25 PM, Rajeev Sampath  wrote:
>
>> Hi,
>>
>> Currently the execution manager supports configuring templates for CEP
>> execution plans. We now have the requirement of supporting a similar
>> template configuration functionality for Spark SQL batch analytics scripts
>> as well.
>>
>> I have checked the possibility of extending our current execution manager
>> to support batch scripts and following are some concerns.
>>
>> - Since execution plans and Spark scripts have different configurations
>> and handled in different mannger by components, at the core, there will be
>> separate implementations for handling these two.
>> - But at admin service level, we can still expose these in a uniform
>> manner in one service.
>> - Maintainability issues that will arise when having these in one
>> component.
>> - From a user perspective these are quite similar, since it will be
>> template configuration support for both and probably makes it easier to
>> treat them uniformly for API analytics use cases etc.
>>
>> Hence, do we need to implment the batch script template feature
>> separately or just extend the existing execution manager?
>> Pls share your ideas on this.
>>
>>
>> Thanks
>> Rajeev
>>
>> --
>> Rajeev Sampath
>> Senior Software Engineer
>> WSO2, Inc.; http://www.wso2.com.
>>
>> Mobile:
>> * +94716265766 <%2B94716265766>*
>>
>
>
>
> --
>
> *S. Suhothayan*
> Technical Lead & Team Lead of WSO2 Complex Event Processor
> *WSO2 Inc. *http://wso2.com
> * *
> lean . enterprise . middleware
>
>
> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog:
> http://suhothayan.blogspot.com/ twitter:
> http://twitter.com/suhothayan  | linked-in:
> http://lk.linkedin.com/in/suhothayan *
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*V. Mohanadarshan*
*Senior Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] CEP does not accept requests after receiving a high rate of events for some time

[Mohanadarshan Vivekanandalingam]

>
> Hi,
>
>
Hi Isuru,

Please find my comments below..



> This is regarding the analytics for US Election 2016 Tweets. The ESB uses
> Twitter Connector to find tweets with some specific hashtags and sends the
> tweets as events to CEP via HTTP. The CEP version is 4.0.0.
>
> The CEP receives the events via the Tomcat HTTP Connector (port 9763). As
> mentioned in $subject, the CEP fails to accept requests as there are no
> worker threads to handle the requests.
>
> I have attached a thread dump and I analyzed it using the ThreadLogic
> application [1]. All 250 HTTP worker threads (http-nio-9763-exec-*) are in
> "PARKING" state and following is a part of stack trace in each thread.
>
> "http-nio-9763-exec-102" #810 daemon prio=5 os_prio=0
> tid=0x7f7678010800 nid=0x710 waiting on condition [0x7f75da48f000]
>*java.lang.Thread.State: WAITING (parking)*
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0005cc6bd6a8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350)
>
>
>
>
> *at
> org.wso2.carbon.event.input.adapter.http.HTTPEventAdapter$1.rejectedExecution(HTTPEventAdapter.java:99)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
> at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
> at
> org.wso2.carbon.event.input.adapter.http.HTTPMessageServlet.doPost(HTTPMessageServlet.java:177)*
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
> at
> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
>
>
> The HTTPMessageServlet submits a HTTPRequestProcessor [2] to an executor
> service and the executor service rejects it as the worker queue is full.
> However in the RejectedExecutionHandler, the task is put back to same
> queue [3]. Here the thread gets parked while waiting for some condition.
> This is why the Tomcat HTTP connector can no longer process any requests.
>
>
Yes, above implementation is done purposefully.. We had a requirement where
we need to block the HTTP requests when there is no thread (or space in the
queue) rather dropping the events. That is why above HTTP adapter is
implemented in such way.. Because of above implementation, there will not
be an event loss at receiver level.


> Then I checked the thread pool used in [2] and found out that all 100
> threads (pool-75-thread-*) in that pool are also in "PARKING" state.
> Following is the stack trace.
>
>
> "pool-75-thread-100" #758 prio=5 os_prio=0 tid=0x7f7634017800
> nid=0x6ba runnable [0x7f75dd8c4000]
> *   java.lang.Thread.State: TIMED_WAITING (parking)*
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
> at *com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246)*
> at
> *org.wso2.siddhi.core.stream.input.SingleStreamEntryValve.send(SingleStreamEntryValve.java:74)*
> at
> org.wso2.siddhi.core.stream.input.SingleStreamEntryValve.send(SingleStreamEntryValve.java:99)
> at
> org.wso2.siddhi.core.stream.input.InputHandler.send(InputHandler.java:49)
> at
> org.wso2.carbon.event.processor.core.internal.listener.SiddhiInputEventDispatcher.sendEvent(SiddhiInputEventDispatcher.java:39)
> at
> org.wso2.carbon.event.processor.core.internal.listener.AbstractSiddhiInputEventDispatcher.consumeEvent(AbstractSiddhiInputEventDispatcher.java:92)
> at
> org.wso2.carbon.event.stream.core.internal.EventJunction.sendEvent(EventJunction.java:142)
> at
> org.wso2.carbon.event.receiver.core.internal.management.InputEventDispatcher.onEvent(InputEventDispatcher.java:27)
> at
> org.wso2.carbon.event.receiver.core.internal.EventReceiver.sendEvent(EventReceiver.java:256)
> at
> org.wso2.carbon.event.receiver.core.internal.EventReceiver.processMappedEvent(EventReceiver.java:200)
> at
> org.wso2.carbon.event.receiver.core.internal.EventReceiver$MappedEventSubscription.onEvent(EventReceiver.java:307)
> at
> org.wso2.carbon.event.input.adapter.core.internal.InputAdapterRuntime.onEvent(InputAdapterRuntime.java:109)
> at
> org.wso2.carbon.event.input.adapter.http.HTTPMessageServlet$HTTPRequestProcessor.run(HTTPMessageServlet.java:210)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>  

Re: [Architecture] CEP 3.0 towards a Complete eventing solution to the platform

[Mohanadarshan Vivekanandalingam]

Hi All,

We have completed the development of following components...

   - Transport Adaptor
   - Transport Adaptor Manager

Now, we are further working on other components and doing improvements for
them...

Thanks,
Mohan


On Wed, May 8, 2013 at 7:59 PM, Sriskandarajah Suhothayan wrote:

> I have attached the comportment diagram for the following
>
>- Transport Adaptor
>- Transport Adaptor Manager
>- Event Builder
>- Event Processor
>
> Regards,
> Suho
>
>
> On Thu, Mar 14, 2013 at 10:50 PM, Sriskandarajah Suhothayan  > wrote:
>
>> After the discussion with Srinath we came up with the following CEP
>> architecture.
>> Here we have five components for CEP
>>
>>- Input Transport Adaptor - to receive incoming (XML, JSON, etc )
>>messages via (JMS, Thrift, etc)
>>- Event Builder - to convert the (XML, JSON, etc ) messages into
>>Siddhi/Common Events
>>- Event Processor
>>- Event Formatter - to convert  Siddhi/Common Events back to (XML,
>>JSON, etc ) messages
>>- Output Transport Adaptor - to publish output (XML, JSON, etc )
>>messages via (JMS, Thrift, etc)
>>
>> With this model any WSO2 server (E.g BAM) can use the Input Transport
>> Adaptor & Event Builder to receive incoming events
>> and use Output Transport Adaptor & Event Formatter to notify outputs E.g
>> Email,
>> At the same time CEP can also be embedded in to other products (E.g ESB,
>> ELB, SS) & invoked using  OSGi services buy installing Event Processor (and
>> if necessary Event Builder & Event Formatter)
>>
>> This model also enables events to be passed from one QueryPlan (Bucket)
>> to another within CEP, which was not possible in CEP 2.x.
>>
>> Thoughts?
>>
>> Thanks
>> Suho
>>
>>
>>
>> On Fri, Jan 11, 2013 at 3:58 PM, Amila Suriarachchi wrote:
>>
>>> looks good. +1 for terminology.
>>>
>>> thanks,
>>> Amila.
>>>
>>> On Thu, Jan 10, 2013 at 12:13 PM, Sriskandarajah Suhothayan <
>>> s...@wso2.com> wrote:
>>>
 With CEP 2.0 we made Siddhi stable and working for most common
 use-cases.

 With CEP 3.0 we need to make CEP work seamlessly with other systems and
 also provide solutions to all eventing needs of the platform.
 This is because we have various throttling and notification
 implementations in the current platform, and with this solution we can have
 a unified throttling engine using Siddhi, and a unified notification model
 provided by CEP so that whoever needs E-mail,SMS,etc notifications can
 simply use that.

 I propose we should go for an architecture as described in the
 attachment.

 With this, other products (Stratos,BAM,AM,AF) can easily use modules
 like Notification, and Query&EventProcessor, for notification and Embedded
 Event processing (throttling) needs.

 In addition as discussed at the mail "Bringing CEP input mapping
 configuration into Siddhi" we will make input and output mappings as Siddhi
 based configuration instead of the current XML format, hence the Bucket
 configuration file will not be an XML but be a Script file containing
 Siddhi.
 Though we move input,output mapping to Siddhi I still believe we need
 to have UI for both input and output mappings as it will be more intuitive
 for the users and as it also improves user friendliness.

 Thoughts?

 Suho



 --
 *S. Suhothayan
 *
 Software Engineer,
 Data Technologies Team,
  *WSO2, Inc. **http://wso2.com
  *
 *lean.enterprise.middleware.*

 *email: **s...@wso2.com* * cell: (+94) 779 756 757
 blog: **http://suhothayan.blogspot.com/*
 *
 twitter: **http://twitter.com/suhothayan*
 *
 linked-in: **http://lk.linkedin.com/in/suhothayan*
 *
 *

 ___
 Architecture mailing list
 Architecture@wso2.org
 https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


>>>
>>>
>>> --
>>> *Amila Suriarachchi*
>>>
>>> Software Architect
>>> WSO2 Inc. ; http://wso2.com
>>> lean . enterprise . middleware
>>>
>>> phone : +94 71 3082805
>>>
>>> ___
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> --
>> *S. Suhothayan
>> *
>> Software Engineer,
>> Data Technologies Team,
>>  *WSO2, Inc. **http://wso2.com
>>  *
>> *lean.enterprise.middleware.*
>>
>> *email: **s...@wso2.com* * cell: (+94) 779 756 757
>> blog: **http://suhothayan.blogspot.com/*
>> *
>> twitter: **http://twitter.com/suhothayan* 
>> *
>> linked-in: **http://lk.linkedin.com/in/suhothayan*
>> *
>> *
>>
>
>
>
> --
> *S. Suhothayan
> *
> *Software Engineer,
> Member, Management Committee - Data Technologies Team,
> *

Re: [Architecture] CEP 3.0 towards a Complete eventing solution to the platform

[Mohanadarshan Vivekanandalingam]

Hi All,

Please find some detailed information regarding the Transport Adaptors in
the below link [1].  We are now writing different Transport Adaptor
implementations like WSO2Event-Sender, WSO2Event-Receiver, email,
WSEventLocal & etc and completed some of them...

We are further working on EventBuilder, EventProcessor & EventFormatter at
the moment..


[1]
https://docs.google.com/a/wso2.com/presentation/d/1cto7wBG1y5KfeJqGFgp2FlqUeZz-VcdU30Tvmdwhlpo/edit?usp=sharing


Thanks,
Mohan



On Fri, May 10, 2013 at 3:02 PM, Mohanadarshan Vivekanandalingam <
mo...@wso2.com> wrote:

> Hi All,
>
> We have completed the development of following components...
>
>- Transport Adaptor
>- Transport Adaptor Manager
>
> Now, we are further working on other components and doing improvements for
> them...
>
> Thanks,
> Mohan
>
>
> On Wed, May 8, 2013 at 7:59 PM, Sriskandarajah Suhothayan 
> wrote:
>
>> I have attached the comportment diagram for the following
>>
>>- Transport Adaptor
>>- Transport Adaptor Manager
>>- Event Builder
>>- Event Processor
>>
>> Regards,
>> Suho
>>
>>
>> On Thu, Mar 14, 2013 at 10:50 PM, Sriskandarajah Suhothayan <
>> s...@wso2.com> wrote:
>>
>>> After the discussion with Srinath we came up with the following CEP
>>> architecture.
>>> Here we have five components for CEP
>>>
>>>- Input Transport Adaptor - to receive incoming (XML, JSON, etc )
>>>messages via (JMS, Thrift, etc)
>>>- Event Builder - to convert the (XML, JSON, etc ) messages into
>>>Siddhi/Common Events
>>>- Event Processor
>>>- Event Formatter - to convert  Siddhi/Common Events back to (XML,
>>>JSON, etc ) messages
>>>- Output Transport Adaptor - to publish output (XML, JSON, etc )
>>>messages via (JMS, Thrift, etc)
>>>
>>> With this model any WSO2 server (E.g BAM) can use the Input Transport
>>> Adaptor & Event Builder to receive incoming events
>>> and use Output Transport Adaptor & Event Formatter to notify outputs E.g
>>> Email,
>>> At the same time CEP can also be embedded in to other products (E.g ESB,
>>> ELB, SS) & invoked using  OSGi services buy installing Event Processor (and
>>> if necessary Event Builder & Event Formatter)
>>>
>>> This model also enables events to be passed from one QueryPlan (Bucket)
>>> to another within CEP, which was not possible in CEP 2.x.
>>>
>>> Thoughts?
>>>
>>> Thanks
>>> Suho
>>>
>>>
>>>
>>> On Fri, Jan 11, 2013 at 3:58 PM, Amila Suriarachchi wrote:
>>>
>>>> looks good. +1 for terminology.
>>>>
>>>> thanks,
>>>> Amila.
>>>>
>>>> On Thu, Jan 10, 2013 at 12:13 PM, Sriskandarajah Suhothayan <
>>>> s...@wso2.com> wrote:
>>>>
>>>>> With CEP 2.0 we made Siddhi stable and working for most common
>>>>> use-cases.
>>>>>
>>>>> With CEP 3.0 we need to make CEP work seamlessly with other systems
>>>>> and also provide solutions to all eventing needs of the platform.
>>>>> This is because we have various throttling and notification
>>>>> implementations in the current platform, and with this solution we can 
>>>>> have
>>>>> a unified throttling engine using Siddhi, and a unified notification model
>>>>> provided by CEP so that whoever needs E-mail,SMS,etc notifications can
>>>>> simply use that.
>>>>>
>>>>> I propose we should go for an architecture as described in the
>>>>> attachment.
>>>>>
>>>>> With this, other products (Stratos,BAM,AM,AF) can easily use modules
>>>>> like Notification, and Query&EventProcessor, for notification and Embedded
>>>>> Event processing (throttling) needs.
>>>>>
>>>>> In addition as discussed at the mail "Bringing CEP input mapping
>>>>> configuration into Siddhi" we will make input and output mappings as 
>>>>> Siddhi
>>>>> based configuration instead of the current XML format, hence the Bucket
>>>>> configuration file will not be an XML but be a Script file containing
>>>>> Siddhi.
>>>>> Though we move input,output mapping to Siddhi I still believe we need
>>>>> to have UI for both input and output mappings as it will be more 

[Architecture] [Dev] WSO2 CEP 3.0.0 Milestone 2 Released!

[Mohanadarshan Vivekanandalingam]

*WSO2 Complex Event Processor 3.0.0 Milestone 2 Released!*

Date :5th July 2013

This milestone is available at :
https://svn.wso2.org/repos/wso2/people/suho/packs/cep/3.0.0/wso2cep-3.0.0-M2.zip

To run the samples please follow the read-me,
wso2cep-3.0.0/samples/cep-samples/ReadMe.txt


Following are the bug fixes, improvements and the new features

Bug

   - [CEP-258 ] - When trying to
   deploy the multiple configuration files, File reference is missing
   - [CEP-261 ] - Table is not
   properly generated when creating WSO2 Event mapping and Map mapping

Improvement

   - [CEP-179 ] - It is better if the
   UI supports to check the validity of "Event Sink URL" when subscribing to a
   topic
   - [CEP-243 ] - Separate
   implementation classes & UI for handing in/out sequences of transport
   adapators
   - [CEP-247 ] - Making Coalesce as
   a ExecutorFunction
   - [CEP-248 ] - Removing conversion
   types from output attribute processor
   - [CEP-249 ] - Replacing Condition
   Extentions using bool returning Expression Extentions
   - [CEP-251 ] - White-board pattern
   for Transport Adaptors and Event sources
   - [CEP-257 ] - Event Tracer output
   in UI need to be formatted
   - [CEP-260 ] - UI to retrieve the
   deployed Event Formatter configuration details
   - [CEP-262 ] - EventFormatter to
   receive WSO2Event and format it to other different types.. For products
   like BAM
   - [CEP-267 ] - Listing/editing
   undeployed event builders
   - [CEP-268 ] - Handing dependency
   between the components when removing the configuration files

New Feature

   - [CEP-156 ] - Add a default
   timeStamp convertor
   - [CEP-162 ] - Add Attribute Type
   Convetors to Siddhi
   - [CEP-215 ] - Adding snapshot
   output support for Siddhi window
   - [CEP-231 ] - New component
   called "EventFormatter" to handle the events when sending out from the CEP
   - [CEP-250 ] -
   IsMatchExecutorFunction to regex matching
   - [CEP-252 ] - Wiring between the
   components when adding the configuration files
   - [CEP-253 ] -
   ConvertExecutorFunction for type conversion
   - [CEP-264 ] - Added the ability
   for Siddhi to retrieve properties from registry
   - [CEP-265 ] - Added the ability
   for Siddhi to retrieve resources from registry
   - [CEP-266 ] - Adding/editing
   event builders through UI

Task

   - [CEP-246 ] - Merging Transport
   adaptors and manager compoenents
   - [CEP-263 ] - Added new icon
   images



We welcome your feedback and would love to hear your thoughts on this
release of CEP.

CEP team

-- 
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com *
*lean.enterprise.middleware.*
*
*
email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [Dev] WSO2 CEP 3.0.0 Milestone 2 Released!

[Mohanadarshan Vivekanandalingam]

Hi Cyril,

We are on the way of creating the documentation for CEP 3.0.0...
We'll share the documentation with the CEP 3.0.0 release soon...

Yes, 'non occurrence' of events is already implemented and can be achieved
using the event tables in CEP 3.0.0..

We'll update when CEP 3.0.0 released... Thanks for your interest on CEP...

Regards,
Mohan





On Wed, Jul 10, 2013 at 1:47 PM, Cyril Rognon  wrote:

> Hello cep team,
>
> thank you for this new milestone. I am wondering if there is any
> documentation link for  the upcoming 3.0.0 version.
>
> As far as I know, there is still no logical 'not' operator that we could
> use to filter the 'non occurrence' of events
>
> in siddhi language there is
>
> * **::=  *
>
> it would be nice to be able to say*
> *
>
> * **::= not  *
> I reckon I have not looked into the code to evaluate the cost of this
> requirement, so feel free to turn it down if it is prohibitive.
>
> Maybe eventTables will allow one to simulate this ?
>
> thanks,
> Cyril
>
>
> On Fri, Jul 5, 2013 at 10:47 PM, Mohanadarshan Vivekanandalingam <
> mo...@wso2.com> wrote:
>
>> *WSO2 Complex Event Processor 3.0.0 Milestone 2 Released!*
>>
>> Date :5th July 2013
>>
>> This milestone is available at :
>> https://svn.wso2.org/repos/wso2/people/suho/packs/cep/3.0.0/wso2cep-3.0.0-M2.zip
>>
>> To run the samples please follow the read-me,
>> wso2cep-3.0.0/samples/cep-samples/ReadMe.txt
>>
>>
>> Following are the bug fixes, improvements and the new features
>>
>> Bug
>>
>>- [CEP-258 <https://wso2.org/jira/browse/CEP-258>] - When trying to
>>deploy the multiple configuration files, File reference is missing
>>- [CEP-261 <https://wso2.org/jira/browse/CEP-261>] - Table is not
>>properly generated when creating WSO2 Event mapping and Map mapping
>>
>> Improvement
>>
>>- [CEP-179 <https://wso2.org/jira/browse/CEP-179>] - It is better if
>>the UI supports to check the validity of "Event Sink URL" when subscribing
>>to a topic
>>- [CEP-243 <https://wso2.org/jira/browse/CEP-243>] - Separate
>>implementation classes & UI for handing in/out sequences of transport
>>adapators
>>- [CEP-247 <https://wso2.org/jira/browse/CEP-247>] - Making Coalesce
>>as a ExecutorFunction
>>- [CEP-248 <https://wso2.org/jira/browse/CEP-248>] - Removing
>>conversion types from output attribute processor
>>- [CEP-249 <https://wso2.org/jira/browse/CEP-249>] - Replacing
>>Condition Extentions using bool returning Expression Extentions
>>- [CEP-251 <https://wso2.org/jira/browse/CEP-251>] - White-board
>>pattern for Transport Adaptors and Event sources
>>- [CEP-257 <https://wso2.org/jira/browse/CEP-257>] - Event Tracer
>>output in UI need to be formatted
>>- [CEP-260 <https://wso2.org/jira/browse/CEP-260>] - UI to retrieve
>>the deployed Event Formatter configuration details
>>- [CEP-262 <https://wso2.org/jira/browse/CEP-262>] - EventFormatter
>>to receive WSO2Event and format it to other different types.. For products
>>like BAM
>>- [CEP-267 <https://wso2.org/jira/browse/CEP-267>] - Listing/editing
>>undeployed event builders
>>- [CEP-268 <https://wso2.org/jira/browse/CEP-268>] - Handing
>>dependency between the components when removing the configuration files
>>
>> New Feature
>>
>>- [CEP-156 <https://wso2.org/jira/browse/CEP-156>] - Add a default
>>timeStamp convertor
>>- [CEP-162 <https://wso2.org/jira/browse/CEP-162>] - Add Attribute
>>Type Convetors to Siddhi
>>- [CEP-215 <https://wso2.org/jira/browse/CEP-215>] - Adding snapshot
>>output support for Siddhi window
>>- [CEP-231 <https://wso2.org/jira/browse/CEP-231>] - New component
>>called "EventFormatter" to handle the events when sending out from the CEP
>>- [CEP-250 <https://wso2.org/jira/browse/CEP-250>] -
>>IsMatchExecutorFunction to regex matching
>>- [CEP-252 <https://wso2.org/jira/browse/CEP-252>] - Wiring between
>>the components when adding the configuration files
>>- [CEP-253 <https://wso2.org/jira/browse/CEP-253>] -
>>ConvertExecutorFunction for type conversion
>>- [CEP-264 <https://wso2.org/jira/browse/CEP-264>] - Added the
>>ability for Siddhi to retrieve properties from registry
>>- [CEP-265 <https://wso2.

Re: [Architecture] [Dev] WSO2 CEP 3.0.0 Milestone 2 Released!

[Mohanadarshan Vivekanandalingam]

Hi,

Yes that's true, But at the moment we don't have any time associated with
"not occurrence"... We need to flush the data out from event
tables separately,
or we need to find any other approach.. I think Suho can add more on this...

Regards,
Mohan


On Fri, Jul 12, 2013 at 1:39 PM, Srinath Perera  wrote:

> Hi Mohan,
>
> Is there a time period associated with "not occurrence"? I think we need
> to have that or we will fill up the disk and slow it down as time passes.
>
> --Srinath
>
>
> On Wed, Jul 10, 2013 at 6:31 PM, Mohanadarshan Vivekanandalingam <
> mo...@wso2.com> wrote:
>
>> Hi Cyril,
>>
>> We are on the way of creating the documentation for CEP 3.0.0...
>> We'll share the documentation with the CEP 3.0.0 release soon...
>>
>> Yes, 'non occurrence' of events is already implemented and can be
>> achieved using the event tables in CEP 3.0.0..
>>
>> We'll update when CEP 3.0.0 released... Thanks for your interest on CEP...
>>
>> Regards,
>> Mohan
>>
>>
>>
>>
>>
>> On Wed, Jul 10, 2013 at 1:47 PM, Cyril Rognon  wrote:
>>
>>> Hello cep team,
>>>
>>> thank you for this new milestone. I am wondering if there is any
>>> documentation link for  the upcoming 3.0.0 version.
>>>
>>> As far as I know, there is still no logical 'not' operator that we could
>>> use to filter the 'non occurrence' of events
>>>
>>> in siddhi language there is
>>>
>>> * **::=  *
>>>
>>> it would be nice to be able to say*
>>> *
>>>
>>> * **::= not  *
>>> I reckon I have not looked into the code to evaluate the cost of this
>>> requirement, so feel free to turn it down if it is prohibitive.
>>>
>>> Maybe eventTables will allow one to simulate this ?
>>>
>>> thanks,
>>> Cyril
>>>
>>>
>>> On Fri, Jul 5, 2013 at 10:47 PM, Mohanadarshan Vivekanandalingam <
>>> mo...@wso2.com> wrote:
>>>
>>>> *WSO2 Complex Event Processor 3.0.0 Milestone 2 Released!*
>>>>
>>>> Date :5th July 2013
>>>>
>>>> This milestone is available at :
>>>> https://svn.wso2.org/repos/wso2/people/suho/packs/cep/3.0.0/wso2cep-3.0.0-M2.zip
>>>>
>>>> To run the samples please follow the read-me,
>>>> wso2cep-3.0.0/samples/cep-samples/ReadMe.txt
>>>>
>>>>
>>>> Following are the bug fixes, improvements and the new features
>>>>
>>>> Bug
>>>>
>>>>- [CEP-258 <https://wso2.org/jira/browse/CEP-258>] - When trying to
>>>>deploy the multiple configuration files, File reference is missing
>>>>- [CEP-261 <https://wso2.org/jira/browse/CEP-261>] - Table is not
>>>>properly generated when creating WSO2 Event mapping and Map mapping
>>>>
>>>> Improvement
>>>>
>>>>- [CEP-179 <https://wso2.org/jira/browse/CEP-179>] - It is better
>>>>if the UI supports to check the validity of "Event Sink URL" when
>>>>subscribing to a topic
>>>>- [CEP-243 <https://wso2.org/jira/browse/CEP-243>] - Separate
>>>>implementation classes & UI for handing in/out sequences of transport
>>>>adapators
>>>>- [CEP-247 <https://wso2.org/jira/browse/CEP-247>] - Making
>>>>Coalesce as a ExecutorFunction
>>>>- [CEP-248 <https://wso2.org/jira/browse/CEP-248>] - Removing
>>>>conversion types from output attribute processor
>>>>- [CEP-249 <https://wso2.org/jira/browse/CEP-249>] - Replacing
>>>>Condition Extentions using bool returning Expression Extentions
>>>>- [CEP-251 <https://wso2.org/jira/browse/CEP-251>] - White-board
>>>>pattern for Transport Adaptors and Event sources
>>>>- [CEP-257 <https://wso2.org/jira/browse/CEP-257>] - Event Tracer
>>>>output in UI need to be formatted
>>>>- [CEP-260 <https://wso2.org/jira/browse/CEP-260>] - UI to retrieve
>>>>the deployed Event Formatter configuration details
>>>>- [CEP-262 <https://wso2.org/jira/browse/CEP-262>] - EventFormatter
>>>>to receive WSO2Event and format it to other different types.. For 
>>>> products
>>>>like BAM
>>>>- [CEP-267 <https://wso2.org/jira/browse/CEP-267>

[Architecture] Moving out the FIX Transport from Synapse

[Mohanadarshan Vivekanandalingam]

At the moment there are two types of implementations available for FIX
transport.

1) FIX transport which is in the synapse level that used by ESB

2) Basic FIX transport broker(not released) implementation in CEP

But based on the discussion that we made few months ago, It is not good to
have two separate implementations for FIX transport. But
It is not possible to use Synapse level FIX transport by CEP because of
some limitations. Since it is better to move out the FIX transport
from Synapse and allow to use by any products.

Please refer the architecture mail Thread "FIX Broker for CEP"  for more
information. Please find some notes that we came-up from the
discussion  that we had before.

*  Current ESB FIX Transport*

   - It is designed in the synapse level.
   - When ESB receives a FIX message it removes the header and trailer and
   converts it into xml. Because of this implementation it can handle multiple
   FIX message versions easily.
   - ESB uses only one FIX instance to receive and send FIX messages.
   - Currently its converting the Fix message to XML (here we are not using
   the stranded XML format ) provided by quickfix) in the transport itself and
   sending it as a Synapse message
   - Current implementation only supports 500 TPS.

 * Current CEP FIX Transport*

   - Here there will be two parts. To receive the events and to publish the
   events.
   - We are receiving the FIX events, convert that into the map and pass in
   to the Siddhi.
   - Events that received from siddhi is convert into a FIX message type
   which is defined by the user  using the java reflection.
   - We handle only one port (can have many sessions) to create multiple
   sessions

   *Ideas that came up *

   - We need to have single implementation for a specific Transport (one
   FIX transport for both ESB, CEP and etc...)
   - Moving FIX implementation from synapse to Axis2
   - In common scenario mostly people use custom message tags.
   - Need to develop specific formatter builder to create xml/Map.
   - Map the  Fix message header values as Soap headers when converting to
   XML, this will allow ESB to route the messages more efficiently.


Ideas are welcomed on this... WDYT?

Reagrds,
Mohan


-- 
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com *
*lean.enterprise.middleware.*
*
*
email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Moving out the FIX Transport from Synapse

[Mohanadarshan Vivekanandalingam]

Move is not started yet... It is in the discussion level... We need to find
the best way to do that because we
need to get the participation of ESB team and some others for this...

Regards,
Mohan

On Thu, Jul 18, 2013 at 5:40 AM, Samisa Abeysinghe  wrote:

> When the move is done, who is it supposed to be used by ESB?
>
>
> On Wed, Jul 17, 2013 at 9:09 PM, Mohanadarshan Vivekanandalingam <
> mo...@wso2.com> wrote:
>
>> At the moment there are two types of implementations available for FIX
>> transport.
>>
>> 1) FIX transport which is in the synapse level that used by ESB
>>
>> 2) Basic FIX transport broker(not released) implementation in CEP
>>
>> But based on the discussion that we made few months ago, It is not good
>> to have two separate implementations for FIX transport. But
>> It is not possible to use Synapse level FIX transport by CEP because of
>> some limitations. Since it is better to move out the FIX transport
>> from Synapse and allow to use by any products.
>>
>> Please refer the architecture mail Thread "FIX Broker for CEP"  for more
>> information. Please find some notes that we came-up from the
>> discussion  that we had before.
>>
>> *  Current ESB FIX Transport*
>>
>>- It is designed in the synapse level.
>>- When ESB receives a FIX message it removes the header and trailer
>>and converts it into xml. Because of this implementation it can handle
>>multiple FIX message versions easily.
>>- ESB uses only one FIX instance to receive and send FIX messages.
>>- Currently its converting the Fix message to XML (here we are not
>>using the stranded XML format ) provided by quickfix) in the transport
>>itself and sending it as a Synapse message
>>- Current implementation only supports 500 TPS.
>>
>>  * Current CEP FIX Transport*
>>
>>- Here there will be two parts. To receive the events and to publish
>>the events.
>>- We are receiving the FIX events, convert that into the map and pass
>>in to the Siddhi.
>>- Events that received from siddhi is convert into a FIX message type
>>which is defined by the user  using the java reflection.
>>- We handle only one port (can have many sessions) to create multiple
>>sessions
>>
>>*Ideas that came up *
>>
>>- We need to have single implementation for a specific Transport (one
>>FIX transport for both ESB, CEP and etc...)
>>- Moving FIX implementation from synapse to Axis2
>>- In common scenario mostly people use custom message tags.
>>- Need to develop specific formatter builder to create xml/Map.
>>- Map the  Fix message header values as Soap headers when converting
>>to XML, this will allow ESB to route the messages more efficiently.
>>
>>
>> Ideas are welcomed on this... WDYT?
>>
>> Reagrds,
>> Mohan
>>
>>
>> --
>> *V. Mohanadarshan*
>> *Software Engineer,*
>> *Data Technologies Team,*
>> *WSO2, Inc. http://wso2.com *
>> *lean.enterprise.middleware.*
>> *
>> *
>> email: mo...@wso2.com
>> phone:(+94) 771117673
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
>
> Thanks,
> Samisa...
>
> Samisa Abeysinghe
> VP Engineering
> WSO2 Inc.
> http://wso2.com
> http://wso2.org
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com *
*lean.enterprise.middleware.*
*
*
email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Moving out the FIX Transport from Synapse

[Mohanadarshan Vivekanandalingam]

Well, Actually Initial discussion was not done with the participation of
ESB team, above ideas are came-up
based on the call that we (me and suho) had with Asanka few months back...

But I had an offline chat with Kasun.I regarding this, and He also felt it
is better to move FIX transport
as a separate carbon component but we have not done a proper discussion or
plan regarding how this needs to be
proceed and how is this supposed to be used by ESB..

Regards,
Mohan



On Thu, Jul 18, 2013 at 6:55 PM, Samisa Abeysinghe  wrote:

> Well, I was asking, if we move, then how is this supposed to be used by
> ESB?
>
>
> On Thu, Jul 18, 2013 at 10:10 AM, Mohanadarshan Vivekanandalingam <
> mo...@wso2.com> wrote:
>
>> Move is not started yet... It is in the discussion level... We need to
>> find the best way to do that because we
>> need to get the participation of ESB team and some others for this...
>>
>> Regards,
>> Mohan
>>
>> On Thu, Jul 18, 2013 at 5:40 AM, Samisa Abeysinghe wrote:
>>
>>> When the move is done, who is it supposed to be used by ESB?
>>>
>>>
>>> On Wed, Jul 17, 2013 at 9:09 PM, Mohanadarshan Vivekanandalingam <
>>> mo...@wso2.com> wrote:
>>>
>>>> At the moment there are two types of implementations available for FIX
>>>> transport.
>>>>
>>>> 1) FIX transport which is in the synapse level that used by ESB
>>>>
>>>> 2) Basic FIX transport broker(not released) implementation in CEP
>>>>
>>>> But based on the discussion that we made few months ago, It is not good
>>>> to have two separate implementations for FIX transport. But
>>>> It is not possible to use Synapse level FIX transport by CEP because of
>>>> some limitations. Since it is better to move out the FIX transport
>>>> from Synapse and allow to use by any products.
>>>>
>>>> Please refer the architecture mail Thread "FIX Broker for CEP"  for
>>>> more information. Please find some notes that we came-up from the
>>>> discussion  that we had before.
>>>>
>>>> *  Current ESB FIX Transport*
>>>>
>>>>- It is designed in the synapse level.
>>>>- When ESB receives a FIX message it removes the header and trailer
>>>>and converts it into xml. Because of this implementation it can handle
>>>>multiple FIX message versions easily.
>>>>- ESB uses only one FIX instance to receive and send FIX messages.
>>>>- Currently its converting the Fix message to XML (here we are not
>>>>using the stranded XML format ) provided by quickfix) in the transport
>>>>itself and sending it as a Synapse message
>>>>- Current implementation only supports 500 TPS.
>>>>
>>>>  * Current CEP FIX Transport*
>>>>
>>>>- Here there will be two parts. To receive the events and to
>>>>publish the events.
>>>>- We are receiving the FIX events, convert that into the map and
>>>>pass in to the Siddhi.
>>>>- Events that received from siddhi is convert into a FIX message
>>>>type which is defined by the user  using the java reflection.
>>>>- We handle only one port (can have many sessions) to create
>>>>multiple sessions
>>>>
>>>>*Ideas that came up *
>>>>
>>>>- We need to have single implementation for a specific Transport
>>>>(one FIX transport for both ESB, CEP and etc...)
>>>>- Moving FIX implementation from synapse to Axis2
>>>>- In common scenario mostly people use custom message tags.
>>>>- Need to develop specific formatter builder to create xml/Map.
>>>>- Map the  Fix message header values as Soap headers when
>>>>converting to XML, this will allow ESB to route the messages more
>>>>efficiently.
>>>>
>>>>
>>>> Ideas are welcomed on this... WDYT?
>>>>
>>>> Reagrds,
>>>> Mohan
>>>>
>>>>
>>>> --
>>>> *V. Mohanadarshan*
>>>> *Software Engineer,*
>>>> *Data Technologies Team,*
>>>> *WSO2, Inc. http://wso2.com *
>>>> *lean.enterprise.middleware.*
>>>> *
>>>> *
>>>> email: mo...@wso2.com
>>>> phone:(+94) 771117673
>>>>
>>>> ___

Re: [Architecture] [CEP] TryIt like feature to test Siddhi queries

[Mohanadarshan Vivekanandalingam]

Hi Nirmal,

On Fri, Sep 27, 2013 at 2:02 PM, Nirmal Fernando  wrote:

> Hi All,
>
> Currently there's no easy way to test Siddhi queries. What about having a
> $subject?
>
> Of course, when designing this feature one needs to think about how a
> sample event stream be used etc.
>

+1, I am also thinking about this for some time.. I think we can add this
to our road-map.. It will helpful to learn about
siddhi language also.. But we have to design in a manner to send events
which helps to test most of the scenarios covered by siddhi.

Thanks,
Mohan

>
>
> --
> Thanks & regards,
> Nirmal
>
> Senior Software Engineer- Platform Technologies Team, WSO2 Inc.
> Mobile: +94715779733
> Blog: http://nirmalfdo.blogspot.com/
>
>


-- 
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com *
*lean.enterprise.middleware.*
*
*
email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Improving Usability of CEP 3.0

[Mohanadarshan Vivekanandalingam]

Hi,

First of all, Thanks for giving some good feedback in user perspective. I
am sharing my thoughts in my point of view.. Please correct me if i'm
wrong..

On Fri, Sep 27, 2013 at 12:48 PM, Srinath Perera  wrote:

> Playing with CEP 3.0, i feel we have taken backward step in terms of
> usability.
>
> Note I am not saying we should change this in the version we are putting
> out now, but fix it soon afterword.
>
> CEP 3.0 has introduced a processing pipeline that give users lot of
> flexibility.
> pipeline includes: event receiver, event builder, cep plan, event
> formatter, event sender
>
> This is great. Ideas come from Axis2 arch and we all agreed.
>
> However, CEP forces users to configure all parts of the pipeline all the
> time.
> Now users have to go though all the steps (and we do not have a wizard
> that guide you though :().
>

Whether execution plan wizard not helping for you to achieve this ?.. It
guide users to configure the flow.. Or whether we are missing anything
there..

>
> Problem 1: This is BAD idea, and this make life of the users very hard and
> force them to understand lot of new concepts.
>
> Axis2 has even bigger pipeline, but it does not ask at least one of the
> questions when you want to deploy a service. All extensions  are hidden
> inside defaults and users will only see them if the want to extend the
> flow. If user do not
> know about them, still simple case is simple.
>
> Problem2: Another key issue is while streams are core concept, there is no
> way to list all available streams or add them.
>
> missongFollowing is my proposal to fix the problem.
>
> 1) Add default event listeners (e.g thrift) and senders and make them
> available out of the box.  Others users may configure via Carbon configure
> view (not in main). Then even without configuring  a listener, users can
> use thrift listener.
>

+1, we have already discussed this matter and we have done some development
on this upto execution plan level, we have not touched other parts because
it might break the flow at the release time and also it needs some design
level changes. We are going to focus on this with high priority for next
release.

>
> 2) In main menu, have only ExecuationPlan and Streams. Those are only
> central concepts. (E.g. In WSAS we have services and webapps here).  When
> you creating new Execution plans, it is configured with thrift and data
> bridge formatters for default. Users can edit and change that. When they
> edit, then can either define formatters inline (they are not named .. and
> not added to list .. just like anonymous endpoints  in ESB) or they can
> look them up from global list configured via Configure menu.
>

+1..

>
> 3) Same for streams. But streams are always named. But they can add
> streams without leaving execution plan UI.
>
> +1, I think you have already mentioned regarding this.. Yes it good to
have a common UI to list the streams.


> 4) In in deployment config files, we should support user to give anonymous
> event formatters and builders inline in the execution plan if the wish to.
> They can skip them altogether and then thrift default event formatters and
> builders are used.
>
>
Goal is to keep simple things simple. Now if I need to define a query, I
> say "Add Execution Plan", define streams inline, give the query and I am
> done.
> If I like to do complex things, I can do that too.
>
>
Yes, We can think about this but we needs to consider some important
things. As rajeev mentioned we can do it, in that manner but before that we
have to make sure that it will don't affect the sync between the
configurations. We needs to go through the design again and check how we
are going to achieve that  but we can't allow tight coupling between the
components. Anyway I think, if we can complete the (1) and (2), then it
gives more clear way to achieve this..

Thanks & Regards,
Mohan


> Pls note we (including myself) did not catch this earlier. I am just
> trying to fix the problem, not to put down the nice architecture we have
> done with the pipeline.
>
> WDYT?
>
> --Srinath
>
> --
> 
> Srinath Perera, Ph.D.
>   Director, Research, WSO2 Inc.
>   Visiting Faculty, University of Moratuwa
>   Member, Apache Software Foundation
>   Research Scientist, Lanka Software Foundation
>   Blog: http://srinathsview.blogspot.com/
>   Photos: http://www.flickr.com/photos/hemapani/
>Phone: 0772360902
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*V. Mohanadarshan*
*Software Engineer,*
*Data Technologies Team,*
*WSO2, Inc. http://wso2.com *
*lean.enterprise.middleware.*
*
*
email: mo...@wso2.com
phone:(+94) 771117673
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Changes to CEP 3.0.0 to improve usablity

[Mohanadarshan Vivekanandalingam]

Hi,


On Wed, Oct 2, 2013 at 9:06 PM, Lasantha Fernando  wrote:
>>>
 Hi all,

 The following changes were proposed to CEP 3.0.0 to improve the
 usability of CEP.

 1) EventStream UI is the central place to store all the event streams.
 Event stream management will be decoupled from the EB/EF/EP components and
 a separate component will be created to manage event streams.

 2) Execution Plan UI is the main UI that user is going to use. It will
 popup a dialog to create event streams if a new event stream is needed. It
 will also pop up event builder creation dialog and event formatter creation
 dialog if it is needed for the user.

 But it will not store any values in the backend until the execution
 plan is saved and will save all the related configurations once as an
 atomic transaction.

>>> It is OK to do that if that make it easier for you to implement (but it
>>> is not a requirements).
>>>
>>
>> I think I made everyone misunderstand by using a word like 'atomic
>> transaction'... :-). What I meant was something like this.
>>
>> If the user is doing a custom mapping and wants to save an event
>> builder/formatter, I think we can save it then and there at the time of
>> creation in the UI.
>>
>> However, if the user opts to just create a stream definition and just
>> saves the execution plan, the event processor component will have the
>> responsibility of creating an event builder (or formatter) to match that
>> stream which allows passthrough of events. The event builder would do
>> nothing and simply expose the stream definition that is already incoming
>> via the event adaptor.
>>
>> However, after saving this automatically created builder, when trying to
>> save the execution plan, there might be some exception due to incorrect
>> siddhi syntax or because some other dependency for the execution plan to
>> become active is not met. In that case, the execution plan might not get
>> deployed at all and the created event builder will be leftover. I was
>> thinking that to maintain consistency, in such a case, we would have to
>> ensure that the created event builder would be deleted as well (or the
>> steps done so far be rollbacked).
>>
>> If it is an edge case, then am OK with simply trying to deploy the
>> execution plan + associated event builder + associated event formatter and
>> in case of some deployments being unsuccessful, we can simply leave the
>> artifact in inactive/faulty state.
>>
>
> Ah ok, yes we should cleanup if the execution deployment failed.
>
>>
>>
>>>
 The saving of the execution plan can include the following sub steps.

 1. Create a related event-builder (optional)

 2. Create an execution plan (mandatory)

 3. Create related event-formatters (optional)

 4. Create stream definitions

 Out of the above sub steps, only step 4 will be persisted to the
 backend immediately (since stream definitions are independent from the flow
 configuration). Persisting configuration of sub steps 1-3 would go to
 backend at once and the event-processor component will have the
 responsibility of creating the needed event-builders/formatters and ensure
 the wiring of the flow properly.

 Also, for the case of custom mapping for event builders, changing the
 event builder mapping will effectively change the stream definition that is
 exported by the event builder. Since this stream definition is managed by
 another component, I think we would have to decide on a policy on whether
 we cascade changes for a builder or not. Some possible options regarding
 this will be

 1. When event builder mapping is changed, the exported stream
 definition will change

 2. When the event builder mapping is changed, we force the user to
 change the associated streamdef name or version

 Similarly, when changing the stream definition, we would need to decide
 how the changes will be handled by the separate components.

>>>
>>> Stream definition is the first class thing. There is no concept of
>>> builder importing an stream definition. Builder must agree with stream
>>> definition.
>>> If user change stream definition, we mark corresponding builders/
>>> formatters as faulty. If user try to change the builder/ formatter and it
>>> does not agree with stream definition, we throw an error.  So you first fix
>>> stream definition, then fix the builder.
>>>
>>
>> Hmm... am totally +1 with this from an architectural point of view. It
>> will be clean. But from a usability perspective, it will be a bit of a
>> hassle to the user if they want to change the stream definition since they
>> will have to change in one more additional place when changing the stream
>> definition. But probably this is an unavoidable issue and maybe we can
>> think of something to make it a bit more user-friendly for the next release?
>>
>
> I