Re: [Architecture] Making IOT - APIM communication via REST api's

[Sinthuja Ragendran]

Hi Susinda,

Please see my response inline.

On Thu, Nov 3, 2016 at 11:58 AM, Susinda Perera  wrote:

> Hi All
>
> Currently IOT communicates with APIM components via Java/OSGI api's and
> services. Therefore $subject is needed to properly decouple and make IOT
> cloud ready.
> Consider the following points where IOT uses APIM. Sub-points are huw i'm
> planing to implement
>
> *1. At server startup - It creates and publish apis to APIM*
> 1.1 Creates a client using DCR-endpoint - [1]
> 1.2 Gets a token from token-endpoint using the consumer key/secret
> received from 1.1 - [2]
> 1.3 Creates api from publisher apis using the token received at 1.2 - [3]
> 1.4 Publish api (change life-cycle to PUBLISHED), using the token received
> at 1.2 and api-ID received from 1.3 -[4]
>

> *2. Before an api call *
> 2.1 Create a app calling DCR endpoint[1] - get consumer/key secret
> 2.2 Get a token for  the app created in 2.1 by calling token-endpoint[2]
> 2.3 Create auth app (needs the token received in 2.2) using publisherApi[5]
>

I think from here you meant store API right, not publisher API. I don't
think we do have create application, subscription, etc details from
publisher API, rather it's from store APIs.


> 2.4 Search apis from a given tag using publisherApi[6]
> 2.5 Subscribe to apis (from 2.4) to the app created in 2.3 (needs the
> token received in 2.2) -  using publisherApi[7]
>

In addition to above we also, need to consider a case where the APIs are
getting published after the API application is getting created, ie, future
APIs. If any APIs that are created later after the application is created,
will also need to be subscribed from the created application.

Are we registering one application for all APIs that is there in the
system? In that case are we going to create application with Unlimilted
tier? How are we handling the subscription tier related information for the
published APIs?


> 2.6 Generate keys for app (2.3) (needs the token received in 2.2) -  using
> publisherApi[8]
> 2.7 Get a token from token-endpoint[2] using consumer key/secret received
> at 2.6 above.
>
> *3. When invoking an API -  Does the key validation via APIM*
> 3.1 Uses the token created at 2.7
>
> *4. When device publish its events to MQTT - Does the key validation via
> APIM*
> *?*
>
> *Endpoints being call*
> [1] - http://localhost:9763/client-registration/v0.9/register
> [2] - https://localhost:8243/token
> [3] - https://localhost:9443/api/am/publisher/v0.10/apis
> [4] - https://localhost:9443/api/am/publisher/v0.10/apis/change-
> lifecycle?apiId=
> [5] - https://localhost:9443/api/am/store/v0.10/applications
> [6] - https://localhost:9443/api/am/store/v0.10/apis
> [7] - https://localhost:9443/api/am/store/v0.10/subscriptions
> [8] - https://localhost:9443/api/am/store/v0.10/applications/gen
> erate-keys?applicationId=
>
> *Configs needed*
> (1.1) - DCREndpoint, username and password of a user who has permissions
> to create client-app, callbackUrl,clientName,tokenScope,owner,grantTypem,
> saasApp
> (1.2) - TokenEndpoint, username, (password if we use password grant-type),
> certificate + certPassword if use jwt grant-type
> (1.3 - 1.4) - PublisherApiEndpoint
> (2.1) - DCREndpoint, username and password of a user who has permissions
> to create client-app
> (2.2) - TokenEndpoint, username (and password if we use password
> grant-type)
> (2.3) - StoreApiEndpoint, username and password of a user who has
> permissions to create auth-app, throttlingTier, description, name,
> callbackUrl
> (2.4) - StoreApiEndpoint, tags
> (2.5) - StoreApiEndpoint, tier
> (2.6) - StoreApiEndpoint,
> (2.7) - TokenEndpoint, username (password of the *logged in user* if we
> use password grant-type), certificate + certPassword if use jwt grant-type
>
> *Questions*
> Q1. Can we make 1.1 and 2.1 apps to be SaaS apps
> Q2. Can we use a single (same) app for both 1.1 and 1.2
>

I believe here you mean 1.1 and 2.1, where you create DCR apps right? IMO
yeah, I don't see a need to have two separate application registered to
invoke store and publisher APIs.


> Q3. What is the grant-type we will be using
>

IMO we need to use grant type which could get have the final API invocation
user specific details, and hence I think we should go to password grant
type, or SAML grant type.


> Q4. What would be the additional requirements if we consider multi-tenancy
>

I don't think the current DCR endpoint will allow to create apps in another
tenant, and hence when each tenant is getting registered or login to the
system for the first time, we need create per tenant based DCR applications
(for Publisher and Store API), and API store applications for the device to
invoke.

Q5. Do we perform 1.* in every-time when server starts up
>

I think we need check if the application is already registered or so, and
then register the applications if needed. May be we can have a flag in a
persistence store to indicate this during the initialisation phase. As
mentioned above, not

Re: [Architecture] Making IOT - APIM communication via REST api's

[Sinthuja Ragendran]

Hi Ruwan,

On Thu, Nov 3, 2016 at 1:42 PM, Ruwan Yatawara  wrote:

> Hi Sinthuja,
>
> Please find my comments inline.
>
> Thanks and Regards,
>
> Ruwan Yatawara
>
> Associate Technical Lead,
> WSO2 Inc.
>
> email : ruw...@wso2.com
> mobile : +94 77 9110413
> blog : http://ruwansrants.blogspot.com/
>   https://500px.com/ruwan_ace
> www: :http://wso2.com
>
>
> On Thu, Nov 3, 2016 at 1:18 PM, Sinthuja Ragendran 
> wrote:
>
>> Hi Susinda,
>>
>> Please see my response inline.
>>
>> On Thu, Nov 3, 2016 at 11:58 AM, Susinda Perera  wrote:
>>
>>> Hi All
>>>
>>> Currently IOT communicates with APIM components via Java/OSGI api's and
>>> services. Therefore $subject is needed to properly decouple and make IOT
>>> cloud ready.
>>> Consider the following points where IOT uses APIM. Sub-points are huw
>>> i'm planing to implement
>>>
>>> *1. At server startup - It creates and publish apis to APIM*
>>> 1.1 Creates a client using DCR-endpoint - [1]
>>> 1.2 Gets a token from token-endpoint using the consumer key/secret
>>> received from 1.1 - [2]
>>> 1.3 Creates api from publisher apis using the token received at 1.2 - [3]
>>> 1.4 Publish api (change life-cycle to PUBLISHED), using the token
>>> received at 1.2 and api-ID received from 1.3 -[4]
>>>
>>
>>> *2. Before an api call *
>>> 2.1 Create a app calling DCR endpoint[1] - get consumer/key secret
>>> 2.2 Get a token for  the app created in 2.1 by calling token-endpoint[2]
>>> 2.3 Create auth app (needs the token received in 2.2) using
>>> publisherApi[5]
>>>
>>
>> I think from here you meant store API right, not publisher API. I don't
>> think we do have create application, subscription, etc details from
>> publisher API, rather it's from store APIs.
>>
>>
>>> 2.4 Search apis from a given tag using publisherApi[6]
>>> 2.5 Subscribe to apis (from 2.4) to the app created in 2.3 (needs the
>>> token received in 2.2) -  using publisherApi[7]
>>>
>>
>> In addition to above we also, need to consider a case where the APIs are
>> getting published after the API application is getting created, ie, future
>> APIs. If any APIs that are created later after the application is created,
>> will also need to be subscribed from the created application.
>>
>
> Yes, this needs to be acomodated. Maybe we can have this configured as an
> action during api lifecycle transition.
>
>>
>> Are we registering one application for all APIs that is there in the
>> system? In that case are we going to create application with Unlimilted
>> tier? How are we handling the subscription tier related information for the
>> published APIs?
>>
>>
>>> 2.6 Generate keys for app (2.3) (needs the token received in 2.2) -
>>>  using publisherApi[8]
>>> 2.7 Get a token from token-endpoint[2] using consumer key/secret
>>> received at 2.6 above.
>>>
>>> *3. When invoking an API -  Does the key validation via APIM*
>>> 3.1 Uses the token created at 2.7
>>>
>>> *4. When device publish its events to MQTT - Does the key validation via
>>> APIM*
>>> *?*
>>>
>>> *Endpoints being call*
>>> [1] - http://localhost:9763/client-registration/v0.9/register
>>> [2] - https://localhost:8243/token
>>> [3] - https://localhost:9443/api/am/publisher/v0.10/apis
>>> [4] - https://localhost:9443/api/am/publisher/v0.10/apis/change-
>>> lifecycle?apiId=
>>> [5] - https://localhost:9443/api/am/store/v0.10/applications
>>> [6] - https://localhost:9443/api/am/store/v0.10/apis
>>> [7] - https://localhost:9443/api/am/store/v0.10/subscriptions
>>> [8] - https://localhost:9443/api/am/store/v0.10/applications/gen
>>> erate-keys?applicationId=
>>>
>>> *Configs needed*
>>> (1.1) - DCREndpoint, username and password of a user who has permissions
>>> to create client-app, callbackUrl,clientName,tokenScope,owner,grantTypem,
>>> saasApp
>>> (1.2) - TokenEndpoint, username, (password if we use password
>>> grant-type), certificate + certPassword if use jwt grant-type
>>> (1.3 - 1.4) - PublisherApiEndpoint
>>> (2.1) - DCREndpoint, username and password of a user who has permissions
>>> to create client-app
>>> (2.2) - TokenEndpoint, username (and password if we use password
>>> grant-type)
>>> (2.3) - StoreApiEndpoint, username and password of a us

Re: [Architecture] Dashboard Component Permission Model

[Sinthuja Ragendran]

Hi Tanya,

Small question.

On Wed, Jan 4, 2017 at 10:43 AM, Tanya Madurapperuma  wrote:

> Hi IS Team and all,
>
> Purpose of this mail is to discuss about $ Subject.
>
> There are two main artifact types associated with the dashboard component;
> Dashboards and Widgets.
>
> *Widgets*
> According to the current implementation a widget can be an UUF fragment or
> an UUF component.
> So the plan is to have permission info for each widget in a config
> yaml/json file which will be stored in the file system.
>
> *Dashboards*
> With the previous versions of Dashboard Server we used the role based
> model instead of permission model due to limitations in creating
> permissions dynamically in C4.
> Since this limitation is not there in C5 (@ IS team : please correct me if
> I am wrong), for dashboards also we will go with the permission model.
> Dashboard permission will be stored in the dashboard json file in the file
> system for predefined dashboards. If there is any change in the dashboard
> json file, it will be stored in the database.
>

In that case also can't we update the dashboard.json in the file system
directly? IMHO having the same artefact both in file system, and database
will bring problems of syncing, and I think it's better to have one config
storage.

>
> So from the dashboard components POV, we will consider that dashboard
> components knows the principle.
> And there will be an isAuthorized method provided from identity end where
> we can pass the resource and action to validate whether the logged in user
> is authorized for that particular action.
>
> Based on the above model we have following questions.
> 1. How can we call the isAuthorized method from dashboard component ?
> 2. Is there any standard / approval process for permission strings ?
> 3. How should we register the permissions dynamically at the time of
> creating a dashboard?
>
> Appreciate your insight.
>
> Thanks,
> Tanya
>
> --
> Tanya Madurapperuma
>
> Senior Software Engineer,
> WSO2 Inc. : wso2.com
> Mobile : +94718184439 <071%20818%204439>
> Blog : http://tanyamadurapperuma.blogspot.com
>



-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] tenant specific MQTT receivers in DAS , not listening to topics once tenant get unloaded

[Sinthuja Ragendran]

Hi,

As the receiver configurations are deployable artefacts, those will be
active when the tenant is loaded. One approach is to have all tenants
loaded indefinitely. I think this will have high memory. And therefore we
internally discussed below approach to handling this problem.

Instead of having multiple MQTT receiver configurations per tenant to
handle this, implement a specialised/privileged MQTT event receiver which
could handle multiple subscriptions on behalf of tenants, and it's only
deployable in the super tenant mode. In that case, this event receiver will
have the topic URI with {tenantDomain} placeholder and it is used to
subscribe to the specific tenanted topic. And then, based on which topic
the event has arrived the tenant flow will be started and an event will be
inserted into specific tenant space. By this way, only the tenants which
are actively used/sending events will be loaded, and not all tenants are
required to be loaded.

Please share your thoughts on this. Also, AFAIR we had the similar
requirement for Task execution. @Anjana, how are we handling that?

Thanks,
Sinthuja.

On Mon, Mar 20, 2017 at 10:50 AM, Jasintha Dasanayake 
wrote:

> HI All
>
> When DAS working in tenant mode and a particular tenant has MQTT
> receivers, those cannot be activated once tenants get unloaded. For an
> example , if I restart the DAS then those tenants specific MQTT receivers
> are not loaded unless we explicitly load that particular tenant. IMO,
> expected behavior would be, those receivers should be loaded and subscribed
> to a particular topic without loading the tenants explicitly.
>
> Are there any known mechanism to address this particular problem ?
>
> Thanks and Regards
> /jasintha
>
> --
>
> *Jasintha Dasanayake**Associate Technical Lead*
>
> *WSO2 Inc. | http://wso2.com lean . enterprise .
> middleware*
>
>
> *mobile :- 0711-368-118 <071%20136%208118>*
>



-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] tenant specific MQTT receivers in DAS , not listening to topics once tenant get unloaded

[Sinthuja Ragendran]

Hi Ayyoob,

On Mon, Mar 20, 2017 at 1:54 PM, Ayyoob Hamza  wrote:

> Agree that it's better to handle this in super tenant layer due to its
> complexity, but I wonder whether this needs to be specific for
> MQTT scenarios. Because we need the same requirement for HTTP and thrift
> receivers. So for HTTP and Thrift can we have it through super tenant space
> or do we have to think of a generic solution on how to deploy the artifacts
> for the tenants.
>

Thrift case is different, because there is only one listener for all
tenants and we don't start different endpoints/listeners per tenant. And
the tenent flow is started, once the tenant is authenticated and sending
the events. In the case of HTTP, we do have the same problem as MQTT. In
HTTP case, there is seperate HTTP endpoints registered for different
tenants, and AFAIR those will be removed once the tenant is unloaded. So
far, we didn't have problems in the deployments as HTTP or other receivers
than thrift is used in the multi tenant case.

>
> In addition to recievers, we are using publishers and
> tenant-specific summarisation scripts, does these gets undeployed when the
> tenant is unloaded ?.
>

Summarization scripts will work, AFAIR the tasks registered for the tenants
will be still active though the tenant is unloaded, and once the trigger
has been received for the task, it will be loading the tenant and executing
the script. For the event publishers, the same issue exists, but before the
publisher gets tiggered the event should be arrived into the core, and by
that time the tenant is already loaded and hence I believe the publishers
will not be having this problem.

Thanks,
Sinthuja.


>
> Thanks,
> *Ayyoob Hamza*
> *Software Engineer*
> WSO2 Inc.; http://wso2.com
> email: ayy...@wso2.com cell: +94 77 1681010 <%2B94%2077%207779495>
>



-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] tenant specific MQTT receivers in DAS , not listening to topics once tenant get unloaded

[Sinthuja Ragendran]

Hi Jasintha,

I have a concern on how are we going to handle already created tenants if
we are going to go with different receiver configurations per tenant? It
works for the case if this component is attached for the server runtime
from the initial stages, but if it is to be incorporated into an already
running system then how that the tenanted receiver artefact is going to be
connected? Are you planning to add some component into analytics server
side, to create the receiver configurations for all tenants? I would prefer
this logic to be handled within the custom new receiver implementation,
hence if such receiver configuration is there, automatically the required
logic will be executed, and we can handle all those necessary checks and
validation in the receiver implementation.


On Tue, Mar 21, 2017 at 4:03 PM, Jasintha Dasanayake 
wrote:

> HI
>
> Thanks for all feedback , So I am going to implement this by creating
> event receiver configuration dynamically for each tenant and deploying into
> DAS using the event receiver admin service. As Mohan mentioned , IMO
> keeping a per tenant receiver will be a better option because an existing
> subscriptions will not be effected when adding/removing another tenant's
> artifacts.
>

IMHO if you have one receiver configuration/or seperate receiver
configuration doesn't make a difference here, in both cases any tenanted
behaviour will not interfere with other.

Additionally, if you are going with seperate receiver configurations per
tenant, then you need to make sure, that you don't create multiple receiver
configurations for same tenant.

Thanks,
Sinthuja.


>
> Thanks
> /Jasintha
>
> On Tue, Mar 21, 2017 at 11:27 AM, Anjana Fernando  wrote:
>
>> Hi,
>>
>> On Mon, Mar 20, 2017 at 11:02 AM, Sinthuja Ragendran 
>> wrote:
>>
>>> Hi,
>>>
>>> As the receiver configurations are deployable artefacts, those will be
>>> active when the tenant is loaded. One approach is to have all tenants
>>> loaded indefinitely. I think this will have high memory. And therefore we
>>> internally discussed below approach to handling this problem.
>>>
>>> Instead of having multiple MQTT receiver configurations per tenant to
>>> handle this, implement a specialised/privileged MQTT event receiver which
>>> could handle multiple subscriptions on behalf of tenants, and it's only
>>> deployable in the super tenant mode. In that case, this event receiver will
>>> have the topic URI with {tenantDomain} placeholder and it is used to
>>> subscribe to the specific tenanted topic. And then, based on which topic
>>> the event has arrived the tenant flow will be started and an event will be
>>> inserted into specific tenant space. By this way, only the tenants which
>>> are actively used/sending events will be loaded, and not all tenants are
>>> required to be loaded.
>>>
>>> Please share your thoughts on this. Also, AFAIR we had the similar
>>> requirement for Task execution. @Anjana, how are we handling that?
>>>
>>
>> Yes, the tasks and their definitions are stored in the super tenant
>> space. So they get triggered appropriately, and as required, any tenant
>> specific resources would be loaded by the task implementation.
>>
>> Cheers,
>> Anjana.
>>
>>
>>>
>>> Thanks,
>>> Sinthuja.
>>>
>>> On Mon, Mar 20, 2017 at 10:50 AM, Jasintha Dasanayake >> > wrote:
>>>
>>>> HI All
>>>>
>>>> When DAS working in tenant mode and a particular tenant has MQTT
>>>> receivers, those cannot be activated once tenants get unloaded. For an
>>>> example , if I restart the DAS then those tenants specific MQTT receivers
>>>> are not loaded unless we explicitly load that particular tenant. IMO,
>>>> expected behavior would be, those receivers should be loaded and subscribed
>>>> to a particular topic without loading the tenants explicitly.
>>>>
>>>> Are there any known mechanism to address this particular problem ?
>>>>
>>>> Thanks and Regards
>>>> /jasintha
>>>>
>>>> --
>>>>
>>>> *Jasintha Dasanayake**Associate Technical Lead*
>>>>
>>>> *WSO2 Inc. | http://wso2.com <http://wso2.com/>lean . enterprise .
>>>> middleware*
>>>>
>>>>
>>>> *mobile :- 0711-368-118 <071%20136%208118>*
>>>>
>>>
>>>
>>>
>>> --
>>> *Sinthuja Rajendran*
>>> Technical Lead
>>> WSO2, Inc.:http://wso2.com
>>>
>>> Blog: http://sinthu-rajan.blogspot.com/
>>> Mobile: +94774273955 <077%20427%203955>
>>>
>>>
>>>
>>
>>
>> --
>> *Anjana Fernando*
>> Associate Director / Architect
>> WSO2 Inc. | http://wso2.com
>> lean . enterprise . middleware
>>
>
>
>
> --
>
> *Jasintha Dasanayake**Associate Technical Lead*
>
> *WSO2 Inc. | http://wso2.com <http://wso2.com/>lean . enterprise .
> middleware*
>
>
> *mobile :- 0711-368-118 <071%20136%208118>*
>



-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [IoT] Improve Performance and Reliability of Push notifications

[Sinthuja Ragendran]

Hi,

+1 for the second option of handling it simply within our code level
without going through the additional hop.

On Wed, Mar 22, 2017 at 3:58 PM, Chathura Ekanayake 
wrote:

> I think option 2 is better. Using a message store will also eventually
> access a persistent storage (probably with multiple io calls), so the
> additional db call in option 2 is not significant. In addition, option 1
> adds an additional hop to the message flow.
>
> Why do we need a concurrent queue for option 2 (as we anyway store all
> operations in db)? Is it as a cache?
>
> - Chathura
>
> On Wed, Mar 22, 2017 at 11:52 AM, Waruna Jayaweera 
> wrote:
>
>> Hi,
>>
>> In  Device + Server Communication Push notifications used to notify
>> devices to retrieve the device operations. This also use in policy
>> monitoring where we send push notification to all devices during monitoring
>> task. There are different push notification strategies as for APNS, for
>> android - GCM/FCM and for Windows - WNS also MQTT based push notifications.
>>
>> *Problem*
>> In current implementation when you set device operation to multiple
>> devices, or during policy monitoring
>> all push notifications are send all devices at once. This will cause
>> sudden request burst when all devices try access the server for device
>> operations. We also needs to think about of reliability of push
>> notifications if those got failed. To solve this issue we have multiple
>> solutions.
>>
>> *Solutions*
>>
>> *1) Use Message store and Message Processors*
>>
>> We can send each push notification message to inbuilt synapse message
>> store via proxy. Then we can have sampling message processor to send those
>> push notifications to given notification provider. In sampling message
>> processor we can define no of message per given time and time interval per
>> batch of messages. Push notification logic will be inside sequence with
>> custom mediator.
>>
>> Pros;
>> Reliability can be achieved using jms message store.
>> minimum code
>>
>> Cons;
>> IOT server have to depend on message processors.
>>
>>
>> *2) Java Queue with Scheduler Task*
>>
>> We can have java concurrent queue to store device notifications as batch
>> jobs.
>>
>
Do you mean that you are going to have store all device notifications jobs
in memory?  Ie, it scales up with number of devices?

Scheduler task will be there to read jobs and process them and send
>> notifications. Batch job object will content set of devices given batch
>> size and scheduler task will be run based on delay interval.
>>
>
Thanks,
Sinthuja.


>> Pros;
>> Do not need to rely on other feature
>>
>> Cons;
>> We need to maintain push status in database so there will be additional
>> database call in order to make reliability of notification sending. Later
>> we can have retry logic to failed notifications based on db status.
>>
>> If  device list is below the no of devices per batch, those commands will
>> send immediately.
>> IMO option 2 is better since we do not rely on any other implementation.
>> Appreciate any suggestions.
>>
>> Thanks,
>> Waruna
>>
>> --
>> Regards,
>>
>> Waruna Lakshitha Jayaweera
>> Senior Software Engineer
>> WSO2 Inc; http://wso2.com
>> phone: +94713255198 <+94%2071%20325%205198>
>> http://waruapz.blogspot.com/
>>
>>
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [IoT] Improve Performance and Reliability of Push notifications

[Sinthuja Ragendran]

Hi Waruna,

On Wed, Mar 22, 2017 at 10:41 PM, Waruna Jayaweera  wrote:

> Hi Chathura,
>
> We can not read from db , since in a clustered environment when multiple
> nodes try read from same operation table will cause duplication of push
> message sending.
>

Can't we make use of dynamic partitioning for this case? i.e, we can
dynamically partition the push requests based on number of nodes available
in the cluster, and particular node will be serving the requests allocated
for that node only.

Anyway these bulk device request will come to given node at a time. So
> device can store them  in queue
>

You mean the node can store them in queue? Actually what sort of details
you are going to hold in-memory? Is it basically going to have some sort of
map where the device id is key, and the list of messges that needs to be
pushed into the system as a list? In that case, scaling will be a problem?
And what if that node goes down, how can we reload the state? I may be
missing some details, but can you please explain full flow in bit more
detail?

Thanks,
Sinthuja.

for temporary for sending push messages as batches .  If node restart we
> can have retry task ( Only in manager node) to read from database.
>

> Thanks,
> Waruna
>
> On Wed, Mar 22, 2017 at 3:58 PM, Chathura Ekanayake 
> wrote:
>
>> I think option 2 is better. Using a message store will also eventually
>> access a persistent storage (probably with multiple io calls), so the
>> additional db call in option 2 is not significant. In addition, option 1
>> adds an additional hop to the message flow.
>>
>> Why do we need a concurrent queue for option 2 (as we anyway store all
>> operations in db)? Is it as a cache?
>>
>> - Chathura
>>
>> On Wed, Mar 22, 2017 at 11:52 AM, Waruna Jayaweera 
>> wrote:
>>
>>> Hi,
>>>
>>> In  Device + Server Communication Push notifications used to notify
>>> devices to retrieve the device operations. This also use in policy
>>> monitoring where we send push notification to all devices during monitoring
>>> task. There are different push notification strategies as for APNS, for
>>> android - GCM/FCM and for Windows - WNS also MQTT based push notifications.
>>>
>>> *Problem*
>>> In current implementation when you set device operation to multiple
>>> devices, or during policy monitoring
>>> all push notifications are send all devices at once. This will cause
>>> sudden request burst when all devices try access the server for device
>>> operations. We also needs to think about of reliability of push
>>> notifications if those got failed. To solve this issue we have multiple
>>> solutions.
>>>
>>> *Solutions*
>>>
>>> *1) Use Message store and Message Processors*
>>>
>>> We can send each push notification message to inbuilt synapse message
>>> store via proxy. Then we can have sampling message processor to send those
>>> push notifications to given notification provider. In sampling message
>>> processor we can define no of message per given time and time interval per
>>> batch of messages. Push notification logic will be inside sequence with
>>> custom mediator.
>>>
>>> Pros;
>>> Reliability can be achieved using jms message store.
>>> minimum code
>>>
>>> Cons;
>>> IOT server have to depend on message processors.
>>>
>>>
>>> *2) Java Queue with Scheduler Task*
>>>
>>> We can have java concurrent queue to store device notifications as batch
>>> jobs. Scheduler task will be there to read jobs and process them and send
>>> notifications. Batch job object will content set of devices given batch
>>> size and scheduler task will be run based on delay interval.
>>>
>>> Pros;
>>> Do not need to rely on other feature
>>>
>>> Cons;
>>> We need to maintain push status in database so there will be additional
>>> database call in order to make reliability of notification sending. Later
>>> we can have retry logic to failed notifications based on db status.
>>>
>>> If  device list is below the no of devices per batch, those commands
>>> will send immediately.
>>> IMO option 2 is better since we do not rely on any other implementation.
>>> Appreciate any suggestions.
>>>
>>> Thanks,
>>> Waruna
>>>
>>> --
>>> Regards,
>>>
>>> Waruna Lakshitha Jayaweera
>>> Senior Software Engineer
>>> WSO2 Inc; http://wso2.com
>>> phone: +94713255198 <+94%2071%20325%205198>
>>> http://waruapz.blogspot.com/
>>>
>>>
>>
>
>
> --
> Regards,
>
> Waruna Lakshitha Jayaweera
> Senior Software Engineer
> WSO2 Inc; http://wso2.com
> phone: +94713255198 <071%20325%205198>
> http://waruapz.blogspot.com/
>
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [IoT] Improve Performance and Reliability of Push notifications

[Sinthuja Ragendran]

Hi Geeth,


On Thu, Mar 23, 2017 at 11:42 AM, Geeth Munasinghe  wrote:

> Hi all,
>
> IMO, having DB update and in-memory queues are two different methods we
> can resolve this issue. Updating DB supports near reliable messaging and
> this can be done only from a single server. Because we cannot have multiple
> servers reading and updating database without coordination. Implementing
> coordination will be much complex task.
>

IMO we need to go with database sharding for this, as these are independant
notifications for devices, I don't think we need the complex coordinatio
logic for this.

>
>
> And im-memory queues can be run from any worker node, hence supports
> distributed architecture. This way it is will easy to manage and implement.
>

You mean to use the distributed in-memory queue? i.e., hazelcast queue as
we are in C4 based components? I think we had experienced some issues with
that in past, and trying hence we should avoid using it IMO.

Thanks,
Sinthuja.


>
> So in both approaches, there are pros and cons. DB update approach method
> is if some one needs reliable push notification. And in-memory queues can
> support system which does not need reliable messaging. In my opinion, emm
> use cases are such. If one message is not delivered to the device will not
> affect much, because if it receives at least one push message later, device
> will wake up and will retrieve all the pending operation at one go.
>
> Therefore I think we need to implement both of these approaches and let
> user select whatever best-way they deemed necessary.
>
> And apart from that, these two approaches requires managing the load only
> when device wake up message is sent as the push notification. But in MQTT
> or whole message is sent as push notification, we do not need any of the
> above approaches. AFAIT So most of IoT related use case will not require
> this kind of load balancing as it can be done from push messaging server.
>
> Thanks
> Geeth
>
> On Thu, Mar 23, 2017 at 9:17 AM, Chathura Ekanayake 
> wrote:
>
>> Without having a separate retry task, can we have a single task running
>> in one one to read messages from the db and send to devices. Then the
>> behavior for sending incoming operation requests and sending messages after
>> a restart will be similar. In addition, we can avoid an in-memory store of
>> messages, which could cause scalability issues.
>>
>> - Chathura
>>
>> On Wed, Mar 22, 2017 at 10:41 PM, Waruna Jayaweera 
>> wrote:
>>
>>> Hi Chathura,
>>>
>>> We can not read from db , since in a clustered environment when multiple
>>> nodes try read from same operation table will cause duplication of push
>>> message sending. Anyway these bulk device request will come to given node
>>> at a time. So device can store them in queue for temporary for sending push
>>> messages as batches.  If node restart we can have retry task ( Only in
>>> manager node) to read from database.
>>>
>>> Thanks,
>>> Waruna
>>>
>>> On Wed, Mar 22, 2017 at 3:58 PM, Chathura Ekanayake 
>>> wrote:
>>>
 I think option 2 is better. Using a message store will also eventually
 access a persistent storage (probably with multiple io calls), so the
 additional db call in option 2 is not significant. In addition, option 1
 adds an additional hop to the message flow.

 Why do we need a concurrent queue for option 2 (as we anyway store all
 operations in db)? Is it as a cache?

 - Chathura

 On Wed, Mar 22, 2017 at 11:52 AM, Waruna Jayaweera 
 wrote:

> Hi,
>
> In  Device + Server Communication Push notifications used to notify
> devices to retrieve the device operations. This also use in policy
> monitoring where we send push notification to all devices during 
> monitoring
> task. There are different push notification strategies as for APNS, for
> android - GCM/FCM and for Windows - WNS also MQTT based push 
> notifications.
>
> *Problem*
> In current implementation when you set device operation to multiple
> devices, or during policy monitoring
> all push notifications are send all devices at once. This will cause
> sudden request burst when all devices try access the server for device
> operations. We also needs to think about of reliability of push
> notifications if those got failed. To solve this issue we have multiple
> solutions.
>
> *Solutions*
>
> *1) Use Message store and Message Processors*
>
> We can send each push notification message to inbuilt synapse message
> store via proxy. Then we can have sampling message processor to send those
> push notifications to given notification provider. In sampling message
> processor we can define no of message per given time and time interval per
> batch of messages. Push notification logic will be inside sequence with
> custom mediator.
>
> Pros;
> Reliability can be achieved using jms message store.
> mini

Re: [Architecture] ESB Analytics Mediation Event Publishing Mechanism

[Sinthuja Ragendran]

Hi Supun,

I think we could simplify the json message bit more. Instead of 'null' for
the payload attributes in the events section, you could use the actual
payload name directly if there is a payload for that event. And in that
case, we could eliminate the 'events' section from the 'payloads' section.
For the given example, it could be altered as below.

{
'events': [{
'messageId': 'aaa',
'componentId': '111',
'payload': '*payload1*',
'componentName': 'Proxy:TestProxy',
'output-payload':null
}, {
'messageId': 'bbb',
'componentId': '222',
'componentName': 'Proxy:TestProxy',
'payload': '*payload1*',
'output-payload':null
}, {
'messageId': 'ccc',
'componentId': '789',
'payload': '*payload2*',
'componentName': 'Proxy:TestProxy',
'output-payload':'*payload2*'
}],

'payloads': {
'*payload1*': 'xml-payload-1',
'*payload2*': 'xml-payload-2',
}
}

Thanks,
Sinthuja.

On Wed, Feb 10, 2016 at 10:18 AM, Supun Sethunga  wrote:

> Hi Budhdhima/Viraj,
>
> As per the discussion we had yesterday, follow is the format of the json
> contains aggregated event details, to be sent to DAS. (you may change the
> attribute names of events).
>
> To explain it further, "events" contains the details about each event sent
> by each mediator. Payload may or may not be populated. "Payloads" section
> contains unique payloads and the mapping to the events their fields.
> (eg:  'xml-payload-2' maps to the 'payload' and 'output-payload' fields of
> the 3rd event).
>
> {
> 'events': [{
> 'messageId': 'aaa',
> 'componentId': '111',
> 'payload': null,
>
'componentName': 'Proxy:TestProxy',
> 'output-payload':null
> }, {
> 'messageId': 'bbb',
> 'componentId': '222',
> 'componentName': 'Proxy:TestProxy',
> 'payload': null,
> 'output-payload':null
> }, {
> 'messageId': 'ccc',
> 'componentId': '789',
> 'payload': null,
> 'componentName': 'Proxy:TestProxy',
> 'output-payload':null
> }],
>
> 'payloads': [{
> 'payload': 'xml-payload-1',
> 'events': [{
> 'eventIndex': 0,
> 'attributes':['payload']
> }, {
> 'eventIndex': 1,
> 'attributes':['payload']
> }]
> }, {
> 'payload': 'xml-payload-2',
> 'events': [{
> 'eventIndex': 2,
> 'attributes':['payload','output-payload']
> }]
> }]
> }
>
> Please let us know any further clarifications is needed, or if there's
> anything to be modified/improved.
>
> Thanks,
> Supun
>
> On Tue, Feb 9, 2016 at 11:05 AM, Isuru Udana  wrote:
>
>> Hi Kasun,
>>
>> On Tue, Feb 9, 2016 at 10:10 AM, Kasun Indrasiri  wrote:
>>
>>> I think for trancing use case we need to publish events one by one from
>>> each mediator (we can't aggregate all such events as it also contains the
>>> message payload)
>>>
>> I think we can still do that with some extra effort.
>> Most of the mediators in a sequence flow does not alter the message
>> payload. We can store the payload only for the mediators which alter the
>> message payload. And for others, we can put a reference to the previous
>> entry. By doing that we can save the memory to a great extent.
>>
>> Thanks.
>>
>>
>>>
>>> -- Forwarded message --
>>> From: Supun Sethunga 
>>> Date: Mon, Feb 8, 2016 at 2:54 PM
>>> Subject: Re: ESB Analytics Mediation Event Publishing Mechanism
>>> To: Anjana Fernando 
>>> Cc: "engineering-gr...@wso2.com" , Srinath
>>> Perera , Sanjiva Weerawarana ,
>>> Kasun Indrasiri , Isuru Udana 
>>>
>>>
>>> Hi all,
>>>
>>> Ran some simple performance tests against the new relational provider,
>>> in comparison with the existing one. Follow are the results:
>>>
>>> *Records in Backend DB Table*: *1,054,057*
>>>
>>> *Conversion:*
>>> Spark Table
>>> id a b c
>>> Backend DB Table 1 xxx yyy zzz
>>> id data 1 ppp qqq rrr
>>> 1
>>> [{'a':'aaa','b':'bbb','c':'ccc'},{'a':'xxx','b':'yyy','c':'zzz'},{'a':'ppp','b':'qqq','c':'rrr'}]
>>>  --
>>> To --> 1 aaa bbb ccc
>>> 2
>>> [{'a':'aaa','b':'bbb','c':'ccc'},{'a':'xxx','b':'yyy','c':'zzz'},{'a':'ppp','b':'qqq','c':'rrr'}]
>>> 2 xxx yyy zzz
>>> 2 aaa bbb ccc
>>> 2 ppp qqq rrr
>>>
>>>
>>>
>>> *Avg Time for Query Execution:*
>>>
>>> Querry
>>> Execution time (~ sec)
>>> Existing Analytics Relation Provider New (ESB) Analytics Relation
>>> Provider* * New relational provider split a single row to multiple
>>> rows. Hence the number of rows in the table equivalent to 3 times (as each
>>> row is split to 3 rows) as the original table.
>>> SELECT COUNT(*) FROM ; 13 16
>>> SELECT * FROM  ORDER BY id ASC; 13 16
>>> SELECT * FROM  WHERE id=98435; 13 16
>>> SELECT id,a,first(b),first(c) FROM  GROUP BY id,a ORDER BY id ASC;
>>> 18 26
>>>
>>> Regards,
>>> Supun
>>>
>>> On Wed, Feb 3, 2016 at 3:36 PM, Supun Sethunga  wrote:
>>>
 Hi all,

 I have started working on implementing a new "relation" / "relation
 provider", to serve the above requirement. This basically is a modified
 version of the existing "Carbon Analytics" relation provider.

 Here I have assumed that the encapsulated data for a single execution
 flow are stored in a single row, and the data about the mediators
 invoked during the

Re: [Architecture] ESB Analytics Mediation Event Publishing Mechanism

[Sinthuja Ragendran]

Hi Supun,


On Wed, Feb 10, 2016 at 11:14 AM, Supun Sethunga  wrote:

> Hi Sinthuja,
>
> Agree on the possibility of simplifying the json. We also discussed on the
> same matter yesterday, but the complication came up was, by an event in the
> "events" list, payload could be either referenced, or defined
> in-line.(made as it is, so that it can be generalized for other fields as
> well if needed, other than payloads.).
>
In such a case, if we had defined as 'payload': '*payload1**', *we would
> not know if its the actual payload, or a reference to the payload in the
> "payloads" section.
>
> With the suggested format, DAS will only go and map the payload if its
> null.
>
>
IMHO we could solve this issue as having conversions. Basically we could
use $payloads:payload1 to reference the elements as a convention. If the
element starts with '$' then it's the reference, not the actual payload. In
that case if there is a new element introduced, let's say foo and you need
to access the property property1, then it will have the reference as
$foo:property1.

Thanks,
Sinthuja.



> Regards,
> Supun
>
> On Wed, Feb 10, 2016 at 10:52 AM, Sinthuja Ragendran 
> wrote:
>
>> Hi Supun,
>>
>> I think we could simplify the json message bit more. Instead of 'null'
>> for the payload attributes in the events section, you could use the actual
>> payload name directly if there is a payload for that event. And in that
>> case, we could eliminate the 'events' section from the 'payloads' section.
>> For the given example, it could be altered as below.
>>
>> {
>> 'events': [{
>> 'messageId': 'aaa',
>> 'componentId': '111',
>> 'payload': '*payload1*',
>> 'componentName': 'Proxy:TestProxy',
>> 'output-payload':null
>> }, {
>> 'messageId': 'bbb',
>> 'componentId': '222',
>> 'componentName': 'Proxy:TestProxy',
>> 'payload': '*payload1*',
>> 'output-payload':null
>> }, {
>> 'messageId': 'ccc',
>> 'componentId': '789',
>> 'payload': '*payload2*',
>> 'componentName': 'Proxy:TestProxy',
>> 'output-payload':'*payload2*'
>> }],
>>
>> 'payloads': {
>> '*payload1*': 'xml-payload-1',
>> '*payload2*': 'xml-payload-2',
>> }
>> }
>>
>> Thanks,
>> Sinthuja.
>>
>> On Wed, Feb 10, 2016 at 10:18 AM, Supun Sethunga  wrote:
>>
>>> Hi Budhdhima/Viraj,
>>>
>>> As per the discussion we had yesterday, follow is the format of the json
>>> contains aggregated event details, to be sent to DAS. (you may change the
>>> attribute names of events).
>>>
>>> To explain it further, "events" contains the details about each event
>>> sent by each mediator. Payload may or may not be populated. "Payloads"
>>> section contains unique payloads and the mapping to the events their
>>> fields. (eg:  'xml-payload-2' maps to the 'payload' and 'output-payload'
>>> fields of the 3rd event).
>>>
>>> {
>>> 'events': [{
>>> 'messageId': 'aaa',
>>> 'componentId': '111',
>>> 'payload': null,
>>>
>> 'componentName': 'Proxy:TestProxy',
>>> 'output-payload':null
>>> }, {
>>> 'messageId': 'bbb',
>>> 'componentId': '222',
>>> 'componentName': 'Proxy:TestProxy',
>>> 'payload': null,
>>> 'output-payload':null
>>> }, {
>>> 'messageId': 'ccc',
>>> 'componentId': '789',
>>> 'payload': null,
>>> 'componentName': 'Proxy:TestProxy',
>>> 'output-payload':null
>>> }],
>>>
>>> 'payloads': [{
>>> 'payload': 'xml-payload-1',
>>> 'events': [{
>>> 'eventIndex': 0,
>>> 'attributes':['payload']
>>> }, {
>>> 'eventIndex': 1,
>>> 'attributes':['payload']
>>> }]
>>> }, {
>>> 'payload': &#

Re: [Architecture] ESB Analytics Mediation Event Publishing Mechanism

[Sinthuja Ragendran]

Hi Supun,

Also this json event format is being sent as event payload in wso2 event,
and wso2 event is being published by the data publisher right? Correct me
if i'm wrong.

Thanks,
Sinthuja.

On Wed, Feb 10, 2016 at 11:26 AM, Sinthuja Ragendran 
wrote:

> Hi Supun,
>
>
> On Wed, Feb 10, 2016 at 11:14 AM, Supun Sethunga  wrote:
>
>> Hi Sinthuja,
>>
>> Agree on the possibility of simplifying the json. We also discussed on
>> the same matter yesterday, but the complication came up was, by an event in
>> the "events" list, payload could be either referenced, or defined
>> in-line.(made as it is, so that it can be generalized for other fields as
>> well if needed, other than payloads.).
>>
> In such a case, if we had defined as 'payload': '*payload1**', *we would
>> not know if its the actual payload, or a reference to the payload in the
>> "payloads" section.
>>
>> With the suggested format, DAS will only go and map the payload if its
>> null.
>>
>>
> IMHO we could solve this issue as having conversions. Basically we could
> use $payloads:payload1 to reference the elements as a convention. If the
> element starts with '$' then it's the reference, not the actual payload. In
> that case if there is a new element introduced, let's say foo and you need
> to access the property property1, then it will have the reference as
> $foo:property1.
>
> Thanks,
> Sinthuja.
>
>
>
>> Regards,
>> Supun
>>
>> On Wed, Feb 10, 2016 at 10:52 AM, Sinthuja Ragendran 
>> wrote:
>>
>>> Hi Supun,
>>>
>>> I think we could simplify the json message bit more. Instead of 'null'
>>> for the payload attributes in the events section, you could use the actual
>>> payload name directly if there is a payload for that event. And in that
>>> case, we could eliminate the 'events' section from the 'payloads' section.
>>> For the given example, it could be altered as below.
>>>
>>> {
>>> 'events': [{
>>> 'messageId': 'aaa',
>>> 'componentId': '111',
>>> 'payload': '*payload1*',
>>> 'componentName': 'Proxy:TestProxy',
>>> 'output-payload':null
>>> }, {
>>> 'messageId': 'bbb',
>>> 'componentId': '222',
>>> 'componentName': 'Proxy:TestProxy',
>>> 'payload': '*payload1*',
>>> 'output-payload':null
>>> }, {
>>> 'messageId': 'ccc',
>>> 'componentId': '789',
>>> 'payload': '*payload2*',
>>> 'componentName': 'Proxy:TestProxy',
>>> 'output-payload':'*payload2*'
>>> }],
>>>
>>> 'payloads': {
>>> '*payload1*': 'xml-payload-1',
>>> '*payload2*': 'xml-payload-2',
>>> }
>>> }
>>>
>>> Thanks,
>>> Sinthuja.
>>>
>>> On Wed, Feb 10, 2016 at 10:18 AM, Supun Sethunga 
>>> wrote:
>>>
>>>> Hi Budhdhima/Viraj,
>>>>
>>>> As per the discussion we had yesterday, follow is the format of the
>>>> json contains aggregated event details, to be sent to DAS. (you may change
>>>> the attribute names of events).
>>>>
>>>> To explain it further, "events" contains the details about each event
>>>> sent by each mediator. Payload may or may not be populated. "Payloads"
>>>> section contains unique payloads and the mapping to the events their
>>>> fields. (eg:  'xml-payload-2' maps to the 'payload' and 'output-payload'
>>>> fields of the 3rd event).
>>>>
>>>> {
>>>> 'events': [{
>>>> 'messageId': 'aaa',
>>>> 'componentId': '111',
>>>> 'payload': null,
>>>>
>>> 'componentName': 'Proxy:TestProxy',
>>>> 'output-payload':null
>>>> }, {
>>>> 'messageId': 'bbb',
>>>> 'componentId': '222',
>>>> 'componentName': 'Proxy:TestProxy',
>>>> 'payload': null,
>>>> 'output-payload':null
>>>> }, {
>

Re: [Architecture] [Analytics] Improvements to Lucene based Aggregate functions (Installing Aggregates as OSGI components)

[Sinthuja Ragendran]

Hi Gimantha,

On Thu, Mar 17, 2016 at 9:05 PM, Gimantha Bandara  wrote:

> Hi all,
>
> DAS 3.0.0 and DAS 3.0.1 has 5 built-in Aggregates[1][2] namely MIN, MAX,
> SUM, AVG and COUNT which can be used in interactive analytics. Sometimes
> some scenarios may require custom aggregate functions for aggregate
> calculations. For this we have introduced a new interface from
> carbon-analytics/analytics-core, which has three methods which need to be
> implemented.
>
> Interface :
> org.wso2.carbon.analytics.dataservice.core.indexing.aggregates.AggregateFunction
>
> Following are the three methods that need to be implemented.
>
> *public void process(RecordValuesContext ctx, String[] aggregateFields)
> throws AnalyticsException*;
>
> This method will get called for all the records which need to be
> aggregated. RecordValuesContext will contain the record values of the
> current record being processed. aggregateFields will contain an array of
> fields which will be used for the aggregation. The order of the
> aggregateFields will matter when we implement the logic. For example, lets
> say we are going to implement SUM aggregate. Then we know that only one
> field will be required to calculate SUM and always aggregateFields will
> contain one field name in it, which is the name of the field being SUMed.
>
> public void process(RecordValuesContext ctx, String[] aggregateFields)
> throws AnalyticsException {
> if (aggregateFields == null || aggregateFields.length == 0) {
> throw new AnalyticsException("Field to be aggregated, is
> missing");
> }
> Object value = ctx.getValue(aggregateFields[0]);
> if (value == null) {
> throw new AnalyticsException("Error while calculating SUM:
> value of the field, " +
>  aggregateFields[0] + " is null");
> }
> if (value instanceof Number) {
> sum += ((Number)value).doubleValue();
> } else {
> throw new AnalyticsException("Error while calculating Average:
> Value '" + value.toString() +
>  "', being aggregated is not
> numeric.");
> }
> }
>
>
What are the other details the RecordValueContext has? Is it only having
the values for the fields to be aggregated? If so, I think we can simple
have Map for this as key, value pair right? I cannot think of a case where
the aggregate function is based some different field than the field which
needs to be aggregated.  Sorry, if I'm missing something..

Thanks,
Sinthuja.

>
>
>
> *public Object finish() throws AnalyticsException*
> Once the aggregation is complete for a group or for whole record set, this
> method will be called. This will return the final result of the aggregate
> function. for example, the SUM aggregate,
>
> public Number finish() throws AnalyticsException {
> return sum;
> }
>
>
>
> *public String getAggregateName()*
> This method is used to identify the Aggregate function uniquely among
> other Aggregate functions. You can return a preferred string inside this
> method.
>
> *Calling custom aggregate functions through Javascript API*
>
> var queryInfo = {
> tableName:"Students", //table name on which the aggregation is
> performed
> searchParams : {
> groupByField:"location", //grouping field if any
> query : "Grade:10" //additional filtering query
> aggregateFields:[
> {
> fields:["Height", "Weight"], //fields necessary for aggregate
> function
> aggregate:"CUSTOM_AGGREGATE", //unique name of the aggregate
> function, this is what we return using "getAggregateName" method above.
> alias:"aggregated_result" //Alias for the result of the
> aggregate function
> }]
> }
> }
>
> client.searchWithAggregates(queryInfo, function(data) {
>   console.log (data["message"]);
> }, function(error) {
>   console.log("error occured: " + error["message"]);
> });
>
>
> *Note that the order  elements in attribute "fields" will be the same
> order of aggregateFields parameter's element order in above process method.
> That is Height will be aggregateFields[0] and Weight will be
> aggregateFields[1] in process method. Based on that order,
> "CUSTOM_AGGREGATE" should be implemented.*
>
>
>
> *Aggregates REST APIs*This is as same as the Javascript API.
>
> POST https://localhost:9443/analytics/aggregates
> {
>  "tableName":"Students",
>  "groupByField":"location",
>  "aggregateFields":[
>{
>  "fields":["Height", "Weight"],
>  "aggregate":"CUSTOM_AGGREGATE",
>  "alias":"aggregated_result"
>}]
> }
>
> [1]
> https://docs.wso2.com/display/DAS301/Retrieving+Aggregated+Values+of+Given+Records+via+REST+API
> [2]
> https://docs.wso2.com/display/DAS301/Retrieving+Aggregated+Values+of+Given+Records+via+JS+API
> --
> Gimantha Bandara
> Software Engineer
> WSO2. Inc : http://wso2.com
> Mobile : +94714961919
>
> _

Re: [Architecture] Incremental Processing Support in DAS

[Sinthuja Ragendran]

Hi Sachith,

On Mon, Mar 21, 2016 at 5:15 PM, Sachith Withana  wrote:

> Hi all,
>
> We are adding incremental processing capability to Spark in DAS.
> As the first stage, we added time slicing to Spark execution.
>
> Here's a quick introduction into that.
>
> *Execution*:
>
> In the first run of the script, it will process all the data in the given
> table and store the last processed event timestamp.
> Then from the next run onwards it would start processing starting from
> that stored timestamp.
>
> Until the query that contains the data processing part, completes, last
> processed event timestamp would not be overridden with the new value.
> This is to ensure that the data processing for the query wouldn't have to
> be done again, if the whole query fails.
> This is ensured by adding a commit query after the main query.
> Refer to the Syntax section for the example.
>
> *Syntax*:
>
> In the spark script, incremental processing support has to be specified
> per table, this would happen in the create temporary table line.
>
> ex: CREATE TEMPORARY TABLE T1 USING CarbonAnalytics options (tableName
> "test",
> *incrementalProcessing "T1,3600");*
>

Why do we need T1 with  incrementalProcessing "T1,3600" ? Isn't having only
the time duration 3600 is enough given that this is coming under T1 carbon
analytics properties.

I believe with this, the user needs to create another temporary table if
the user wants to write incremental query for the same physical table. For
example, for the below table T1, if I want to do some different analytics
and insert the results back to table T3, then I have to create another
temporary table for test.

Is the temporary tables are defined globally? If I had a different script
and can I define the same temporary table name and pass some different
incremental processing configuration? IMHO, it should be local to the
specific script, not global as the users may not have track of all the
temporary table names they define in other scripts.

>
> INSERT INTO T2 SELECT username, age GROUP BY age FROM T1;
>
> INC_TABLE_COMMIT T1;
>

Do we need to have the another explicit query to submit the timestamp? It
may induce to have user errors if he/she forgets to include it. Can't we
commit the changes after the successful execution of the given actual
insert query?


>
> The last line is where it ensures the processing took place successfully
> and replaces the lastProcessed timestamp with the new one.
>
> *TimeWindow*:
>
> To do the incremental processing, the user has to provide the time window
> per which the data would be processed.
> In the above example. the data would be summarized in *1 hour *time
> windows.
>

Ie, it will process the data from current timestamp-3600  to current time
stamp? For example, lets say I have configured the incremental processing,
and 1 and 2nd run was successful, and my analyzer node is down for 3 hours
and my receiver is collecting the events still. So when I bring back my
analyzer node, will the next iteration considers the full data from where
it has processed last or it will consider only last 1 hour data?

Thanks,
Sinthuja.

>
> *WindowOffset*:
>
> Events might arrive late that would belong to a previous processed time
> window. To account to that, we have added an optional parameter that would
> allow to
> process immediately previous time windows as well ( acts like a buffer).
> ex: If this is set to 1, apart from the to-be-processed data, data related
> to the previously processed time window will also be taken for processing.
>

> *Limitations*:
>
> Currently, multiple time windows cannot be specified per temporary table
> in the same script.
> It would have to be done using different temporary tables.
>
>
>
> *Future Improvements:*
> - Add aggregation function support for incremental processing
>
>
> Thanks,
> Sachith
> --
> Sachith Withana
> Software Engineer; WSO2 Inc.; http://wso2.com
> E-mail: sachith AT wso2.com
> M: +94715518127
> Linked-In: 
> https://lk.linkedin.com/in/sachithwithana
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Associate Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Implementing proper security model for dashboard server

[Sinthuja Ragendran]

Hi Sumedha,

On Thu, Apr 28, 2016 at 1:58 PM, Sumedha Rubasinghe 
wrote:

> Geesara,
> This is a model that should be coming out of Dashboard perspective.
>
> If we take a look @ basic building blocks of DS, its (similar to what you
> have mentioned)
> - Gadget
> - Dashboard
> - Wizards
> - Dashboard Admin panel
> - etc
>
> Each of these elements should have a permission model associated per
> instance.
>

Yeah +1, as per now the permission checks are not implemented for these
operations, but we need to incorporate that as well.


> For example, defining "permission to view any gadget" is not enough.  But
> rather it should be "permission to view Gadget X".
> Same should be extended for all other building blocks. (AFAIK, this is not
> there for gadgets as of now)
>
> These need to be stored @ gadget server level and evaluated b4 rendering
> any gadget.
>

Yeah, actually we are planning to implement the role based access control
for gadgets and then again different views of the dashboard page based on
roles.


>
> Permissions to BE
> 
> Once presentation layer permissions are sorted, it becomes responsibility
> of Gadget / Dashboard author to figure out mapping
> those permissions to a backend API.
>
> There are multiple ways to do this based on how backend is secured.
> - Passing session cookie obtained @ login to backend
> - Obtaining a token (OAuth2) using session cooking (using an OAuth2 grant
> type)
> - If SSO is enabled, obtaining a token (OAuth2) using SAML Token
> - IdP enabled deployment
>
> Ensuring backend API's permission requirements match front end user's
> privileges is part of author's
> responsibility. This is not something DS product needs to worry about.
>

Exactly, but I think there should be some API provided by DS server
(shindig features), so that the users can just call the necessary methods
with respective parameters to get oauth token. WDYT?

Thanks,
Sinthuja.


> If by any chance backend is written using WSO2 technologies, we can
> leverage concepts like
> - Sharing same identity provider for both DS and BE server
> - passing authorisation details from FE to BE using JWT/SAML Response /
> User profile
>
>
> Permissions when gadgets being embedded into other products without
> dashboard
> 
> This is yet another perspective of the same problem. This also can be
> solved if we follow same principles
> mentioned above.
> - Having gadget instance level permission definition
> - Way to obtain a gadget by passing in authorisation details (using one of
> the methods mentions above)
>

> Same applies for dashboards.
>
>
> On Thu, Apr 28, 2016 at 1:00 AM, Geesara Prathap  wrote:
>
>> *Requirement:*
>> *When dashboard retrieving data from some REST APIs which are secured, we
>> do require proper security model in place in order to identify who can
>> access this dashboard and at which level should it be done. In addition,how
>> can dashboard be going to communicate with respective REST API securely?*
>>
>>
>>
>>  Figure 01:
>> Dashboard Server
>>
>>
>> Data providers for gadgets need to communicate with DS securely. Most of
>> the cases data providers are some REST APIs. There might be a situation
>> which dashboard will be getting data from different data providers as well.
>> In the DS perspective, there must be an effective way to tackle these
>> security related issues up to some extent. Referring to figure 1, we are
>> having three places where we can address these issues.
>>
>>- gadget level
>>- per-dashboard level
>>- dashboard server level
>>
>> What would be the proper place which we can address security concerns in
>> a proper manner?  If we try to address this at gadget level, It will be too
>> mush of  granularity which may be preventing the acceptable performance of
>> data retrieval from data providers as well as too mush of load to DS
>> itself.  Also having problems user authentication and authorization at this
>> level as well as per dashboard level. Dashboard server level would be the
>> ideal place which we can address all those  security concerns in a
>> conventional manner. Any advice and suggestions will be greatly appreciated
>> regarding this.
>>
>> Thanks,
>> Geesara,
>>
>> --
>> Geesara Prathap Kulathunga
>> Software Engineer
>> WSO2 Inc; http://wso2.com
>> Mobile : +940772684174
>>
>>
>
>
> --
> /sumedha
> m: +94 773017743
> b :  bit.ly/sumedha
>



-- 
*Sinthuja Rajendran*
Associate Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Data Bridge Agent Publisher for C5 products

[Sinthuja Ragendran]

Hi IsuruP,

Please find the comments inline.

On Thu, Jun 2, 2016 at 12:18 PM, Isuru Perera  wrote:

> Hi,
>
> This is regarding $subject and the main problem we have is that there is
> no Carbon 5 compatible feature for data bridge agent.
>

Data bridge agent publisher is not depends on carbon, and it has some
dependencies for carbon utils, and carbon base, which we can eliminate by
passing proper configurations in data-agent-config.xml. In that case, what
do you mean by it's not compatible by Carbon 5 as it's anyhow doesn't
depend on carbon features?


>
> Since Data Bridge Agent is already an OSGi bundle, we can use it within C5
> products. But we have to include it with some feature.
>
> For example, Carbon Metrics needs to publish events to DAS. So, is it okay
> if I keep data bridge agent in Metrics feature?
>

No, I don't think that is a good option. Because the publisher is a generic
feature, and it doesn't have any relation ship to metrics feature other
then metrics feature is using data publisher feature. In that case, you
need to have just importFeature defn for the datapublisher feature form the
metrics feature.


>
> Other problem is that the current Data Bridge Agent is written for Carbon
> 4.x based products. For example it uses CarbonUtils to find the location of
> data-agent-config.xml. The CarbonUtils class used by the agent is only
> available in C4.
>
> We can avoid this by giving a configuration file to the agent. Then there
> will be no NoClassDefFoundError. However as the next step, the agent
> requires the client-trustore.jks and the password to the truststore.
>
> How should we give the trust store configuration in Carbon 5? For now, we
> use the system properties: javax.net.ssl.trustStore and
> javax.net.ssl.trustStorePassword.
>

I think in C5 also the above mentioned system properties will be exposed,
anyhow I'm not sure though. Anyhow you can set above configuration from
data-agent-config.xml as well, so you may have it in the static location
now.

Thanks,
Sinthuja.


> The Message Broker is trying to use Carbon Metrics for analytics and we
> need to know what are the recommendations for using Data Bridge Agent in C5.
>

> Thanks,
>
> Best Regards,
>
> --
> Isuru Perera
> Associate Technical Lead | WSO2, Inc. | http://wso2.com/
> Lean . Enterprise . Middleware
>
> about.me/chrishantha
> Contact: +IsuruPereraWSO2 
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Associate Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Data Bridge Agent Publisher for C5 products

[Sinthuja Ragendran]

Hi,


On Thu, Jun 2, 2016 at 3:18 PM, Sriskandarajah Suhothayan 
wrote:

> Are we going to use data bridge in C5 ?
> C5 have Netty based transports, cant we use one of them to publish events
> to DAS, since DAS have the capability to receiving from any transport
> protocol via extensions this will not be a problem.
>
> In my opinion we should not be depending on data publisher as it have
> several issues like events can get out of ordered, dropped and its not
> reliable. We should have a publishing framework which is independent of the
> transport, so users can pic Thrift, HTTP or AMQP based on their use cases.
>
>
+1. I believe there will be a generic extension point in the C5 for
receiver and sender implementations, and DAS also should use those receiver
implementations to receive the events, instead of having its own receivers.
In that case, the thrift also in just an additional protocol where we need
to write the sender and receiver for that.

Thanks,
Sinthuja.


> WDYT?
>
> Suho
>
> On Thu, Jun 2, 2016 at 2:35 PM, Kishanthan Thangarajah <
> kishant...@wso2.com> wrote:
>
>> A separate point to note.
>>
>> Thinking along the AS 6.0 and C5 aspect, what we need is a library, where
>> we could use that in both OSGi env and non-OSGi environments. It should not
>> have any direct dependency on carbon API's. Currently, with AS 6.0, we are
>> using the data publisher from C4 without any code changes and removing
>> unwanted dependencies. If we planning to write this again, we should come
>> up with the minimum library that could be used in a OSGi and non-OSGi env.
>>
>>
>> On Thu, Jun 2, 2016 at 1:40 PM, Isuru Perera  wrote:
>>
>>> Hi,
>>>
>>> On Thu, Jun 2, 2016 at 1:10 PM, Sinthuja Ragendran 
>>> wrote:
>>>
>>>> Hi IsuruP,
>>>>
>>>> Please find the comments inline.
>>>>
>>>> On Thu, Jun 2, 2016 at 12:18 PM, Isuru Perera  wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> This is regarding $subject and the main problem we have is that there
>>>>> is no Carbon 5 compatible feature for data bridge agent.
>>>>>
>>>>
>>>> Data bridge agent publisher is not depends on carbon, and it has some
>>>> dependencies for carbon utils, and carbon base, which we can eliminate by
>>>> passing proper configurations in data-agent-config.xml.
>>>>
>>> Yes. This is what should be done.
>>>
>>> I think we should avoid carbon dependencies in data publisher. We can
>>> have some Carbon specific component to initialize data publisher in Carbon
>>> (OSGi) environment.
>>>
>>>> In that case, what do you mean by it's not compatible by Carbon 5 as
>>>> it's anyhow doesn't depend on carbon features?
>>>>
>>> As I mentioned earlier, the publisher has Carbon 4.x dependencies. So,
>>> we need to workaround problems like NoClassDefFoundError for
>>> CarbonUtils etc.
>>>
>>>>
>>>>
>>>>>
>>>>> Since Data Bridge Agent is already an OSGi bundle, we can use it
>>>>> within C5 products. But we have to include it with some feature.
>>>>>
>>>>> For example, Carbon Metrics needs to publish events to DAS. So, is it
>>>>> okay if I keep data bridge agent in Metrics feature?
>>>>>
>>>>
>>>> No, I don't think that is a good option. Because the publisher is a
>>>> generic feature, and it doesn't have any relation ship to metrics feature
>>>> other then metrics feature is using data publisher feature. In that case,
>>>> you need to have just importFeature defn for the datapublisher feature form
>>>> the metrics feature.
>>>>
>>> Yes. The correct way is to import data publisher feature. However there
>>> is no such feature available now. Since Metrics needs the publisher
>>> dependencies, I thought we can include those dependencies until we have a
>>> data publisher designed to work with C5. When someone wants to install
>>> metrics feature, it should work without any issue. Right now, I cannot do a
>>> release of Carbon Metrics till I have answers to the questions raised in
>>> this mail thread.
>>>
>>>>
>>>>
>>>>>
>>>>> Other problem is that the current Data Bridge Agent is written for
>>>>> Carbon 4.x based products. For example it uses CarbonUtils to find the
>>>>> location of

Re: [Architecture] Access Level Model For WSO2 Dashboard Server

[Sinthuja Ragendran]

According to the chat I had with Johann, we have came up with this model.
Basically as per now in C4 we don't have a way to add a permission
dynamically, and similar requirement has been raised in IS as well and they
have come up with this model.

The main reason for this is, there might be some dashboard that you are
working on and want to share between some users and grant some privileges.
It's not doable with global permission, bcz having an edit permission for
say sales dashboard doesn't mean you have the permission for admin
dashboard. Therefore this needs to be controlled per dashboard, not
globally.

Thanks,
Sinthuja.

On Tue, Jun 7, 2016 at 11:12 AM, Sriskandarajah Suhothayan 
wrote:

> Why are we not using different permissions for each dashboard than using
>  roles. I believe using permissions will be more scalable than using roles.
> WDYT?
>
> Regards
> Suho
>
> On Tue, Jun 7, 2016 at 2:38 PM, Nisala Nanayakkara 
> wrote:
>
>> Hi Udara,
>>
>> Since these are internal roles, they are not stored in LDAP. So it will
>> work fine.
>>
>> Thanks,
>> Nisala
>>
>> On Tue, Jun 7, 2016 at 10:57 AM, Udara Rathnayake 
>> wrote:
>>
>>> Another question, ​Is this going to work if we have to connect to a
>>> read-only LDAP/A
>>> ​D​
>>> userstore?
>>>
>>> On Tue, Jun 7, 2016 at 9:43 AM, Tanya Madurapperuma 
>>> wrote:
>>>
 Is this model scalable? Because per dashboard we will have to create 4
 internal roles. So if we have N number of dashboards we will end up having
 4 * N number of internal roles.

 @ IS team : is this approach fine? Or is there any better approach?

 Thanks,
 Tanya

 On Mon, Jun 6, 2016 at 3:44 PM, Nisala Nanayakkara 
 wrote:

> adding Johan and Manuranga
>
> Thanks,
> Nisala
>
> On Mon, Jun 6, 2016 at 3:41 PM, Nisala Nanayakkara 
> wrote:
>
>> Hi all,
>>
>> I am working on implementing an access levels model for WSO2
>> Dashboard Server. Currently global permission model for 
>> create/delete/login
>> is implemented by Megala. Since it does not support to provide per
>> dashboard level access for the users. I am going to extend it and 
>> implement
>> a permission model that can be used to provide per dashboard level access
>> for the users.
>>
>> In order to implement this feature, I am going to add four roles at
>> dashboard creation time as follows,
>>
>>- internal/dashboard/{dashboardID}/editor
>>- internal/dashboard/{dashboardID}/viewer
>>- internal/dashboard/{dashboardID}/settings
>>- internal/dashboard/{dashboardID}/delete
>>
>> At the dashboard creation time, the user who creates the dashboard
>> will get all the four roles. But other users have to get above roles to 
>> do
>> appropriate actions to the dashboard. So that we can set above four roles
>> for the users and They will be given different access levels according to
>> their roles.
>>
>> Please feel free to give any feedback.
>>
>> Thanks,
>> Nisala
>> --
>> *Nisala Niroshana Nanayakkara,*
>> Software Engineer
>> Mobile:(+94)717600022
>> WSO2 Inc., http://wso2.com/
>>
>
>
>
> --
> *Nisala Niroshana Nanayakkara,*
> Software Engineer
> Mobile:(+94)717600022
> WSO2 Inc., http://wso2.com/
>



 --
 Tanya Madurapperuma

 Senior Software Engineer,
 WSO2 Inc. : wso2.com
 Mobile : +94718184439
 Blog : http://tanyamadurapperuma.blogspot.com

>>>
>>>
>>>
>>> --
>>> Regards,
>>> UdaraR
>>>
>>
>>
>>
>> --
>> *Nisala Niroshana Nanayakkara,*
>> Software Engineer
>> Mobile:(+94)717600022
>> WSO2 Inc., http://wso2.com/
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
>
> *S. Suhothayan*
> Technical Lead & Team Lead of WSO2 Complex Event Processor
> *WSO2 Inc. *http://wso2.com
> * *
> lean . enterprise . middleware
>
>
> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog:
> http://suhothayan.blogspot.com/ twitter:
> http://twitter.com/suhothayan  | linked-in:
> http://lk.linkedin.com/in/suhothayan *
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Feature - Applying a custom theme for a dashboard

[Sinthuja Ragendran]

Hi Jerad et all,

Let's use this approach if we want to apply some custom theme per
dashboard, without modifying the default theme. I know as we have merged
the theme you guys have developed for analytics effort as default theme, it
should not be a issue now. :) But in case, if you come across future
requirements, let's utilize this.

Thanks,
Sinthuja.

On Mon, Jun 6, 2016 at 6:34 AM, Prabushi Samarakoon 
wrote:

> Hi all,
>
> With the new release WSO2 DS 2.1.0 M2, we have introduced the support for
> applying a custom theme per dashboard in WSO2 Dashboard Server.
>
> User can deploy their custom themes as a Carbon Application Archive file
> through the admin console. One of these deployed themes can be selected as
> the dashboard theme, when the user create a dashboard. User can also change
> the selected theme, using dashboard settings configurations. Users are
> allowed to add a custom theme, only for the view of the dashboard. Users
> can add their own script files, template files and styles to customize the
> view of a particular dashboard.
>
> Further details about this feature can be found in the document attached
> here.
>
> Thanks,
> Prabushi
>
> --
> *Prabushi Samarakoon*
> Software Engineer
> Mobile: +94715434580
> Email: prabus...@wso2.com
>



-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Access Level Model For WSO2 Dashboard Server

[Sinthuja Ragendran]

Hi Manu,

On Tue, Jun 7, 2016 at 3:24 PM, Manuranga Perera  wrote:

> Following names make sense as roles:
>
>- internal/dashboard/{dashboardID}/editor
>- internal/dashboard/{dashboardID}/viewer
>
> but
>
>- internal/dashboard/{dashboardID}/settings
>- internal/dashboard/{dashboardID}/delete
>
> Sounds a bit weird for role names. As Suho said, it can be modeled as
> permissions as well.
>

If we want to model with the permissions then we should be able to add the
permissions dynamically, but this is not possible with current carbon -
4.x. And as I have mentioned above, this cannot be included in the global
level as well, because having a settings or delete privileges for dashboard
- X, doesn't mean you have the same privileges for dashboard - Y. And hence
we thought of going with roles approach for this one as well. I agree, the
role names for settings and delete is bit odd, we need to come up with
proper names for those. :)


>
> 1) Do you intend to add people to each via DS UI? (In that case roles
> might be easier)
>

The dashboard creator will be automatically assigned to all of the above
mentioned roles. If he/she want to share with some additional group/role,
he/she can add to the allowed roles list. In the case that if he/she wants
to share only with some users, they needs to be added to management UI. But
I agree, to have the proper flow we need to have UI in DS side for this.

2) Does "settings" make sense, because if you are an editor, anyway you'll
> have full access to the JSON, don't you?
>

In settings you have the full privileges, ie, you can even remove the user
who initially created the dashboard, IMHO it provides the full control of
the dashboard. Designer doesn't have such privileges, he/she can only
add/remove gadgets, pages etc which is related to designing the dashboard.
Therefore we need to have a different role to control the access of the
settings page.

Thanks,
Sinthuja.


> On Tue, Jun 7, 2016 at 7:59 AM, Sinthuja Ragendran 
> wrote:
>
>> According to the chat I had with Johann, we have came up with this model.
>> Basically as per now in C4 we don't have a way to add a permission
>> dynamically, and similar requirement has been raised in IS as well and they
>> have come up with this model.
>>
>> The main reason for this is, there might be some dashboard that you are
>> working on and want to share between some users and grant some privileges.
>> It's not doable with global permission, bcz having an edit permission for
>> say sales dashboard doesn't mean you have the permission for admin
>> dashboard. Therefore this needs to be controlled per dashboard, not
>> globally.
>>
>> Thanks,
>> Sinthuja.
>>
>> On Tue, Jun 7, 2016 at 11:12 AM, Sriskandarajah Suhothayan > > wrote:
>>
>>> Why are we not using different permissions for each dashboard than using
>>>  roles. I believe using permissions will be more scalable than using roles.
>>> WDYT?
>>>
>>> Regards
>>> Suho
>>>
>>> On Tue, Jun 7, 2016 at 2:38 PM, Nisala Nanayakkara 
>>> wrote:
>>>
>>>> Hi Udara,
>>>>
>>>> Since these are internal roles, they are not stored in LDAP. So it will
>>>> work fine.
>>>>
>>>> Thanks,
>>>> Nisala
>>>>
>>>> On Tue, Jun 7, 2016 at 10:57 AM, Udara Rathnayake 
>>>> wrote:
>>>>
>>>>> Another question, ​Is this going to work if we have to connect to a
>>>>> read-only LDAP/A
>>>>> ​D​
>>>>> userstore?
>>>>>
>>>>> On Tue, Jun 7, 2016 at 9:43 AM, Tanya Madurapperuma 
>>>>> wrote:
>>>>>
>>>>>> Is this model scalable? Because per dashboard we will have to create
>>>>>> 4 internal roles. So if we have N number of dashboards we will end up
>>>>>> having 4 * N number of internal roles.
>>>>>>
>>>>>> @ IS team : is this approach fine? Or is there any better approach?
>>>>>>
>>>>>> Thanks,
>>>>>> Tanya
>>>>>>
>>>>>> On Mon, Jun 6, 2016 at 3:44 PM, Nisala Nanayakkara 
>>>>>> wrote:
>>>>>>
>>>>>>> adding Johan and Manuranga
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Nisala
>>>>>>>
>>>>>>> On Mon, Jun 6, 2016 at 3:41 PM, Nisala Nanayakkara 
>>>>>>> wrote:
>>>>>>>
>>>>>>

Re: [Architecture] Feature - Applying a custom theme for a dashboard

[Sinthuja Ragendran]

Hi Jerad,

Ok great, I'll be not coming to office this and next week. @Prabushi,
please do schedule the meeting on this with Jerad and team, and get their
thoughts.

Thanks,
Sinthuja

On Wednesday, June 8, 2016, Jerad Rutnam  wrote:

> Hi Sinthuja,
>
> +1 for this initiation. This is great. And I had some concerns about the
> extension part of it. Let's schedule a review meeting, so we can share our
> thoughts. :D
>
> Cheers!
> Jerad
>
>
> On Tue, Jun 7, 2016 at 6:51 PM, Sinthuja Ragendran  > wrote:
>
>> Hi Jerad et all,
>>
>> Let's use this approach if we want to apply some custom theme per
>> dashboard, without modifying the default theme. I know as we have merged
>> the theme you guys have developed for analytics effort as default theme, it
>> should not be a issue now. :) But in case, if you come across future
>> requirements, let's utilize this.
>>
>> Thanks,
>> Sinthuja.
>>
>> On Mon, Jun 6, 2016 at 6:34 AM, Prabushi Samarakoon > > wrote:
>>
>>> Hi all,
>>>
>>> With the new release WSO2 DS 2.1.0 M2, we have introduced the support
>>> for applying a custom theme per dashboard in WSO2 Dashboard Server.
>>>
>>> User can deploy their custom themes as a Carbon Application Archive file
>>> through the admin console. One of these deployed themes can be selected as
>>> the dashboard theme, when the user create a dashboard. User can also change
>>> the selected theme, using dashboard settings configurations. Users are
>>> allowed to add a custom theme, only for the view of the dashboard. Users
>>> can add their own script files, template files and styles to customize the
>>> view of a particular dashboard.
>>>
>>> Further details about this feature can be found in the document attached
>>> here.
>>>
>>> Thanks,
>>> Prabushi
>>>
>>> --
>>> *Prabushi Samarakoon*
>>> Software Engineer
>>> Mobile: +94715434580
>>> Email: prabus...@wso2.com
>>> 
>>>
>>
>>
>>
>> --
>> *Sinthuja Rajendran*
>> Technical Lead
>> WSO2, Inc.:http://wso2.com
>>
>> Blog: http://sinthu-rajan.blogspot.com/
>> Mobile: +94774273955
>>
>>
>>
>
>
> --
> *Jerad Rutnam*
> *Software Engineer*
>
> WSO2 Inc.
> lean | enterprise | middleware
> M : +94 77 959 1609 | E : je...@wso2.com
>  | W : www.wso2.com
>


-- 
Sent from iPhone
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Feature - Applying a custom theme for a dashboard

[Sinthuja Ragendran]

Hi Dunith,

On Wed, Jun 8, 2016 at 9:54 AM, Dunith Dhanushka  wrote:

> Hi Prabushi,
>
> +1 for this initiation.
>
> Let's say I have developed a custom theme and a dashboard. Now I want to
> pack this into a CApp and deploy into another PC. In that case I'll have to
> pack dashboard definition JSON + theme into the CApp.
>
> My question is how can I specify the name of the custom theme in the
> dashboard definition? Will adding "theme" attribute work?
>

Yes, that will work. And not necessarily you need to pack the theme that
you have mentioned in the dashboard.json along with the car file, if the
theme is already there in the pack or the environment. Ie, lets say an
organization has added a theme default in their distribution, then you can
just mention in the dashboard.json and use it.

Thanks,
Sinthuja.

>
> Thanks,
> Dunith
>
> On Wed, Jun 8, 2016 at 2:10 PM, Sinthuja Ragendran 
> wrote:
>
>> Hi Jerad,
>>
>> Ok great, I'll be not coming to office this and next week. @Prabushi,
>> please do schedule the meeting on this with Jerad and team, and get their
>> thoughts.
>>
>> Thanks,
>> Sinthuja
>>
>>
>> On Wednesday, June 8, 2016, Jerad Rutnam  wrote:
>>
>>> Hi Sinthuja,
>>>
>>> +1 for this initiation. This is great. And I had some concerns about the
>>> extension part of it. Let's schedule a review meeting, so we can share our
>>> thoughts. :D
>>>
>>> Cheers!
>>> Jerad
>>>
>>>
>>> On Tue, Jun 7, 2016 at 6:51 PM, Sinthuja Ragendran 
>>> wrote:
>>>
>>>> Hi Jerad et all,
>>>>
>>>> Let's use this approach if we want to apply some custom theme per
>>>> dashboard, without modifying the default theme. I know as we have merged
>>>> the theme you guys have developed for analytics effort as default theme, it
>>>> should not be a issue now. :) But in case, if you come across future
>>>> requirements, let's utilize this.
>>>>
>>>> Thanks,
>>>> Sinthuja.
>>>>
>>>> On Mon, Jun 6, 2016 at 6:34 AM, Prabushi Samarakoon >>> > wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> With the new release WSO2 DS 2.1.0 M2, we have introduced the support
>>>>> for applying a custom theme per dashboard in WSO2 Dashboard Server.
>>>>>
>>>>> User can deploy their custom themes as a Carbon Application Archive
>>>>> file through the admin console. One of these deployed themes can be
>>>>> selected as the dashboard theme, when the user create a dashboard. User 
>>>>> can
>>>>> also change the selected theme, using dashboard settings configurations.
>>>>> Users are allowed to add a custom theme, only for the view of the
>>>>> dashboard. Users can add their own script files, template files and styles
>>>>> to customize the view of a particular dashboard.
>>>>>
>>>>> Further details about this feature can be found in the document
>>>>> attached here.
>>>>>
>>>>> Thanks,
>>>>> Prabushi
>>>>>
>>>>> --
>>>>> *Prabushi Samarakoon*
>>>>> Software Engineer
>>>>> Mobile: +94715434580
>>>>> Email: prabus...@wso2.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Sinthuja Rajendran*
>>>> Technical Lead
>>>> WSO2, Inc.:http://wso2.com
>>>>
>>>> Blog: http://sinthu-rajan.blogspot.com/
>>>> Mobile: +94774273955
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> *Jerad Rutnam*
>>> *Software Engineer*
>>>
>>> WSO2 Inc.
>>> lean | enterprise | middleware
>>> M : +94 77 959 1609 | E : je...@wso2.com | W : www.wso2.com
>>>
>>
>>
>> --
>> Sent from iPhone
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "WSO2 Engineering Group" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to engineering-group+unsubscr...@wso2.com.
>> For more options, visit https://groups.google.com/a/wso2.com/d/optout.
>>
>
>
>
> --
> Regards,
>
> Dunith Dhanushka,
> Associate Technical Lead
> WSO2 Inc,
>
> Mobile - +94 71 8615744
> Blog - *https://medium.com/@dunithd <https://medium.com/@dunithd>*
> Twitter - @dunithd <http://twitter.com/dunithd>
>



-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Feature - Applying a custom theme for a dashboard

[Sinthuja Ragendran]

Hi Prabushi,

On Wed, Jun 8, 2016 at 10:57 AM, Prabushi Samarakoon 
wrote:

> Hi,
>
> @Sinthuja,
> We had an offline discussion with Jerad, and it seems the proposed theme
> extension part wont be needed for our immediate release. So shall we
> schedule a meeting after you come back here?
>
>
Can you briefly explain what actually proposed? Yeah, we may not able
include all in the current release, so i'm +1 to push it after two weeks.

Thanks,
Sinthuja.


> @Dunith,
> There is a 'theme' attribute already defined in the dashboard json file.
> You can specify the custom theme there and use it.
>
> Thanks and Regards,
> Prabushi
>
> On Wed, Jun 8, 2016 at 2:24 PM, Dunith Dhanushka  wrote:
>
>> Hi Prabushi,
>>
>> +1 for this initiation.
>>
>> Let's say I have developed a custom theme and a dashboard. Now I want to
>> pack this into a CApp and deploy into another PC. In that case I'll have to
>> pack dashboard definition JSON + theme into the CApp.
>>
>> My question is how can I specify the name of the custom theme in the
>> dashboard definition? Will adding "theme" attribute work?
>>
>> Thanks,
>> Dunith
>>
>> On Wed, Jun 8, 2016 at 2:10 PM, Sinthuja Ragendran 
>> wrote:
>>
>>> Hi Jerad,
>>>
>>> Ok great, I'll be not coming to office this and next week. @Prabushi,
>>> please do schedule the meeting on this with Jerad and team, and get their
>>> thoughts.
>>>
>>> Thanks,
>>> Sinthuja
>>>
>>>
>>> On Wednesday, June 8, 2016, Jerad Rutnam  wrote:
>>>
>>>> Hi Sinthuja,
>>>>
>>>> +1 for this initiation. This is great. And I had some concerns about
>>>> the extension part of it. Let's schedule a review meeting, so we can share
>>>> our thoughts. :D
>>>>
>>>> Cheers!
>>>> Jerad
>>>>
>>>>
>>>> On Tue, Jun 7, 2016 at 6:51 PM, Sinthuja Ragendran 
>>>> wrote:
>>>>
>>>>> Hi Jerad et all,
>>>>>
>>>>> Let's use this approach if we want to apply some custom theme per
>>>>> dashboard, without modifying the default theme. I know as we have merged
>>>>> the theme you guys have developed for analytics effort as default theme, 
>>>>> it
>>>>> should not be a issue now. :) But in case, if you come across future
>>>>> requirements, let's utilize this.
>>>>>
>>>>> Thanks,
>>>>> Sinthuja.
>>>>>
>>>>> On Mon, Jun 6, 2016 at 6:34 AM, Prabushi Samarakoon <
>>>>> prabus...@wso2.com> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> With the new release WSO2 DS 2.1.0 M2, we have introduced the support
>>>>>> for applying a custom theme per dashboard in WSO2 Dashboard Server.
>>>>>>
>>>>>> User can deploy their custom themes as a Carbon Application Archive
>>>>>> file through the admin console. One of these deployed themes can be
>>>>>> selected as the dashboard theme, when the user create a dashboard. User 
>>>>>> can
>>>>>> also change the selected theme, using dashboard settings configurations.
>>>>>> Users are allowed to add a custom theme, only for the view of the
>>>>>> dashboard. Users can add their own script files, template files and 
>>>>>> styles
>>>>>> to customize the view of a particular dashboard.
>>>>>>
>>>>>> Further details about this feature can be found in the document
>>>>>> attached here.
>>>>>>
>>>>>> Thanks,
>>>>>> Prabushi
>>>>>>
>>>>>> --
>>>>>> *Prabushi Samarakoon*
>>>>>> Software Engineer
>>>>>> Mobile: +94715434580
>>>>>> Email: prabus...@wso2.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Sinthuja Rajendran*
>>>>> Technical Lead
>>>>> WSO2, Inc.:http://wso2.com
>>>>>
>>>>> Blog: http://sinthu-rajan.blogspot.com/
>>>>> Mobile: +94774273955
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Jerad Rutnam*
>>>> *

Re: [Architecture] Access Level Model For WSO2 Dashboard Server

[Sinthuja Ragendran]

Hi Nisala,

On Thu, Jun 9, 2016 at 8:13 AM, Nisala Nanayakkara  wrote:

> Hi Sinthuja,
>
> This email is to clarify several issues regarding this feature. Up-to now
> I have created a four internal roles as above. All these four roles are
> assigned to the user who created the dashboard initially. If we want to
> give specific permission to another user, we can assign appropriate role to
> that user. As an example if we want to give access to the settings page to
> a user, we can assign appropriate role to that user.
>
> But lets think that we have to give access to settings page of dashboard X
> for all the users who have role A. Then how can we achieve that use-case
> here ?
>

The operations of the dashboard will be assigned initially with the above
mentioned  roles, but you should be able include other roles also for that
operation as same as we have now currently. And hence it's just adding
roleA also along with internal/dashboardA/viewer role for example.

> Are we going to the add UI configuration in settings page as for editor
> and viewer ?
>

Yes, there should be separate options to assign and remove roles per
operations.

> Otherwise we have to go through all the users who have role A and assign
> them with the dashboard X settings role using carbon management console.
>
> Another suggestion from me, Shall we create a single role called 'owner'
> by merging settings role and delete role as manuranga mentioned ?
>

+1.

Thanks,
Sinthuja.

>
> Thanks,
> Nisala
>
> On Tue, Jun 7, 2016 at 9:15 PM, Manuranga Perera  wrote:
>
>> If we want to model with the permissions then we should be able to add
>>> the permissions dynamically, but this is not possible with current carbon -
>>> 4.x. And as I have mentioned above, this cannot be included in the global
>>> level as well, because having a settings or delete privileges for dashboard
>>> - X, doesn't mean you have the same privileges for dashboard - Y. And hence
>>> we thought of going with roles approach for this one as well. I agree, the
>>> role names for settings and delete is bit odd, we need to come up with
>>> proper names for those. :)
>>>
>>
>> I think it is possible to dynamically create any permissions via the API
>> even in C4
>>
>> 2) Does "settings" make sense, because if you are an editor, anyway
 you'll have full access to the JSON, don't you?
>>>
>>> In settings you have the full privileges, ie, you can even remove the
>>> user who initially created the dashboard, IMHO it provides the full control
>>> of the dashboard. Designer doesn't have such privileges, he/she can only
>>> add/remove gadgets, pages etc which is related to designing the dashboard.
>>> Therefore we need to have a different role to control the access of the
>>> settings page.
>>
>>
>> We may call this "Owners"?
>>
>> --
>> With regards,
>> *Manu*ranga Perera.
>>
>> phone : 071 7 70 20 50
>> mail : m...@wso2.com
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> *Nisala Niroshana Nanayakkara,*
> Software Engineer
> Mobile:(+94)717600022
> WSO2 Inc., http://wso2.com/
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Modification of the gadget generation wizard's extension structure

[Sinthuja Ragendran]

Hi Manu,

On Wed, Jun 8, 2016 at 8:40 PM, Manuranga Perera  wrote:

> 1) We had a chat about how we can use gadget parameters instead of
>>> generation, have you guys considered that approach?
>>>
>>> We have cases like database credentials which should not be shown to the
>> user and should not be editable from the gadget properties. Further we also
>> need a model that gadget should be greeted by some privileged user and used
>> by others, so the gadget generation is the appropriate model to handle
>> this.
>>
>  I thought we can come up with a way to attach permissions to the params,
> so they can be hidden depending on the user. (this will help us generally
> not just for gadget gen case)
>

Yeah, initially i also thought we can go with such an approach. But then
there are some dynamic params that you need to populate based on the
previous selection that you have done. Ie, for example if it's batch
data/realtime/rdbms then the form params that you need to display is
different for each. If we are going to have such capabilities then it's
equal to have wizard apporach, and I don't really see much difference.

Anyhow our objective was to create a framework, where you can plug any data
providers and chart types easily and generate the gadget. With this we also
have removed the dependency of analytics gadget generation part from the
carbon-dashboards repo, and people can have their own providers and chart
types, in their repo and they can include that in the build time to get
only into their product.

Thanks,
Sinthuja.

Edit/ Re-generate function is not supported yet.
>>>
>>> 2) The issues is, with this model it will be harder to support that even
>>> in the future. At least we should serialize all the parameters with the
>>> generated gadget.
>>>
>> Editing is something we can achieve with very little effort, if we store
>> all the properties that we have passed in the UI in a config file within
>> the gadget then we can simply load that to the gadget generation wizard
>> when we need to modify the gadget.
>>
> Ya, that's what I also meant by serialize. But in that case the UX is
> broken compared to the parameter method, since the user still has to go
> though the wizard again, and you can't see the values from the left panel.
>
> --
> With regards,
> *Manu*ranga Perera.
>
> phone : 071 7 70 20 50
> mail : m...@wso2.com
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [DS]Embeddable gadgets feature for Dashboard Server

[Sinthuja Ragendran]

Hi all,

So far based on all the discussions above I see below facts.

1) APIM log viewer app is using portal as 'Gadget Store', and wants to
embed the gadgets which are residing in the portal app.
2) Without any additional authentication process, the user should be able
to embed the gadget to APIM log viewer app.
3) APIM admin app is using the js, libs, etc from the portal app into their
app, by just importing the files. Also currently you have shiding feature
in the server.

Based on above facts, IMO DS is not correctly exploited here. Basically
portal is NOT a gadget store, and we have the in built gadget store to
cater the main functionality of the DS which is creating dashboards and
managing them. Hence if APIM just want to use gadget a place to store
gadget and render it, then it's not a best place to do so. As you already
have shiding feature in your pack, I would suggest that you just add your
gadgets somewhere in APIM log viewer app it self, and render it. That will
not require and authentication nor perf issues.

Additionally you can only include the js files you required into the APIM
log viewer app it self, and remove the portal app entirely from APIM pack.
Because importing the js files from another application, is a wrong
practice, and we can only make sure we don't change the external APIs not
the internal js files, etc, and that will heavily break APIM log viewer,
which will be a maintenance night mare.

As embedded gadget functionality for DS, we need to embed the gadget
including the settings which have been done to the DS as it's shown in the
DS server (such as permissions to page, gadgets, views, themes at DS, and
gadget settings). Also as we are working on several level of authorization
for gadgets, pages, and views, logged in user information is mandatory and
user should be specifically say which actually he/she want to embed into
their app as the same gadget may have different settings in each
pages/views. Therefore we need the logged in user information, and for that
as I have already mentioned we support basic auth and SSO, which will cover
the third party use cases . Other than that we will include an loading icon
than showing the SSO redirection for this which will provide better user
experience. But anyhow, this is for the sake of DS embeddable gadget
feature, and not for APIM log viewer usecase which is simply embedding a
gadget into their app. IMO the APIM log viewer usecase, it's far more
easier to just maintain the gadgets in their app itself as mentioned above
without actually using the portal app, because anyhow none of the DS main
functionalities not used in the intended way.

Thanks,
Sinthuja.

On Tue, Jun 14, 2016 at 8:29 PM, Manuranga Perera  wrote:

> There are a few ways to get rid of iframe:
> 1) Ignore security and come up with a way to just embed as divs. But this
> may break the parent
> 2) Use Google Caja. We have to feather look into this. It seems they have
> some compatibility with Shindig as well.
> 3) Use WebComponents. This approach needs significant research and coming
> up with new ways to leverage latest browser specs.
> --
> With regards,
> *Manu*ranga Perera.
>
> phone : 071 7 70 20 50
> mail : m...@wso2.com
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [Analytics] Allowing analytics data to be published to super tenant space

[Sinthuja Ragendran]

Hi,

On Thu, Jun 23, 2016 at 5:48 PM, Amila Maha Arachchi 
wrote:

>
>
> On Thu, Jun 23, 2016 at 4:22 PM, Dunith Dhanushka  wrote:
>
>>
>>
>> On Thu, Jun 23, 2016 at 3:46 PM, Anjana Fernando  wrote:
>>
>>> Hi Amila,
>>>
>>> On Thu, Jun 23, 2016 at 11:52 AM, Amila Maha Arachchi 
>>> wrote:
>>>
 All,

 1. We should allow to decide whether to publish data in super tenant
 mode or tenant mode

>>>
>>> This is possible, but the problem is, it complicates the ESB analytics
>>> solution, where we will have to maintain two different versions which would
>>> implement the two scenarios. So IMO, it would be better to follow a single
>>> approach which would be overall best flexibility, which we discussed
>>> earlier, where we publish and manage data in tenants, but execute the Spark
>>> scripts in the super tenant.
>>>
>>>
 2. If its the ST mode, we deploy the car in ST space.
 3. Data gets published and stored in one table. i.e. no table per tenant

>>>
>>> The current connectors, e.g. RDBMS / HBase etc.. use the mechanism of
>>> creating a table per analytics table/tenant. In those connectors, this
>>> behavior cannot be changed, where mainly there are technical difficulties
>>> in doing so also, when filtering out different tenant's data and all.
>>> Anyways, usually in database systems, there is no limit in the number of
>>> physical tables created and all. And also, you will not access these tables
>>> directly, but will communicate via the REST APIs if required.
>>>
>>>
 4. Spark runs against that table

>>>
>>> With the new improvements, we anyway get similar type of an interface
>>> where the Spark script will automatically read data from all the tenants
>>> and process the data in one go.
>>>
>>>
 5. Dashboard should be a SaaS app which filters data from the analyzed
 table.

>>>
>>> I guess that maybe possible, but that will need some changes in the
>>> current dashboards. @Dunith, can you comment on this please. Also, is there
>>> any issue in deploying a dashboard per tenant?, which is the current
>>> situation.
>>>
>>
>>  Hi Anjana,
>> Currently dashboards and other related artifacts are stored in tenant
>> aware manner.
>>
>> Dashboard definitions are stored in the logged in user's tenant registry
>> so that it'll be isolated only to that tenant. Gadget's read data through
>> analytics.jag which in turn filters data by looking at logged in user's
>> tenant domain.
>>
>> So I believe current implementation statisfies the requirements of a SaaS
>> app.
>>
>
>
We do have option to share the dashboard from super tenant to other tenants
in the DS server [1] (it will be available in next milestone which will be
done tomorrow). Anyhow the gadgets and APIs need to be supported to get the
tenant information and display accordingly, but IMHO that will be better
option in this usecase. Instead of copying the dashboard for other tenants.

[1] mail thread - [Architecture] Sharing Dashboards that have gadgets with
role restriction

Thanks,
Sinthuja


> Can you add this to the roadmap for the next release please :). I see this
> will end up with another migration though :(
>
>>
>>> Cheers,
>>> Anjana.
>>>
>>>

 Can above be facilitated?

 Regards,
 Amila.

 --
 *Amila Maharachchi*
 Senior Technical Lead
 WSO2, Inc.; http://wso2.com

 Blog: http://maharachchi.blogspot.com
 Mobile: +94719371446


>>>
>>>
>>> --
>>> *Anjana Fernando*
>>> Senior Technical Lead
>>> WSO2 Inc. | http://wso2.com
>>> lean . enterprise . middleware
>>>
>>
>>
>>
>> --
>> Regards,
>>
>> Dunith Dhanushka,
>> Associate Technical Lead
>> WSO2 Inc,
>>
>> Mobile - +94 71 8615744
>> Blog - *https://medium.com/@dunithd *
>> Twitter - @dunithd 
>>
>
>
>
> --
> *Amila Maharachchi*
> Senior Technical Lead
> WSO2, Inc.; http://wso2.com
>
> Blog: http://maharachchi.blogspot.com
> Mobile: +94719371446
>
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

[Architecture] WSO2 Dashboard Server 2.1.0-M3 Released!

[Sinthuja Ragendran]

WSO2 Dashboard Server 2.1.0-M3 Released!

WSO2 Dashboard Server team is pleased to announce the release of WSO2
Dashboard Server 2.1.0 Milestone 3. This release comes with the following
bug fixes and features. You can download the distribution from [1].
Bug

[UES-924] - In drill down dashboards (pub-sub model) if subscriber gadget
get maximized it gets redrawn in default state
[UES-933] - Failed to load resource error when loading usa-social gadget
[UES-944] - Maximize gadget is not fitting the screen correctly when the
left pane is expanded
[UES-947] - Scroll bar is shown in the middle of the page when left
navigation pane is collapsed in view mode
[UES-952] - Maximization is not working in designer mode
[UES-955] - Show an error in the front end when the user input db query is
incorrect
[UES-956] - Validate database URL, table name and credentials in configure
provider pane
[UES-960] - Side bar which lists the pages in the view mode is not linked
correctly
[UES-962] - Exported Dashboard.json do not load the pages other than the
landing page.
[UES-966] - location need to be fixed where the backed files are copied
Improvement

[UES-936] - Role based permissions for creating, managing and deleting
dashboards
[UES-967] - Improving and fixing issues in gadget generation wizard
New Feature

[UES-959] - Share Dashboards among tenants
[UES-963] - Role based access control for gadgets
[UES-965] - Add/Delete gadget and layout from portal app
Task

[UES-922] - Update create gadgets flow with new UI/UX
[UES-953] - Show an error in the front end, if the database driver is not
available
Reporting Issues

Issues can be reported through the Public JIRA
 project for WSO2 Dashboard Server.


[1]
https://github.com/wso2/product-ds/releases/download/v2.1.0-m3/wso2ds-2.1.0-SNAPSHOT.zip


Thanks,

Dashboard Server Team.

-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] ESB Analytics Dashboard Memory Usage

[Sinthuja Ragendran]

Hi Chamila,

I think this is specifically for analytics dashboard right, and not for
simple dashboards? In that case, we need to check how the gadgets have been
written, and because the actual data is being fetched from DAS into this
gadgets, and I believe the fetched data to display in gadget is consuming
more memory. Can you try to delete some gadgets from the above page and see
whether there is an improvement?

@Dunith/Tharik, Whether data is loaded in paginated manner (ie, not load
all, only load the data which is required for the specific page)? And do we
load events with its all attributes or only with specific attributes we
required to show? I believe we have done some testing for ESB analytics
case right, didn't we hit this sort of issue in our local testing?

Thanks,
Sinthuja.


On Thu, Jun 30, 2016 at 9:19 AM, Nipuna Chandradasa 
wrote:

> [Adding Udara,Sinthuja, and Tanya]
>
> Hi Chamila,
>
> This much of memory never used by shindig on gadget rendering. Because
> gadgets are pretty simple as i check them. There must be some pre processes
> going on. That might cause this kind of memory usage.
>
> Thank you,
>
> On Thu, Jun 30, 2016 at 1:46 AM, Chamila De Alwis 
> wrote:
>
>> Hi,
>>
>> ESB 5.0.0 beta analytics dashboard seems to be taking a considerable
>> amount of memory to operate. I noticed the following in
>> Chrome 51.0.2704.103 m (64-bit) in Windows.
>>
>>
>> [image: Inline image 2]
>>
>> Next to Gmail Inbox, ESB Dashboard takes about average 400MB to operate.
>> This can sometimes go up to more than 700MB when browsing through pages.
>>
>> I compared the memory usage against Grafana demo, Grafana Playground [1]
>> and a Kibana4 demo[2], which takes average 150MB and 200MB of memory
>> respectively.
>>
>> IMO there's room for this memory usage to be reduced. Since dashboards
>> would likely be used frequently, reducing the memory footprint would add a
>> considerable amount of value to the user experience.
>>
>> [1] - http://play.grafana.org/
>> [2] - https://kibana.logit.io/app/kibana#/dashboard/Demo-IIS-Dashboard
>>
>> Regards,
>> Chamila de Alwis
>> Committer and PMC Member - Apache Stratos
>> Software Engineer | WSO2 | +94772207163
>> Blog: code.chamiladealwis.com
>>
>>
>>
>
>
> --
> Nipuna Marcus
> *Software Engineer*
> WSO2 Inc.
> http://wso2.com/ - "lean . enterprise . middleware"
> Mobile : +94 (0) 713 667906
> nipu...@wso2.com
>



-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] In-Tenant Data restriction in the DAS platform

[Sinthuja Ragendran]

Hi Dulitha,

On Wed, Jun 29, 2016 at 10:24 PM, Dulitha Wijewantha 
wrote:

> Hi guys,
> Below are somethings I noted when I was writing dashboards for an
> analytics solution.
>
> 1) oAuth protected APIs should be used to retrieve data for gadgets
>
> 2) There should be a way to restrict data for users inside a tenant
>

+1 for above two. And I too think we should bring more fine grained
authorization model for DAS layer, at least in the table/stream level such
that only role-A should be able to access it not all. And again there could
be different level of access per stream/table, some users can only fetch
the data, some can only send, and only some can delete it.

We had similar requirement on dashboard server to protect a dashboard, and
then we came up with a model to create some internal roles per dashboard
during the dashboard creation time, and assign the user who is creating the
dashboard for those internal role by default. Hence only he/she can perform
any actions on the dashboard and it's private for him/her. If the user
would like to share the dashboard, then he/she assign users independently
for the internal roles created or assign a new role for the particular
action.

I think similarly we can handle for the tables as well.

>
> 3) If the user doesn't have authorization to view the data - he shouldn't
> be able to view the corresponding visualization on the dashboard server and
> vice versa.
>

This is bit tricky, as the authorization from dashboard page is something
only required if there are any analytics related gadgets have been included
in the dashboard page, and for others this will not be an issue. We need to
properly handle this case if we include such feature.

Thanks,
Sinthuja.


>
> Cheers~
> --
> Dulitha Wijewantha (Chan)
> Software Engineer - Mobile Development
> WSO2 Inc
> Lean.Enterprise.Middleware
>  * ~Email   duli...@wso2.com *
> *  ~Mobile +94712112165 <%2B94712112165>*
> *  ~Website   dulitha.me *
> *  ~Twitter @dulitharw *
>   *~Github @dulichan *
>   *~SO @chan *
>



-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [Dev] Provide Support to run Jaggery apps in Tomcat Server

[Sinthuja Ragendran]

Hi,

Anyhow ultimately when all the carbon products move to AS 6.0 based
platform, we can merge all those into one.

Thanks,
Sinthuja.

On Mon, Jul 4, 2016 at 12:38 PM, Nisala Nanayakkara  wrote:

> Hi Kasun,
>
> Yes.We can do a performance round. Currently we manage the Tomcat Jaggery/
> AS6 support code base under a new branch [1] in our jaggery repository.
> Current jaggery is based on tomcat version 7 and Tomcat Jaggery
> implementation is based on tomcat version 8. So there are some api changes
> in the jaggery level. So it is difficult to manage these codebases in same
> place.
>
> Thanks,
> Nisala
>
> On Sun, Jul 3, 2016 at 9:01 AM, KasunG Gajasinghe  wrote:
>
>>
>> Good work Nisala.
>>
>> Can we also do a performance round to see whether there are any perf
>> gains when running Jaggery apps in vanilla Tomcat/AS6?
>>
>> How are you going to manage the codebase for Carbon vs Tomcat Jaggery
>> support? Better if we can maintain it in the same place.
>>
>> On Sat, Jul 2, 2016 at 12:51 PM, Nisala Nanayakkara 
>> wrote:
>>
>>> Hi all,
>>>
>>> This is an update regarding the feature 'Provide Support to run Jaggery
>>> apps in Tomcat Server'.
>>>
>>> At present Jaggery scripts can only be executed in a jaggery server.
>>> Because Jaggery Server which is powered by WSO2 carbon OSGi Runtime, is the
>>> Runtime Environment that hosts Jaggery applications. The goal of this
>>> feature is to provide support to run jaggery apps in a pure tomcat server
>>> without any modifications to the jaggery apps.This will enable developers
>>> to write Jaggery scripts and directly deploy them in apache tomcat server.
>>> So this will improve the flexibility of the jaggery apps.
>>>
>>> This is an extension for the Apache tomcat 8 to run the jaggery app
>>> without modifying it. Although this is implemented as a requirement of WSO2
>>> Application Server 6, Users who want to run jaggery apps without jaggery
>>> server, can use this feature to run their jaggery apps in Apache tomcat
>>> server.
>>>
>>> I have attached the documentation and sample jaggery with this email.
>>> Please find these attachments and relevant jaggery branch here[1]. Please
>>> feel free to ask any questions.
>>>
>>> [1] - https://github.com/wso2/jaggery/tree/tomcat-8.0.x-jaggery
>>>
>>> Thanks,
>>> Nisala
>>>
>>> --
>>> *Nisala Niroshana Nanayakkara,*
>>> Software Engineer
>>> Mobile:(+94)717600022
>>> WSO2 Inc., http://wso2.com/
>>>
>>> ___
>>> Dev mailing list
>>> d...@wso2.org
>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>
>>>
>>
>>
>> --
>>
>> *Kasun Gajasinghe*Associate Technical Lead, WSO2 Inc.
>> email: kasung AT spamfree wso2.com
>> linked-in: http://lk.linkedin.com/in/gajasinghe
>> blog: http://kasunbg.org
>>
>>
>>
>
>
>
> --
> *Nisala Niroshana Nanayakkara,*
> Software Engineer
> Mobile:(+94)717600022
> WSO2 Inc., http://wso2.com/
>
> ___
> Dev mailing list
> d...@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Caching Real time analytics data

[Sinthuja Ragendran]

Hi Sachith,

If the use-case is to display the 1 hour analytics data from CEP, then IMO
the he/she need to simply store the CEP results into a persistence store
(DAS or RDBMS via RDBMS event publisher), and then let the gadget read from
the persistence store. I don't think caching is a good option in such cases
because anyhow if the server crashes due to some reason the data is not
going to be shown.

Thanks,
Sinthuja.

On Wed, Jul 6, 2016 at 10:12 PM, Sachith Withana  wrote:

> Hi all,
>
> In the dashboard, the real time data is only shown if the user is logged
> into the dashboard at the time of the data is being pushed.
>
> If the data is being pushed every hour, a new user who logs in would
> potentially have to wait up to one hour to see the real time data, and if
> the user refreshes, then has to wait another hour to see the data, and
> would loose the current data completely.
>
> I understand from the CEP perspective it's similar to fire and forget, but
> can't we add some level of caching to prevent this?
>
> Regards,
> Sachith
> --
> Sachith Withana
> Software Engineer; WSO2 Inc.; http://wso2.com
> E-mail: sachith AT wso2.com
> M: +94715518127
> Linked-In: 
> https://lk.linkedin.com/in/sachithwithana
>



-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Caching Real time analytics data

[Sinthuja Ragendran]

Hi Sachith,

On Wed, Jul 6, 2016 at 11:19 PM, Sachith Withana  wrote:

> Agreed Sinthuja.
>
> But what about for a smaller window size (5/10/15 mins)?
>

Even in this case, I would suggest to persist the result. It makes more
sense to have the realtime analytics when you want to see seconds wise or
less than a minute analysis, and rest of the use cases we need to persist.
And you could persist the data in RDBMS such that when you get a new event,
you update the old value, therefore data doesn't grow. Ie, if you want to
maintain every minute wise analysis data, you would end up storing
60*24=1440 records. Furthermore, as you are just going to insert one event
per a minute, I don't think it will add any performance implication as well.


> The reason why I bought up this issue is, in my case, I use several real
> time gadgets.
> And at the startup, they are all empty for that user until the data gets
> pushed in.
>

Yeah, I understand. :) As CEP is in memory engine, if we want to handle
this situation smoothly even after server crashed, etc IMO we need to
handle this as above described.

Thanks,
Sinthuja.


> As an end user, I would like to see the last status of the real time
> analytics when I log in.
>
> Thanks,
> Sachith
>
> On Wed, Jul 6, 2016 at 12:09 PM, Sinthuja Ragendran 
> wrote:
>
>> Hi Sachith,
>>
>> If the use-case is to display the 1 hour analytics data from CEP, then
>> IMO the he/she need to simply store the CEP results into a persistence
>> store (DAS or RDBMS via RDBMS event publisher), and then let the gadget
>> read from the persistence store. I don't think caching is a good option in
>> such cases because anyhow if the server crashes due to some reason the data
>> is not going to be shown.
>>
>> Thanks,
>> Sinthuja.
>>
>> On Wed, Jul 6, 2016 at 10:12 PM, Sachith Withana 
>> wrote:
>>
>>> Hi all,
>>>
>>> In the dashboard, the real time data is only shown if the user is logged
>>> into the dashboard at the time of the data is being pushed.
>>>
>>> If the data is being pushed every hour, a new user who logs in would
>>> potentially have to wait up to one hour to see the real time data, and if
>>> the user refreshes, then has to wait another hour to see the data, and
>>> would loose the current data completely.
>>>
>>> I understand from the CEP perspective it's similar to fire and forget,
>>> but can't we add some level of caching to prevent this?
>>>
>>> Regards,
>>> Sachith
>>> --
>>> Sachith Withana
>>> Software Engineer; WSO2 Inc.; http://wso2.com
>>> E-mail: sachith AT wso2.com
>>> M: +94715518127
>>> Linked-In: <http://goog_416592669>
>>> https://lk.linkedin.com/in/sachithwithana
>>>
>>
>>
>>
>> --
>> *Sinthuja Rajendran*
>> Technical Lead
>> WSO2, Inc.:http://wso2.com
>>
>> Blog: http://sinthu-rajan.blogspot.com/
>> Mobile: +94774273955
>>
>>
>>
>
>
> --
> Sachith Withana
> Software Engineer; WSO2 Inc.; http://wso2.com
> E-mail: sachith AT wso2.com
> M: +94715518127
> Linked-In: <http://goog_416592669>
> https://lk.linkedin.com/in/sachithwithana
>



-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] ESB Analytics Dashboard Memory Usage

[Sinthuja Ragendran]

Hi,

I tried this scenario with DAS 3.1.0 beta pack, and this is only observable
for the gadgets created by the gadget generation wizard. And I could see a
fast growth of the memory usage when the refresh interval for the gadget is
reduced. I think the data is getting accumulated in the gadget without
getting properly cleaned up. I'll check it further and update.

Thanks,
Sinthuja.

On Thu, Jul 14, 2016 at 10:28 AM, Chamila De Alwis 
wrote:

> Hi Udara,
>
> I've seen this behavior both in Windows and in Ubuntu (and in Ubuntu
> Firefox as well). I even saw memory usage values of more than 2GB for a
> window which was kept open for some time. Could you try with the steps
> mentioned in [1]?
>
> [1] - https://wso2.org/jira/browse/DAS-465
>
>
> Regards,
> Chamila de Alwis
> Committer and PMC Member - Apache Stratos
> Software Engineer | WSO2 | +94772207163
> Blog: code.chamiladealwis.com
>
>
>
> On Wed, Jul 13, 2016 at 11:53 PM, Udara Rathnayake 
> wrote:
>
>> Hi Chamila,
>>
>> I did some profiling for dashboard view with few sample gadgets (using
>> chrome dev tools) sometime back and haven't noticed any memory issues.
>> Noticed memory growth with the time and then gc ran as expects, observed
>> this behavior continuously. Let us try the same in the latest portal app
>> and update you.
>>
>>
>>
>> On Thu, Jul 14, 2016 at 6:50 AM, Chamila De Alwis 
>> wrote:
>>
>>> I'm seeing this behavior in Ubuntu 16.04 Google Chrome as well (for a
>>> newly created dashboard with 4 generated gadgets in one page). The amount
>>> of memory consumed seems to be increasing gradually as the window is kept
>>> open. Sometimes the memory usage goes well beyond 1GB.
>>>
>>> IMO we should do a profiling to see what causes this much of memory to
>>> be used.
>>>
>>> [image: Inline image 1]
>>>
>>>
>>> Regards,
>>> Chamila de Alwis
>>> Committer and PMC Member - Apache Stratos
>>> Software Engineer | WSO2 | +94772207163
>>> Blog: code.chamiladealwis.com
>>>
>>>
>>>
>>> On Thu, Jun 30, 2016 at 11:19 AM, Chamila De Alwis 
>>> wrote:
>>>
>>>> Hi Sinthuja,
>>>>
>>>> I'm seeing around 500MB of memory usage in the DAS 3.1.0-SNAPSHOT
>>>> "Generate Gadget" page as well, specially when the gadget is being
>>>> previewed. At that moment, there is only one gadget in play, and the number
>>>> of records is less than 100.
>>>>
>>>>
>>>> Regards,
>>>> Chamila de Alwis
>>>> Committer and PMC Member - Apache Stratos
>>>> Software Engineer | WSO2 | +94772207163
>>>> Blog: code.chamiladealwis.com
>>>>
>>>>
>>>>
>>>> On Wed, Jun 29, 2016 at 11:53 PM, Sinthuja Ragendran >>> > wrote:
>>>>
>>>>> Hi Chamila,
>>>>>
>>>>> I think this is specifically for analytics dashboard right, and not
>>>>> for simple dashboards? In that case, we need to check how the gadgets have
>>>>> been written, and because the actual data is being fetched from DAS into
>>>>> this gadgets, and I believe the fetched data to display in gadget is
>>>>> consuming more memory. Can you try to delete some gadgets from the above
>>>>> page and see whether there is an improvement?
>>>>>
>>>>> @Dunith/Tharik, Whether data is loaded in paginated manner (ie, not
>>>>> load all, only load the data which is required for the specific page)? And
>>>>> do we load events with its all attributes or only with specific attributes
>>>>> we required to show? I believe we have done some testing for ESB analytics
>>>>> case right, didn't we hit this sort of issue in our local testing?
>>>>>
>>>>> Thanks,
>>>>> Sinthuja.
>>>>>
>>>>>
>>>>> On Thu, Jun 30, 2016 at 9:19 AM, Nipuna Chandradasa 
>>>>> wrote:
>>>>>
>>>>>> [Adding Udara,Sinthuja, and Tanya]
>>>>>>
>>>>>> Hi Chamila,
>>>>>>
>>>>>> This much of memory never used by shindig on gadget rendering.
>>>>>> Because gadgets are pretty simple as i check them. There must be some pre
>>>>>> processes going on. That might cause this kind of memory usage.
>>>>>>
>>>>>> Thank y

Re: [Architecture] ESB Analytics Dashboard Memory Usage

[Sinthuja Ragendran]

Hi,

When i further analyze this issue, it looks like it's coming from charting
library. Basically I commented out chart.insert(data); and then I don't see
the memory growth. So I doubt there is some sort of memory leak from the
charting library.

@Tharik,
Can you please check further in this issue?

Thanks,
Sinthuja.


On Thu, Jul 14, 2016 at 1:53 PM, Sinthuja Ragendran 
wrote:

> Hi,
>
> I tried this scenario with DAS 3.1.0 beta pack, and this is only
> observable for the gadgets created by the gadget generation wizard. And I
> could see a fast growth of the memory usage when the refresh interval for
> the gadget is reduced. I think the data is getting accumulated in the
> gadget without getting properly cleaned up. I'll check it further and
> update.
>
> Thanks,
> Sinthuja.
>
> On Thu, Jul 14, 2016 at 10:28 AM, Chamila De Alwis 
> wrote:
>
>> Hi Udara,
>>
>> I've seen this behavior both in Windows and in Ubuntu (and in Ubuntu
>> Firefox as well). I even saw memory usage values of more than 2GB for a
>> window which was kept open for some time. Could you try with the steps
>> mentioned in [1]?
>>
>> [1] - https://wso2.org/jira/browse/DAS-465
>>
>>
>> Regards,
>> Chamila de Alwis
>> Committer and PMC Member - Apache Stratos
>> Software Engineer | WSO2 | +94772207163
>> Blog: code.chamiladealwis.com
>>
>>
>>
>> On Wed, Jul 13, 2016 at 11:53 PM, Udara Rathnayake 
>> wrote:
>>
>>> Hi Chamila,
>>>
>>> I did some profiling for dashboard view with few sample gadgets (using
>>> chrome dev tools) sometime back and haven't noticed any memory issues.
>>> Noticed memory growth with the time and then gc ran as expects, observed
>>> this behavior continuously. Let us try the same in the latest portal app
>>> and update you.
>>>
>>>
>>>
>>> On Thu, Jul 14, 2016 at 6:50 AM, Chamila De Alwis 
>>> wrote:
>>>
>>>> I'm seeing this behavior in Ubuntu 16.04 Google Chrome as well (for a
>>>> newly created dashboard with 4 generated gadgets in one page). The amount
>>>> of memory consumed seems to be increasing gradually as the window is kept
>>>> open. Sometimes the memory usage goes well beyond 1GB.
>>>>
>>>> IMO we should do a profiling to see what causes this much of memory to
>>>> be used.
>>>>
>>>> [image: Inline image 1]
>>>>
>>>>
>>>> Regards,
>>>> Chamila de Alwis
>>>> Committer and PMC Member - Apache Stratos
>>>> Software Engineer | WSO2 | +94772207163
>>>> Blog: code.chamiladealwis.com
>>>>
>>>>
>>>>
>>>> On Thu, Jun 30, 2016 at 11:19 AM, Chamila De Alwis 
>>>> wrote:
>>>>
>>>>> Hi Sinthuja,
>>>>>
>>>>> I'm seeing around 500MB of memory usage in the DAS 3.1.0-SNAPSHOT
>>>>> "Generate Gadget" page as well, specially when the gadget is being
>>>>> previewed. At that moment, there is only one gadget in play, and the 
>>>>> number
>>>>> of records is less than 100.
>>>>>
>>>>>
>>>>> Regards,
>>>>> Chamila de Alwis
>>>>> Committer and PMC Member - Apache Stratos
>>>>> Software Engineer | WSO2 | +94772207163
>>>>> Blog: code.chamiladealwis.com
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jun 29, 2016 at 11:53 PM, Sinthuja Ragendran <
>>>>> sinth...@wso2.com> wrote:
>>>>>
>>>>>> Hi Chamila,
>>>>>>
>>>>>> I think this is specifically for analytics dashboard right, and not
>>>>>> for simple dashboards? In that case, we need to check how the gadgets 
>>>>>> have
>>>>>> been written, and because the actual data is being fetched from DAS into
>>>>>> this gadgets, and I believe the fetched data to display in gadget is
>>>>>> consuming more memory. Can you try to delete some gadgets from the above
>>>>>> page and see whether there is an improvement?
>>>>>>
>>>>>> @Dunith/Tharik, Whether data is loaded in paginated manner (ie, not
>>>>>> load all, only load the data which is required for the specific page)? 
>>>>>> And
>>>>>> do we load events with its all attributes or only with specific 
>>>>&

Re: [Architecture] [Arch] Adding CEP and ML samples to DAS distribution in a consistent way

[Sinthuja Ragendran]

Hi,

We also need to find a consistent way to maintain the integration tests as
well. CEP, and ML features are being used in DAS, and there is no
integration tests for those components getting executed in the DAS product
build. Similarly there are many UI tests we have in dashboard server as
well, but those are not executed in the products which are using those. As
these are the core functionalities of DAS, IMHO we need to execute the
testcases for each of these components during the product-das build time.

Thanks,
Sinthuja.

On Thu, Aug 4, 2016 at 3:17 PM, Niranda Perera  wrote:

> Hi Suho,
>
> As per the immediate DAS 310 release, we will continue to keep a local
> copy of the samples. I have created a JIRA here [1] to add the suggestion
> provided by Isuru.
>
> Best
>
> [1] https://wso2.org/jira/browse/DAS-481
>
> On Wed, Aug 3, 2016 at 10:02 PM, Sriskandarajah Suhothayan 
> wrote:
>
>> DAS team how about doing it for this release ?
>>
>> Regards
>> Suho
>>
>> On Wed, Aug 3, 2016 at 6:31 PM, Ramith Jayasinghe 
>> wrote:
>>
>>> I think we need to ship samples with product. otherwise, The first
>>> 5-minite experience of users will be negatively affected.
>>>
>>>
>>> ___
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> --
>>
>> *S. Suhothayan*
>> Associate Director / Architect & Team Lead of WSO2 Complex Event
>> Processor
>> *WSO2 Inc. *http://wso2.com
>> * *
>> lean . enterprise . middleware
>>
>>
>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog:
>> http://suhothayan.blogspot.com/ twitter:
>> http://twitter.com/suhothayan  | linked-in:
>> http://lk.linkedin.com/in/suhothayan *
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> *Niranda Perera*
> Software Engineer, WSO2 Inc.
> Mobile: +94-71-554-8430
> Twitter: @n1r44 
> https://pythagoreanscript.wordpress.com/
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] BAM Performance tests

[Sinthuja Ragendran]

Hi,

I have completed upto puppetizing BAM Receiver configurations, and due to
support priorities I wasn't able to work full time on this. Hence cassandra
and analyzer confgurations are still pending. Anyhow since I have done with
the base confgurations, I should be able to provide the complete puppet
scripts by next week.

Thanks,
Sinthuja.


On Fri, Aug 15, 2014 at 6:22 PM, Sanjiva Weerawarana 
wrote:

> Is the setup all automated with Puppet?
>
>
> On Fri, Aug 15, 2014 at 11:26 AM, Thayalan  wrote:
>
>> Hi Srinath,
>>
>> Due to support priority this has not been started yet. However I've
>> already done the environment set-up in performance cloud. Probably I can
>> share the initial findings by next week.
>>
>> Thanks,
>> Thayalan
>>
>>
>> On Fri, Aug 15, 2014 at 8:58 AM, Srinath Perera  wrote:
>>
>>> How are we doing with the subject?
>>> --
>>>  
>>> Director, Research, WSO2 Inc.
>>> Visiting Faculty, University of Moratuwa
>>> Member, Apache Software Foundation
>>> Research Scientist, Lanka Software Foundation
>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera
>>> Site: http://people.apache.org/~hemapani/
>>> Photos: http://www.flickr.com/photos/hemapani/
>>> Phone: 0772360902
>>>
>>> ___
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Sanjiva Weerawarana, Ph.D.
> Founder, Chairman & CEO; WSO2, Inc.;  http://wso2.com/
> email: sanj...@wso2.com; office: (+1 650 745 4499 | +94  11 214 5345)
> x5700; cell: +94 77 787 6880 | +1 408 466 5099; voip: +1 650 265 8311
> blog: http://sanjiva.weerawarana.org/; twitter: @sanjiva
> Lean . Enterprise . Middleware
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Senior Software Engineer 
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [Dev] ES- Asset publisher to BAM

[Sinthuja Ragendran]

Hi,

I see eventName is being used as meta data and payload data; what are these
properties? If it's same property, you don't need to duplicate it and
rather have it in the single place either as meta data or payload data.

Also what is ES_EVENT in correlation data? Can you explain the properties
of the stream definition you have used here?

Thanks.
Sinthuja.

On Mon, Sep 29, 2014 at 4:33 PM, Sriskandarajah Suhothayan 
wrote:

> Please use load-balancing data-publisher to publish data to BAM.
>
> Suho
>
> On Mon, Sep 29, 2014 at 4:23 PM, Manoj Gunawardena 
> wrote:
>
>> Hi Devs,
>>
>> For ES, I am creating a JAGGERY API to publish ES asset events to BAM.
>> The objective is show asset related statistics in BAM dashboard.
>>
>>
>> Inside API, uses Data publisher to send events, related to assets.
>>
>> When an event occurs in ES, which need to gather statistics, the jaggery
>> code will call the API and publish event details to BAM.
>>
>>
>>
>> Ex -: Suppose we need to show information of how many downloads per asset.
>> The asset download JAG page call the publishEvents API.
>>
>>
>>
>> Please go through the diagrom and share your thoughts.
>>
>>
>>
>>
>> Thanks
>>
>>
>>
>> ​
>>  BAM Publsiher
>> 
>> ​
>>
>>
>>
>> --
>> Manoj Gunawardena
>> Tech Lead
>> WSO2, Inc.: http://wso2.com
>> lean.enterprise.middleware
>> Mobile : +94 77 2291643
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
>
> *S. Suhothayan*
> Technical Lead & Team Lead of WSO2 Complex Event Processor
>  *WSO2 Inc. *http://wso2.com
> * *
> lean . enterprise . middleware
>
>
> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog:
> http://suhothayan.blogspot.com/ twitter:
> http://twitter.com/suhothayan  | linked-in:
> http://lk.linkedin.com/in/suhothayan *
>
> ___
> Dev mailing list
> d...@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 
*Sinthuja Rajendran*
Senior Software Engineer 
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

[Architecture] WSO2 BAM 2.5.0 Released !!

[Sinthuja Ragendran]

WSO2 Business Activity Monitor 2.5.0 Release Notes *15th of December 2014*

WSO2 Business Activity Monitor (WSO2 BAM) is a comprehensive framework
designed to solve the problems in the wide area of business activity
monitoring. WSO2 BAM comprises of many modules to give the best of
performance, scalability and customizability. This allows to achieve
requirements of business users, dev ops, CEOs without spending countless
months on customizing the solution without sacrificing performance or the
ability to scale.

WSO2 BAM is powered by WSO2 Carbon, the SOA middleware component platform.

New Features

   - Support for new Input/Output Adaptors
   - Data Purging support from UI and toolbox
   - New Cassandra Hector Datasource Implementation

Issues Fixed in This Release

   - WSO2 BAM 2.5.0 Fixed Issues
   

 Improvement



   - The toolbox search does not seem to generate results for partial
   toobox name
   - Need to update some tool boxes to refer the WSO2BAM_DATASOURCE
   - [Toolbox] Remove username/password from the toolbox in
   streamdefn.properties
   - Enhance Message Tracer agent to capture request url in a REST API
   invoke and transport headers
   - Improved BAM Receiver performance
   - Accessinig the registry for every event bundle
   - Add default timestamp based search support in Activity Monistoring
   Dashboard(Simple Search)
   - Removing reduntant configuration from streamDefn.xml and read all
   cassandra related configs from datasources
   - Cassandra Data deletion support for the Archival Feature
   - Improve BAM to work with all Input adators provided by CEP
   - Support index deletion
   - Supporting to schedhule cassandra data deletion through toolboxes
   - Add comment support to Hive script editor
   - Cassandra Data Source Type for BAM Configurations
   - Add a configuration to specify the async data publisher consistancy
   - Set the hostName parameter to 0.0.0.0 in data-bridge-config.xml file
   by default
   - Even though we purge data there is no way to generate a report or
   identify who performed the deletion
   - [Improvement] better if the archived / purge function validate and
   prevent excluded streams being achieved or purged
   - Add Info logs to show the execution time of each Hive script
   - Resolve the additional configuration changes due to the port offset in
   BAM

 Task

   - Adding puppet v3 deployment scripts for BAM 2.5.0

Known Issues

   - WSO2 BAM 2.5.0 Known Issues
   

Deprecated Features

   - Removing REST API support provided by BAM 2.4.x and prior. Instead of
   the REST API, we can use HTTP input event adaptor with BAM 2.5.0 for the
   same use case.

Reporting Problems

WSO2 encourages you to report issues, enhancements and feature requests for
WSO2 BAM. Use the issue tracker  for
reporting issues.
Discussion Forums

We encourage to use stackoverflow  to
engage with
Support

We are committed to ensuring that your enterprise middleware deployment is
completely supported from evaluation to production. Our unique approach
ensures that all support leverages our open development methodology and is
provided by the very same engineers who build the technology.

For additional support information please refer tohttp://wso2.com/support/


Thank you for your interest in WSO2 Business Activity Monitor
* -The WSO2 Business Activity Monitor Development Team *


-- 
*Sinthuja Rajendran*
Senior Software Engineer 
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] BAM 3.0 REST APIs for AnalyticsDataService / Indexing / Search

[Sinthuja Ragendran]

Hi gimantha,

Please see the comments inline.

On Sun, Jan 18, 2015 at 11:24 PM, Gimantha Bandara 
wrote:

> Hi,
> Currently, I am working on $subject. Basically the methods in
> AnalyticsDataService will be exposed through REST APIs. Please refer to
> Architecture mail thread "*[Architecture] BAM 3.0 Data Layer
> Implementation / RDBMS / Distributed Indexing / Search*" for more
> Details. Following are the supported REST APIs.
>
> 1. Create a table
> *POST /analytics/{tableName}*
>  ; tableName - The name of the table to be created.
>

IMHO the above should be POST to '/analytics/*tables*' and the request
content should have the table name as given below.
{
 "tableName" : "Test"
}

>
> 2. Delete a table
> *DELETE /analytics/{tableName} *
> ; tableName - The name of the table to be deleted.
>

> 3. Check if a table exists
> *GET /analytics/{tableName} *
> ; tableName - The name of the table being checked.
>
> 4. List All the tables
> *GET /analytics/tables*
> ;Response will be an JSON array of table names. e.g. [ "table1" , "table2"
> , "table3" ]
>
> 5. Get the records from a table.
> *GET /analytics/records/{tableName}/{from}/{to}/{start}/{count} *
> ; tableName - The name of the table from which the records are retrieved.
> ; from - The starting time to get records from.
> ; to - The ending time to get records to.
> ; start - The paginated index from value
> ; count - The paginated records count to be read
> ; response - takes the format of the request content of No.7
>

Do we need to have 'records' in the URL?  I think it's better to have
*/analytics/{tableName}/{from}/{to}/{start}/{count}
*

>
> 6. Get the records from a table (By IDs)
> *POST /analytics/records/{tableName}*
> ; tableName - The name of the table from which the records are retrieved.
> ; Content  - A List of IDs of the records to be retrieved in the following
> format.
> [ "id1" , "id2" , "id3" ]
> ; response - takes the format of the request content of No.7
>

Similarly can we have this as * /analytics/{tableName}?*

>
> 7. Create records ( can be created in different tables or in the same )
> *POST /analytics/records*
> ; Content - A list of records in json format like in below.
> [
> {
> "id": "id1",
> "tenantId": -1234,
> "tableName": "tableName1",
> "timestamp": "-mm-dd hh:mm:ss",
> "values":
> {
> "columnName1": "value1",
> "columnName2": "value2"
> }
> },
>{
> "id": "id2",
> "tenantId": -1234,
> "tableName": "tableName2",
> "timestamp": "-mm-dd hh:mm:ss",
> "values":
> {
> "columnName1": "value1",
> "columnName2": "value2"
> }
> },
> ]
>
> 8. Delete records
> *DELETE /analytics/records/{tableName}/{timeFrom}/{timeTo}*
> ; tableName - Name of the table from which the records are deleted.
> ; timeFrom - The starting time to delete records from.
> ; timeTo - The end time to delete records to.
>
>
Again do we need to have 'records' in the middle?  IMHO
/analytics/{tableName}/{timeFrom}/{timeTo} is better.

9. Update records
> *PUT /analytics/records*
> ; Content - As same as the POST method for creating records
>
> 10. Get the record count of table
> *GET /analytics/count/{tableName}*
> ; tableName - The name of the table
>
> 11. Create Indices for a table
> *POST /analytics/indices/{tableName}*
> ; tableName - The name of the table of which the indices are set
> ; Content - takes the following format. TYPE is one of "INTEGER",
> "BOOLEAN", "DOUBLE", "STRING", "FLOAT", "LONG"
> {
> "indexColumnName1" : "TYPE1",
> "indexColumnName2" : "TYPE2"
> }
>

> 12. get the indices of a table
> *GET /analytics/indices/{tableName}*
> ; tableName - The name of the table
> ; Response will be of the format of the previous POST request's Content.
>
> 13. Clear the indices of a table
> *DELETE /analytics/indices/{tableName}*
> ; tableName - The name of the table
>
> 14. Search records of a table
> *POST /analytics/search*
> ; Content - takes the following format
> {
> "tableName": "sampleTableName",
> "language": "sampleLanguageName",
> "query": "sampleQuery",
> "start": "start-location-of-the-result",
> "count": "maximum-number-of-entries-to-return"
> }
>

IMHO this should be a GET request.


Thanks,
Sinthuja.


> If a method does not have a specific response mentioned above, the
> response will take the following format.
>
> {
> "status" : "statusValue",
> "message" : "sampleMessage"
> }
>
> Suggestions and feedbacks are appreciated.
>
> Thanks,
>
> --
> Gimanthaa Bandara
> Software Engineer
> WSO2. Inc : http://wso2.com
> Mobile : +94714961919
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Senior Software Engineer 
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogsp

Re: [Architecture] BAM 3.0 REST APIs for AnalyticsDataService / Indexing / Search

[Sinthuja Ragendran]

Hi Gimantha,

I think following the conventions will be more intuitive to the users, and
we should have a proper reason to not follow. And the post request is
generally needs to be sent to create the object and the back end is going
to decide where to create the tables resource, therefore it should be POST
to '/analytics/tables' and message body should be having the table name. If
you want to use /analytics/{tableName}, then you should use PUT rather POST
[1]. But IMO POST is the most suitable operation for this context.

And through out the below given URL patterns, in order to fetch the records
from a table, you have used '/analytics/records/{tableName}' url pattern
where 'records' is in the middle and it is not the correct data hierarchy
and again I feel it's not the convention. Basically tables contains records
and not records contains tables. Therefore if we use
POST to '/analytics/tables' for create table, then you can use simply user
'analytics/{tableName}' to GET/POST for the tables, IMHO the records is
just an additional segment which is not needed to be here.

[1] http://restcookbook.com/HTTP%20Methods/put-vs-post

Thanks,
Sinthuja.

On Tue, Jan 20, 2015 at 1:29 AM, Gimantha Bandara  wrote:

> Hi Sinduja,
>
> Thank you for the feedback.
>
> On Mon, Jan 19, 2015 at 12:04 PM, Sinthuja Ragendran 
> wrote:
>
>> Hi gimantha,
>>
>> Please see the comments inline.
>>
>> On Sun, Jan 18, 2015 at 11:24 PM, Gimantha Bandara 
>> wrote:
>>
>>> Hi,
>>> Currently, I am working on $subject. Basically the methods in
>>> AnalyticsDataService will be exposed through REST APIs. Please refer to
>>> Architecture mail thread "*[Architecture] BAM 3.0 Data Layer
>>> Implementation / RDBMS / Distributed Indexing / Search*" for more
>>> Details. Following are the supported REST APIs.
>>>
>>> 1. Create a table
>>> *POST /analytics/{tableName}*
>>>  ; tableName - The name of the table to be created.
>>>
>>
>> IMHO the above should be POST to '/analytics/*tables*' and the request
>> content should have the table name as given below.
>> {
>>  "tableName" : "Test"
>> }
>>
> Since the POST takes only the table name, it is straightforward to use it
> as a path parameter.
>
>>
>>> 2. Delete a table
>>> *DELETE /analytics/{tableName} *
>>> ; tableName - The name of the table to be deleted.
>>>
>>
>>> 3. Check if a table exists
>>> *GET /analytics/{tableName} *
>>> ; tableName - The name of the table being checked.
>>>
>>> 4. List All the tables
>>> *GET /analytics/tables*
>>> ;Response will be an JSON array of table names. e.g. [ "table1" ,
>>> "table2" , "table3" ]
>>>
>>> 5. Get the records from a table.
>>> *GET /analytics/records/{tableName}/{from}/{to}/{start}/{count} *
>>> ; tableName - The name of the table from which the records are retrieved.
>>> ; from - The starting time to get records from.
>>> ; to - The ending time to get records to.
>>> ; start - The paginated index from value
>>> ; count - The paginated records count to be read
>>> ; response - takes the format of the request content of No.7
>>>
>>
>> Do we need to have 'records' in the URL?  I think it's better to have  
>> */analytics/{tableName}/{from}/{to}/{start}/{count}
>> *
>>
>
> "records" is there to avoid conflicts with other contexts. As an example,
> If we remove "records", the URL will take the format
> "/analytics/{tableName}", which is already a defined REST API.
>
>>
>>> 6. Get the records from a table (By IDs)
>>> *POST /analytics/records/{tableName}*
>>> ; tableName - The name of the table from which the records are retrieved.
>>> ; Content  - A List of IDs of the records to be retrieved in the
>>> following format.
>>> [ "id1" , "id2" , "id3" ]
>>> ; response - takes the format of the request content of No.7
>>>
>>
>> Similarly can we have this as * /analytics/{tableName}?*
>>
>>>
>>> 7. Create records ( can be created in different tables or in the same )
>>> *POST /analytics/records*
>>> ; Content - A list of records in json format like in below.
>>> [
>>> {
>>> "id": "id1",
>>> "tenantId": -1234,
>>> "tableName": "tableName1",
>&

[Architecture] BAM 3.0 Spark Script and Console UI in Dashboard

[Sinthuja Ragendran]

Hi,

BAM dashboards are used so far to visualize the graphs, etc for presenting
the data, but we are thinking of moving some of the operations from admin
console to dashboard for a seamless user interaction. Basically in BAM 3.0,
the users would like to execute some queries on the console, and then
he/she can export as script and schedule it. And then within same
application he/she can switch the tabs, and visualize the graphs. Since
these are end user related operations and more frequently performed in the
run time, IMHO it's more better to have it in the dashboard so that the
user can stick into one context,  rather switching back and forth to
management console and dashboard.

But we do have other UIs such as streams, input/output adapters etc will
remain in the management console as those are actual devops related
operations.

Please share your thoughts in above approach.

Thanks,
Sinthuja.

-- 
*Sinthuja Rajendran*
Senior Software Engineer 
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] BAM 3.0 Spark Script and Console UI in Dashboard

[Sinthuja Ragendran]

Hi isabelle,

Thanks for your input. Yeah, in this case we are focusing to have end-user
related operations which are not really a devops/admin related operations,
in dashboard it self.

And in the context of doing analysis on the collected data is basically
enduser operation in BAM, and he/she might be interested on some metric of
collected data, and plug into some graph visualize the KPI. In order to do
this, if we have these operations in the dashboard application itself, then
the UX will be smooth. By this way, the enduser will remain to work with
dashboard application, and not necessarily need to switch to  admin or
management console to do their some of their task. Similarly the devops
will continue to work on management console which will facilitate to
configure the server, but not necessarily to see dashboards, etc.

Thanks,
Sinthuja.

On Mon, Mar 9, 2015 at 6:19 PM, Isabelle Mauny  wrote:

> +1 as long as you have a clear user's role separation of the UIs you're
> creating - Don't mix managing/configuring the product with consulting
> dashboards in the same applications (i.e. URL).
>
> I feel that over the platform we have a key need for high-level
> configuration tasks that do not belong in the traditional admin console.
> Either that , or enhancing/segmenting the admin console to again clearly
> separate the tasks that are conducted by different roles.
>
> Isabelle.
>
> -
> *Isabelle Mauny*
> VP, Product Management - WSO2, Inc. - http://wso2.com/
>
>
> On Mon, Mar 9, 2015 at 8:07 AM, Sinthuja Ragendran 
> wrote:
>
>> Hi,
>>
>> BAM dashboards are used so far to visualize the graphs, etc for
>> presenting the data, but we are thinking of moving some of the operations
>> from admin console to dashboard for a seamless user interaction. Basically
>> in BAM 3.0, the users would like to execute some queries on the console,
>> and then he/she can export as script and schedule it. And then within same
>> application he/she can switch the tabs, and visualize the graphs. Since
>> these are end user related operations and more frequently performed in the
>> run time, IMHO it's more better to have it in the dashboard so that the
>> user can stick into one context,  rather switching back and forth to
>> management console and dashboard.
>>
>> But we do have other UIs such as streams, input/output adapters etc will
>> remain in the management console as those are actual devops related
>> operations.
>>
>> Please share your thoughts in above approach.
>>
>> Thanks,
>> Sinthuja.
>>
>> --
>> *Sinthuja Rajendran*
>> Senior Software Engineer <http://wso2.com/>
>> WSO2, Inc.:http://wso2.com
>>
>> Blog: http://sinthu-rajan.blogspot.com/
>> Mobile: +94774273955
>>
>>
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Senior Software Engineer <http://wso2.com/>
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

[Architecture] BAM/CEP Data Publisher refactoring

[Sinthuja Ragendran]

Hi,

I have done the $subject, and this was be mainly focused to provide below
objectives.

   - Unifying all different kind of datapublishers
   (LoadbalancingDatapublisher, AsyncDatapublisher, Datapublisher) currently
   available and provide one Datapublisher.
   - Implement data publisher endpoint as extensible, so that a new
   Endpoint type can be added; by default thrift, and binary is supported.
   - Removing the complications introduced due to improving the data
   publisher to provide load balancing and fail over functionality.

The details on how to use the refactored data publisher has been explained
in [1]. Please note with this change, we also have removed the defineStream
and findStream operation from the data publisher (ie, client from the
client side), and we have moved these operations to the receiver side
(BAM/CEP) as discussed in earlier discussions. The users can define the
stream with toolboxes, or with Event streams UI [2] in the management
console.

Furthermore during this effort there wasn't any changes introduced on the
receiver side (BAM/CEP), and hence old Datapublishers will continue to work
with new BAM 3.0 and CEP 4.0 which is yet to be released. Hence old clients
(APIM, ESB, etc), can send the events to new BAM/CEP without any
problem.Also as far as the streams are defined in older BAM/CEP via
toolboxes, or even streams UI, the new refactored data publisher also will
work with old BAM/CEP.

[1] https://docs.wso2.com/display/BAM300/Data+Publisher
[2] https://docs.wso2.com/display/BAM300/Event+Streams

Thanks,
Sinthuja.

-- 
*Sinthuja Rajendran*
Senior Software Engineer 
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] BAM/CEP Data Publisher refactoring

[Sinthuja Ragendran]

Hi sumedha,

No, we don't have non-secured way of pushing events to BAM/CEP via our
Datapublisher API. But if you have a requirement, then you can use other
transports such as HTTP to send events BAM/CEP via input adapters. This
doesn't require any username/password to push events, but obliviously it's
not fast as Datapublisher API.

Also now CEP team is developing an 'Event Publisher' component which is a
combination of event formatter and output adapter, and with the can
configure where the stream supposed to be sent and transport - JMS, SOAP,
HTTP, Datapublisher, Email, etc. Therefore the carbon servers which is
publishing to BAM/CEP can install this 'Event Publisher' feature, and push
the events to BAM/CEP or any third party endpoints - JMS, SOAP, etc. Hence
if you are requirement is to send events to BAM/CEP from carbon servers
without username, password, with above mentioned approach you can switch
the transport without any code changes to the client side. Eg: APIM can use
the usage stats pushed to Event Publisher as stream, then depending on the
event publisher configuration - JMS, SOAP, HTTP, etc the events will be
pushed to the relevant endpoint.

Thanks,
Sinthuja.

[1] https://docs.wso2.com/display/BAM300/Input+HTTP+Event+Adapter

On Thu, Mar 26, 2015 at 1:10 PM, Sumedha Rubasinghe 
wrote:

> Hi Sinthuja,
> Any plans to update how these endpoints are secured?
>
> I have a requirement where data needs to be published to BAM/CEP by
> randomly provisioned unreliable clients. So using username/password would
> not work.
>
> On Thu, Mar 26, 2015 at 1:04 PM, Sinthuja Ragendran 
> wrote:
>
>> Hi,
>>
>> I have done the $subject, and this was be mainly focused to provide below
>> objectives.
>>
>>- Unifying all different kind of datapublishers
>>(LoadbalancingDatapublisher, AsyncDatapublisher, Datapublisher) currently
>>available and provide one Datapublisher.
>>- Implement data publisher endpoint as extensible, so that a new
>>Endpoint type can be added; by default thrift, and binary is supported.
>>- Removing the complications introduced due to improving the data
>>publisher to provide load balancing and fail over functionality.
>>
>> The details on how to use the refactored data publisher has been
>> explained in [1]. Please note with this change, we also have removed the
>> defineStream and findStream operation from the data publisher (ie, client
>> from the client side), and we have moved these operations to the receiver
>> side (BAM/CEP) as discussed in earlier discussions. The users can define
>> the stream with toolboxes, or with Event streams UI [2] in the management
>> console.
>>
>> Furthermore during this effort there wasn't any changes introduced on the
>> receiver side (BAM/CEP), and hence old Datapublishers will continue to work
>> with new BAM 3.0 and CEP 4.0 which is yet to be released. Hence old clients
>> (APIM, ESB, etc), can send the events to new BAM/CEP without any
>> problem.Also as far as the streams are defined in older BAM/CEP via
>> toolboxes, or even streams UI, the new refactored data publisher also will
>> work with old BAM/CEP.
>>
>> [1] https://docs.wso2.com/display/BAM300/Data+Publisher
>> [2] https://docs.wso2.com/display/BAM300/Event+Streams
>>
>> Thanks,
>> Sinthuja.
>>
>> --
>> *Sinthuja Rajendran*
>> Senior Software Engineer <http://wso2.com/>
>> WSO2, Inc.:http://wso2.com
>>
>> Blog: http://sinthu-rajan.blogspot.com/
>> Mobile: +94774273955
>>
>>
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> /sumedha
> m: +94 773017743
> b :  bit.ly/sumedha
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Senior Software Engineer <http://wso2.com/>
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Replacing BAM Toolbox Format with CAR

[Sinthuja Ragendran]

Hi,

Please see my comments inline.

On Mon, Mar 30, 2015 at 5:24 PM, Iranga Muthuthanthri 
wrote:

>
> Would like to seek clarifications on the following, How it would be
> possible to:
>
> (i) Create the CAR/CApp , would it be through Dev Studio, if so i believe
> tooling does not support this.
>

No, we will not have the car file creating support for the current
DevStudio, and we'll be adding the toolbox support for all BAM/CEP
artefacts once the new DevStudio is released.

>
> (ii) Modification of scripts/Stream definitions after deployment, if its
> through the management console believe there is a limitation as discussed
> on [1]
>

Even though we don't have car file development file support via DevS, we'll
have the car file deployment support for all BAM/CEP artefacts. Therefore
the users have to manually create the car file and deploy that. But as
usual the artefacts which was deployed other than car files, can be managed
via management console.

Thanks,
Sinthuja.

>
> [1] [Dev] Fixing CApp story for Carbon 4.3.0
>
> Regards,
> Iranga
>
> On Thu, Jan 22, 2015 at 11:24 AM, Malith Dhanushka 
> wrote:
>
>> +1 for the approach and we will need proper tooling support for toolbox
>> creation as well.
>>
>
>
>> On Sat, Jan 17, 2015 at 12:28 PM, Anjana Fernando 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> From BAM 3.0, we are thinking of replacing the toolbox packaging to CAR
>>> files. The main motive for this came with CEP also requiring a packaging
>>> format for their artifacts. So either, they also needed to use our toolbox
>>> format, or else, go to a CAR packaging format, which is used with other
>>> artifacts in the platform.
>>>
>>> So basically, as I feel, our artifacts like, stream definitions,
>>> analytics scripts, UI pages are also in the same category as ESBs
>>> sequences, proxies, endpoints etc.. so if they also don't use a new
>>> packaging format, but rather use CAR, we also don't have a special reason
>>> to have a separate one. So for these reasons, and also not to have too many
>>> packaging formats in the platform, we also though of going with the
>>> standard model with CAR.
>>>
>>> CEP have already suggested this for their artifacts in the thread [1].
>>>
>>> If there are any concerns, please shout.
>>>
>>> [1] [Architecture] cApp deployer support for WSO2 CEP
>>>
>>> Cheers,
>>> Anjana.
>>> --
>>> *Anjana Fernando*
>>> Senior Technical Lead
>>> WSO2 Inc. | http://wso2.com
>>> lean . enterprise . middleware
>>>
>>> ___
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> --
>> Malith Dhanushka
>> Software Engineer - Data Technologies
>> *WSO2, Inc. : wso2.com *
>> *Mobile*  : +94 716 506 693
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Thanks & Regards
>
> Iranga Muthuthanthri
> (M) -0777-255773
> Team Product Management
>
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Senior Software Engineer 
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

[Architecture] WSO2 DAS 3.0 M4 Released!

[Sinthuja Ragendran]

Hi,

The WSO2 DAS team is pleased to announce the fourth milestone release of
WSO2 DAS v3.0. The distribution is available at [1]. The release includes
the following new features, improvements, and tasks.
Improvement

   - [BAM-1975 ] - Need to ensure
   that ThreadLocals are cleaned up prior to reuse in Databridge
   - [BAM-1980 ] - Merge FS Data
   Query for RDBMS Connector
   - [BAM-1981 ] - Create carbon
   application deployers for Analytics indices
   - [BAM-1982 ] - Create carbon
   application deployers for Analytics Spark scripts
   - [BAM-1985 ] - Adding Facet
   column and record creation for Message Console
   - [BAM-1987 ] - Add Faceted
   search implementation to Analytics Data Service

New Feature

   - [BAM-1986 ] - Adding data
   purging support for message console

Task

   - [BAM-1983 ] - Adding all CEP
   related features to DAS product
   - [BAM-1984 ] - Renaming BAM to
   DAS


The documentation for DAS v3.0 can be found at [2]. Your feedback is most
welcome, and any issues can be reported to the project at [3].

[1]
https://svn.wso2.org/repos/wso2/people/sinthuja/DAS3.0/wso2das-3.0.0-SNAPSHOT.zip
[2]
https://docs.wso2.com/display/DAS300/WSO2+Data+Analytics+Server+Documentation
[3] https://wso2.org/jira/browse/BAM


- WSO2 DAS Team.

-- 
*Sinthuja Rajendran*
Senior Software Engineer 
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

[Architecture] Privileged user sending events to DAS/CEP as tenant

[Sinthuja Ragendran]

Hi all,

We (Isabelle, Anjana, Suho, NuwanD,  Mohan, Iranga, Jaminda, and Myself)
had a discussion about APIM 1.9 compatibility with DAS last week, and we
need $subject to support APIM with DAS in tenant mode. As per APIM stats
publishing to BAM 2.5.0, the events are published from super tenant (for
all tenants) and having tenant id as separate field in the event. Therefore
in BAM 2.5.0 the events are pushed in cassandra the super-tenant space, and
toolbox and hive scripts runs on the super tenant space and having another
field for tenant id in the summarized table as well. Finally dashboards in
APIM will be connecting to one summarized table, and then filter results
based on the tenant id for the respective tenants to visualize their
statistics.

DAS dashboard is going to live in DAS side rather APIM side, and hence the
respective tenant users can only connect to their respective tenant data
store. Therefore we need to have the statistics events to be pushed into
respective tenant space, rather the super tenant space.

To cater the above requirement, we will have to introduce below approach in
DataPublisher.

1) Create data publisher with privileged super tenant user credentials.
2) setTenantId or setTenantDomain when you want to send the data on behalf
of specific tenant.
3) publish events as usual. If the events are being received by BAM 2.5.0
then it will simply ignore the tenantId/domain part, and it will simply
store the events in the logged in tenant space. In DAS the additional
tenantId/domain will be considered, and it will be inserted in the relevant
tenant space if the logged in user is a valid user with required
permissions.

For the above change to be used at APIM 1.9, we need to do a release for
DataPublisher on 4.2.0 as APIM 1.9 is based on 4.2.0 carbon. I'll work on
this, get the data publisher released soon. Let me know if you have any
concerns in above approach.

Thanks,
Sinthuja.

-- 
*Sinthuja Rajendran*
Associate Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Privileged user sending events to DAS/CEP as tenant

[Sinthuja Ragendran]

Hi Nuwan,

On Tue, May 12, 2015 at 5:32 PM, Nuwan Dias  wrote:

> Aren't we tightly coupling the data publisher and the analyzer by this?
>

No we are not coupling data publishing with analytics here; By the proposed
method, it's simply you can send the tenanted data to respective tenant
space, rather to single super tenant space.

With the change you have proposed, a data published by a particular tenant
> would only be possible to analyze by the same tenant on DAS right?
>

Yes, this is the same way currently other toolboxes behave. Basically the
tenant data will be only analyzed and visualized as per tenant preference,
therefore if the tenant foo.com prefers to visualize the data then he/she
should install the toolbox on DAS. Ideally IMHO there should be a way to
enable to publishing/Monitoring per tenant basis at APIM than enabling
globally for all tenants.


> If that is the case it would cause us to live within a bunch of
> restrictions which might hinder some of our use cases. See below for some
> scenarios.
>
> 1. The analyzer script will have to run on each tenant space, isn't it? If
> so does someone need to manually deploy/configure it per each tenant?
>

Deployment of toolbox is required on DAS.

>
> 2. In API Manager, we have use cases where a particular tenant's API can
> be consumed by other tenant's applications. In this case, the stats related
> to the API are shown under the API owning tenant's space. The stats related
> to the application are shown under the App owning tenant's space. But all
> stats are triggered from the API invocation. At the moment we separate out
> the data based on the information available on the event payload. So we can
> nicely summarize the data accordingly.
>

You still can send the respective tenant data as however you extract
currently, but now it'll go to relevant tenant space, rather single super
tenant space.


>
> If we do the determination of the tenant based on a value in the event
> payload, would it not decouple the publisher and analyzer and cater for
> more possibilities in terms of data summarization/analytics?
>

Having the tenant id in the payload, and setting it in the data publisher
doesn't have any difference in the analytics as far as both are going to be
stored in respective tenant space.

Here the only difference is in the publishing method, where you need to
have separate publishers created for every tenant rather using the same
publisher (you can have a map for tenantId, and datapublisher and then
reuse the data publisher created for tenant). IMO since this is more
cleaner approach, than deciding with convention of having specific
'TenantId' property name. But sending the tenantId in event payload is not
going to have any difference in the analytics perspective, because once the
relevant data is there fore relevant tenant space it's tenant's preference
to deploy the toolbox and analyze it.

Thanks,
Sinthuja.

Thanks,
NuwanD.

>
> On Tue, May 12, 2015 at 11:38 PM, Sinthuja Ragendran 
> wrote:
>
>> Hi all,
>>
>> We (Isabelle, Anjana, Suho, NuwanD,  Mohan, Iranga, Jaminda, and Myself)
>> had a discussion about APIM 1.9 compatibility with DAS last week, and we
>> need $subject to support APIM with DAS in tenant mode. As per APIM stats
>> publishing to BAM 2.5.0, the events are published from super tenant (for
>> all tenants) and having tenant id as separate field in the event. Therefore
>> in BAM 2.5.0 the events are pushed in cassandra the super-tenant space, and
>> toolbox and hive scripts runs on the super tenant space and having another
>> field for tenant id in the summarized table as well. Finally dashboards in
>> APIM will be connecting to one summarized table, and then filter results
>> based on the tenant id for the respective tenants to visualize their
>> statistics.
>>
>> DAS dashboard is going to live in DAS side rather APIM side, and hence
>> the respective tenant users can only connect to their respective tenant
>> data store. Therefore we need to have the statistics events to be pushed
>> into respective tenant space, rather the super tenant space.
>>
>> To cater the above requirement, we will have to introduce below approach
>> in DataPublisher.
>>
>> 1) Create data publisher with privileged super tenant user credentials.
>> 2) setTenantId or setTenantDomain when you want to send the data on
>> behalf of specific tenant.
>> 3) publish events as usual. If the events are being received by BAM 2.5.0
>> then it will simply ignore the tenantId/domain part, and it will simply
>> store the events in the logged in tenant space. In DAS the additional
>> tenantId/domain will be considered, and it will be inserted 

Re: [Architecture] Privileged user sending events to DAS/CEP as tenant

[Sinthuja Ragendran]

Hi sanjeewa,

On Tue, May 12, 2015 at 7:02 PM, Sanjeewa Malalgoda 
wrote:

>
>
> On Tue, May 12, 2015 at 6:07 PM, Sinthuja Ragendran 
> wrote:
>
>> Hi Nuwan,
>>
>> On Tue, May 12, 2015 at 5:32 PM, Nuwan Dias  wrote:
>>
>>> Aren't we tightly coupling the data publisher and the analyzer by this?
>>>
>>
>> No we are not coupling data publishing with analytics here; By the
>> proposed method, it's simply you can send the tenanted data to respective
>> tenant space, rather to single super tenant space.
>>
>> With the change you have proposed, a data published by a particular
>>> tenant would only be possible to analyze by the same tenant on DAS right?
>>>
>>
>> Yes, this is the same way currently other toolboxes behave. Basically the
>> tenant data will be only analyzed and visualized as per tenant preference,
>> therefore if the tenant foo.com prefers to visualize the data then
>> he/she should install the toolbox on DAS. Ideally IMHO there should be a
>> way to enable to publishing/Monitoring per tenant basis at APIM than
>> enabling globally for all tenants.
>>
> I believe still we can write data publisher as we need and we can embed
> tenant ID to message payload.
>

Yes, ofcourse. There is no lookup/validation on event payload or any of the
event attributes. You can send any type of events you want.

Or is it something inherit from base class in a way data publisher
> implementation cannot override.
>

The below is the example usage of the data publisher with proposed change.

DataPublisher dataPublisher = new DataPublisher(url, username, password);
dataPublisher.setTenantDomain("foo.com");
dataPublisher.defineStream("streamDefn");
dataPublisher.publish(new Event( streamId, System.currentTimeMillis(),
new Object[]{"external"}, null,
new Object[]{"test1", "test2"}));


Here a datapublisher object is specific to a tenant. Hence in multi tenant
case, you will have to create a separate data publishers for each tenant
and keep it in a map at APIM usege agent side, (Ex : Key - tenantDomain,
Value - DataPublisher) and then reuse the DataPublisher to send the events
for relevant tenants. Main advantage here is, you can send to relevant
tenant space, you don't need the tenant credentials rather with one valid
privileged user credentials you can send it to respective tenant's space..

Thanks,
Sinthuja

If that is the case tenant ID will be picked automatically and send along
> with payload.
>
> If we can set tenant ID in data publisher side then users can publish
> event to other tenants space(event receiver may see same and push data
> based on tenant ID present in payload).
> How do we address that?
>

> Thanks,
> sanjeewa.
>
>>
>>
>>> If that is the case it would cause us to live within a bunch of
>>> restrictions which might hinder some of our use cases. See below for some
>>> scenarios.
>>>
>>> 1. The analyzer script will have to run on each tenant space, isn't it?
>>> If so does someone need to manually deploy/configure it per each tenant?
>>>
>>
>> Deployment of toolbox is required on DAS.
>>
>>>
>>> 2. In API Manager, we have use cases where a particular tenant's API can
>>> be consumed by other tenant's applications. In this case, the stats related
>>> to the API are shown under the API owning tenant's space. The stats related
>>> to the application are shown under the App owning tenant's space. But all
>>> stats are triggered from the API invocation. At the moment we separate out
>>> the data based on the information available on the event payload. So we can
>>> nicely summarize the data accordingly.
>>>
>>
>> You still can send the respective tenant data as however you extract
>> currently, but now it'll go to relevant tenant space, rather single super
>> tenant space.
>>
>>
>>>
>>> If we do the determination of the tenant based on a value in the event
>>> payload, would it not decouple the publisher and analyzer and cater for
>>> more possibilities in terms of data summarization/analytics?
>>>
>>
>> Having the tenant id in the payload, and setting it in the data publisher
>> doesn't have any difference in the analytics as far as both are going to be
>> stored in respective tenant space.
>>
>> Here the only difference is in the publishing method, where you need to
>> have separate publishers created for every tenant rather using the same
>> publisher (you can have a map for tenantId, and data

Re: [Architecture] Privileged user sending events to DAS/CEP as tenant

[Sinthuja Ragendran]

Hi sanjeewa,

On Tue, May 12, 2015 at 7:39 PM, Sanjeewa Malalgoda 
wrote:

>
>
> On Tue, May 12, 2015 at 7:18 PM, Sinthuja Ragendran 
> wrote:
>
>> Hi sanjeewa,
>>
>> On Tue, May 12, 2015 at 7:02 PM, Sanjeewa Malalgoda 
>> wrote:
>>
>>>
>>>
>>> On Tue, May 12, 2015 at 6:07 PM, Sinthuja Ragendran 
>>> wrote:
>>>
>>>> Hi Nuwan,
>>>>
>>>> On Tue, May 12, 2015 at 5:32 PM, Nuwan Dias  wrote:
>>>>
>>>>> Aren't we tightly coupling the data publisher and the analyzer by
>>>>> this?
>>>>>
>>>>
>>>> No we are not coupling data publishing with analytics here; By the
>>>> proposed method, it's simply you can send the tenanted data to respective
>>>> tenant space, rather to single super tenant space.
>>>>
>>>> With the change you have proposed, a data published by a particular
>>>>> tenant would only be possible to analyze by the same tenant on DAS right?
>>>>>
>>>>
>>>> Yes, this is the same way currently other toolboxes behave. Basically
>>>> the tenant data will be only analyzed and visualized as per tenant
>>>> preference, therefore if the tenant foo.com prefers to visualize the
>>>> data then he/she should install the toolbox on DAS. Ideally IMHO there
>>>> should be a way to enable to publishing/Monitoring per tenant basis at APIM
>>>> than enabling globally for all tenants.
>>>>
>>> I believe still we can write data publisher as we need and we can embed
>>> tenant ID to message payload.
>>>
>>
>> Yes, ofcourse. There is no lookup/validation on event payload or any of
>> the event attributes. You can send any type of events you want.
>>
>> Or is it something inherit from base class in a way data publisher
>>> implementation cannot override.
>>>
>>
>> The below is the example usage of the data publisher with proposed
>> change.
>>
>> DataPublisher dataPublisher = new DataPublisher(url, username, password);
>> dataPublisher.setTenantDomain("foo.com");
>> dataPublisher.defineStream("streamDefn");
>> dataPublisher.publish(new Event( streamId, System.currentTimeMillis(), new 
>> Object[]{"external"}, null,
>> new Object[]{"test1", "test2"}));
>>
>>
>> Here a datapublisher object is specific to a tenant. Hence in multi
>> tenant case, you will have to create a separate data publishers for each
>> tenant and keep it in a map at APIM usege agent side, (Ex : Key -
>> tenantDomain, Value - DataPublisher) and then reuse the DataPublisher to
>> send the events for relevant tenants. Main advantage here is, you can send
>> to relevant tenant space, you don't need the tenant credentials rather with
>> one valid privileged user credentials you can send it to respective
>> tenant's space..
>>
>> Thanks for the clarification sinthuja. Yes it seems this is really cool
> feature.
> And if this works as you mentioned we can still publish events(all events
> belong to tenant) to super tenant space if need and have single
> toolbox(deployed in super tenant space) to summarize them.
>

Yes. But DAS analytics and dashboard will be operating on tenant basis,
therefore if you send all events to super tenant, then only super tenant
can see from the dashboard if the summarized results go back to DAS's data
store.


> And if someone interested per tenant publishing and analyze we may do that
> as well.
>
> And small question, if we take tenant scenario let say tenant need to have
> their own data publisher. Is that something possible? Still they need admin
> credentials.
>

Tenants always can use their own user name and password, and publish the
events which will finally go into their tenant space. They can't actually
use the above proposed model to switch between the tenants, because
switching between tenants is only permissible for super tenant who has a
valid permission, therefore they can't switch the tenant space and send it.
But of course, they can send to their own tenant space.

Thanks,
Sinthuja.

Thanks,
sanjeewa.


Thanks,
> Sinthuja
>
> If that is the case tenant ID will be picked automatically and send along
>> with payload.
>>
>> If we can set tenant ID in data publisher side then users can publish
>> event to other tenants space(event receiver may see same and push data
>> based on tenant ID present in payload).
>> How do we address that?
>>
>
>> Thanks,
>> s

Re: [Architecture] Privileged user sending events to DAS/CEP as tenant

[Sinthuja Ragendran]

Hi all,

We (anjana and me) had an internal discussion of going forward with APIM
1.9 with DAS. Based on some recent discussions with sanjiva et all, there
were some changes to DAS analytics where it should be able to write into
other external RDBMS databases as default spark, other than DAL (Data
Access Layer of DAS), and this external data will be in human readable
format so that we can plug any dahboard/visualization layer. In our initial
plan, we only supported DAL to be datastore, and therefore APIM dashboard
will not work OOB as it will require to use some different APIs, and can't
use simply JDBC to connect to the datastore directly. But given that now
DAS will be able to insert data into other external databases in human
readable format, APIM dashboard will work OOB if the spark scripts are
written using the relevant spark connectors.

Below is the summarization for complying APIM 1.9 with DAS 3.0.

1) APIM 1.9 will continue to push all tenant data into in super tenant
space, NO change required in the publishing method.
2) During DAS 3.0 release we will make sure the other RDBMS connectors
supported by spark will work.
3) Once DAS 3.0 is released, APIM scripts should be migrated to spark
script by using the relavant RDBMS connectos as mentioned in #2 and comply
with same summarized table format of APIM statistics.
4) APIM 1.9 dashboard will be still used to visualize the summarized data,
and DAS dashboard cannot be used for APIM 1.9.

Please raise if you have any concerns in above approach.

Thanks,
Sinthuja.

On Wed, May 13, 2015 at 9:50 AM, Nuwan Dias  wrote:

> Hi all,
>
> So at least for now API Manager and APP Manager have use cases in which
> the summarized data span across tenants. An example scenario would be an
> API of tenant A being consumed by tenant B's application. In this case
> tenant B's dashboard needs to show his application data and tenant A's
> dashboard needs to show the API invocation data. The event is published by
> tenant A's API.
>
> If we were to use the DAS dashboard to achieve the above, what should be
> the recommended way of getting it done? i.e. What would be the best place
> to publish the data to and where should the analytics run?
>
> Thanks,
> NuwanD.
>
> On Wed, May 13, 2015 at 2:18 AM, Sinthuja Ragendran 
> wrote:
>
>> Hi sanjeewa,
>>
>> On Tue, May 12, 2015 at 7:39 PM, Sanjeewa Malalgoda 
>> wrote:
>>
>>>
>>>
>>> On Tue, May 12, 2015 at 7:18 PM, Sinthuja Ragendran 
>>> wrote:
>>>
>>>> Hi sanjeewa,
>>>>
>>>> On Tue, May 12, 2015 at 7:02 PM, Sanjeewa Malalgoda 
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, May 12, 2015 at 6:07 PM, Sinthuja Ragendran >>>> > wrote:
>>>>>
>>>>>> Hi Nuwan,
>>>>>>
>>>>>> On Tue, May 12, 2015 at 5:32 PM, Nuwan Dias  wrote:
>>>>>>
>>>>>>> Aren't we tightly coupling the data publisher and the analyzer by
>>>>>>> this?
>>>>>>>
>>>>>>
>>>>>> No we are not coupling data publishing with analytics here; By the
>>>>>> proposed method, it's simply you can send the tenanted data to respective
>>>>>> tenant space, rather to single super tenant space.
>>>>>>
>>>>>> With the change you have proposed, a data published by a particular
>>>>>>> tenant would only be possible to analyze by the same tenant on DAS 
>>>>>>> right?
>>>>>>>
>>>>>>
>>>>>> Yes, this is the same way currently other toolboxes behave. Basically
>>>>>> the tenant data will be only analyzed and visualized as per tenant
>>>>>> preference, therefore if the tenant foo.com prefers to visualize the
>>>>>> data then he/she should install the toolbox on DAS. Ideally IMHO there
>>>>>> should be a way to enable to publishing/Monitoring per tenant basis at 
>>>>>> APIM
>>>>>> than enabling globally for all tenants.
>>>>>>
>>>>> I believe still we can write data publisher as we need and we can
>>>>> embed tenant ID to message payload.
>>>>>
>>>>
>>>> Yes, ofcourse. There is no lookup/validation on event payload or any of
>>>> the event attributes. You can send any type of events you want.
>>>>
>>>> Or is it something inherit from base class in a way data publisher
>>>>> implementation cannot override.
>>>>

Re: [Architecture] Privileged user sending events to DAS/CEP as tenant

[Sinthuja Ragendran]

Hi NuwanD,


On Wed, May 13, 2015 at 3:35 PM, Nuwan Dias  wrote:

> This plan looks good to me.
>
> We should also look beyond APIM 1.9.0 as well. I would personally like it
> if DAS had support for cross tenant scenarios as well. At least as a
> roadmap item, so we could use the DAS dashboards in the future versions of
> API Manager.
>

Yes, our initial plan to support publishing to different tenants with
special user will be introduced to new datapublisher, therefore in APIM 2.0
you can use that and send it to relevant tenant's space. Lets have a
meeting on this, post APIM 1.9 release and properly design monitoring
solution with DAS dashboard support.

Thanks,
Sinthuja.

Thanks,
NuwanD.

>
> On Wed, May 13, 2015 at 7:51 PM, Sinthuja Ragendran 
> wrote:
>
>> Hi all,
>>
>> We (anjana and me) had an internal discussion of going forward with APIM
>> 1.9 with DAS. Based on some recent discussions with sanjiva et all, there
>> were some changes to DAS analytics where it should be able to write into
>> other external RDBMS databases as default spark, other than DAL (Data
>> Access Layer of DAS), and this external data will be in human readable
>> format so that we can plug any dahboard/visualization layer. In our initial
>> plan, we only supported DAL to be datastore, and therefore APIM dashboard
>> will not work OOB as it will require to use some different APIs, and can't
>> use simply JDBC to connect to the datastore directly. But given that now
>> DAS will be able to insert data into other external databases in human
>> readable format, APIM dashboard will work OOB if the spark scripts are
>> written using the relevant spark connectors.
>>
>> Below is the summarization for complying APIM 1.9 with DAS 3.0.
>>
>> 1) APIM 1.9 will continue to push all tenant data into in super tenant
>> space, NO change required in the publishing method.
>> 2) During DAS 3.0 release we will make sure the other RDBMS connectors
>> supported by spark will work.
>> 3) Once DAS 3.0 is released, APIM scripts should be migrated to spark
>> script by using the relavant RDBMS connectos as mentioned in #2 and comply
>> with same summarized table format of APIM statistics.
>> 4) APIM 1.9 dashboard will be still used to visualize the summarized
>> data, and DAS dashboard cannot be used for APIM 1.9.
>>
>> Please raise if you have any concerns in above approach.
>>
>> Thanks,
>> Sinthuja.
>>
>> On Wed, May 13, 2015 at 9:50 AM, Nuwan Dias  wrote:
>>
>>> Hi all,
>>>
>>> So at least for now API Manager and APP Manager have use cases in which
>>> the summarized data span across tenants. An example scenario would be an
>>> API of tenant A being consumed by tenant B's application. In this case
>>> tenant B's dashboard needs to show his application data and tenant A's
>>> dashboard needs to show the API invocation data. The event is published by
>>> tenant A's API.
>>>
>>> If we were to use the DAS dashboard to achieve the above, what should be
>>> the recommended way of getting it done? i.e. What would be the best place
>>> to publish the data to and where should the analytics run?
>>>
>>> Thanks,
>>> NuwanD.
>>>
>>> On Wed, May 13, 2015 at 2:18 AM, Sinthuja Ragendran 
>>> wrote:
>>>
>>>> Hi sanjeewa,
>>>>
>>>> On Tue, May 12, 2015 at 7:39 PM, Sanjeewa Malalgoda 
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, May 12, 2015 at 7:18 PM, Sinthuja Ragendran >>>> > wrote:
>>>>>
>>>>>> Hi sanjeewa,
>>>>>>
>>>>>> On Tue, May 12, 2015 at 7:02 PM, Sanjeewa Malalgoda <
>>>>>> sanje...@wso2.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, May 12, 2015 at 6:07 PM, Sinthuja Ragendran <
>>>>>>> sinth...@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi Nuwan,
>>>>>>>>
>>>>>>>> On Tue, May 12, 2015 at 5:32 PM, Nuwan Dias 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Aren't we tightly coupling the data publisher and the analyzer by
>>>>>>>>> this?
>>>>>>>>>
>>>>>>>>
>>>>>>>> No we are not coupling data publishing with analytics here; By the
>>>>>>

Re: [Architecture] [DAS][APIM] DAS REST API evaluation for use with APIM

[Sinthuja Ragendran]

Hi,

One of the main feature of DAS is being decoupled from underline datastore,
basically an user should be able to configure RDBMS, cassandra,  Hbase/HDFS
datastore, the client applications nor analyzation scripts are not
necessarily need to change.

In APIM 1.9.x and before, we stored the summarized data back into RDBMS to
be supported with both BAM and DAS. But when we are moving forward with
APIM, IMHO we need to optimize the design based on DAS.

On Tue, Jul 28, 2015 at 11:02 AM, Rukshan Premathunga 
wrote:

> Hi all,
>
> We hope to implement a REST API for WSO2 API manager to expose APIM
> statistic data. For that we evaluated WSO2 DAS REST API to check whether we
> can use it.
> Here are the use cases of the APIM REST API and evaluation result of DAS
> REST API.
>
> *Motivation*:
>
> Currently we use DAS/BAM to generate summarised data and they are stored
> in an RDBMS.
> Next, APIM fetch data from the RDBMS and display it on the dashboard. APIM
> has implemented Client which fetch the statistic data from the RDBMS, which
> is used by the UI to show statistics on the dashboard.
>
> But we have a new requirement to providing these statistics for a bill
> generating application. Since application is separate, we are facing
> problems of expose these statistic data to the application.
> As a solution we suggest implementing REST API, which can be used to
> support above two scenarios (i.e. Current APIM dashboard and the billing
> application).
>
> Other advantages.
>
>
>- Implementing stat dashboard using REST API
>   - Client side data processing getting reduced
>   - Possibilities to expose statistic data to others
>   - Possible to implement appropriate security model
>
>
> *DAS REST API*
>
> DAS also contain the REST api where we can use it to expose analytics data
> of the DAS data analytics layer(DAL). Currently we use DAS to generate
> summarised data of the APIM statistics.
>
> But summarised data are saved in RDBMS since it was easy to fetch data.
> But with DAS they have provide the REST API to access DAL data.
> Because of this if we used DAS rest api, we can generate and store
> summarized data on DAS without using RDBMS.
>
> This will reduce the APIM configuration for statistics and central point
> of DATA storing and processing.
>
>
> Other advantages.
>
>
>- Reduce cost to configure RDBMS
>   - APIM will depend only on DAS
>
>
> *Issues with DAS REST API*
>
> Even though DAS provide the REST API to fetch DAL data, query on DAL data
> is limited. It provide search mechanism with apache Lucene, but it is
> insufficient for APIM requirements.
>
>
> Here are some query examples we used on mysql.
>
> SELECT api,version,apiPublisher,context,SUM(total_fault_count) as
> total_fault_count FROM  `table` WHERE time BETWEEN fromDate AND toDate
> GROUP BY api,version,apiPublisher,context;
>
>
> Here there is 3 main sub functions and queries
>
>
>- Time between- searching possible with apache Lucene range search[1]
>   - Sum - Not possible to calculate the summation
>   - Group by - impossible to grouping and make the calculation like
>   SUM, AVG
>
>
AFAIR you can use FACET type here which connects necssary fields
api,version,apiPublisher,context together, and then also you can pass a
function which is applied on the FACET search result. @Gimantha, please
confirm whether it's possible in DAS. - I know we can get the total rows
count of facet search, but I'm not very sure about passing the operation to
be done on the search result.

Thanks,
Sinthuja.


> With all of these advantages we hope to use to DAS REST API without
> implementing APIM own REST API. But to take the advantages of the
> REST API we have to have a REST API with above queries supported.
>
> Can you please provide a proper way to handle these scenarios?
> Any suggestions are appreciated.
>
> [1] https://lucene.apache.org/core/2_9_4/queryparsersyntax.html#Range
> Searches
>
> Thanks and Regards.
> --
> Rukshan Chathuranga.
> Software Engineer.
> WSO2, Inc.
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Associate Technical Lead
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [DAS][APIM] DAS REST API evaluation for use with APIM

[Sinthuja Ragendran]

Hi Gimantha,

Thanks for the clarification.

AFAIU APIM spark queries will be anyhow executed and summarized into DAS
tables, and the above mentioned sample query is something going to be
executed against the summarized data dynamically according to user's input.
For example, if the use slides the time-range between some values in APIM,
the the gadgets in the APIM dashboard needs to be updated with SUM of
requests, AVG, etc by filtering the summarized data further for the given
time range. I don't think issuing an SparkSQL query dynamically for such
dynamic changes is viable solution.

Thanks,
Sinthuja.

On Tue, Jul 28, 2015 at 1:15 PM, Gimantha Bandara  wrote:

> [Adding Niranda]
>
> Hi Sinduja/Rukshan,
>
> Yes this can be achieved using facets, There are several facets related
> APIs. One is to get the Facet record count. Default behavior is to return
> the number of matching records given the Facet array. We can override the
> default behavior by defining a score function. So each record will
> represent the value of the score function. So id we query for the facet
> count, it will simply return the sum of score function values. This
> behavior is similar to SUM aggregate. But for Aggregates like AVG we need
> the current number of records to calculate the average. We will need to
> make two REST calls ( to get the record count with default score function
> "1" and to get the SUM another record count with the score function
> "aggregating field") since aggregates like AVG are not supported.
>
> As Gihan said, I think Spark SQL will more suitable for this scenario. It
> support basic aggregates and if a custom function required, we can write a
> UDF.
>
> Thanks,
>
> On Tue, Jul 28, 2015 at 11:27 AM, Sinthuja Ragendran 
> wrote:
>
>> Hi,
>>
>> One of the main feature of DAS is being decoupled from underline
>> datastore, basically an user should be able to configure RDBMS, cassandra,
>> Hbase/HDFS datastore, the client applications nor analyzation scripts are
>> not necessarily need to change.
>>
>> In APIM 1.9.x and before, we stored the summarized data back into RDBMS
>> to be supported with both BAM and DAS. But when we are moving forward with
>> APIM, IMHO we need to optimize the design based on DAS.
>>
>> On Tue, Jul 28, 2015 at 11:02 AM, Rukshan Premathunga 
>> wrote:
>>
>>> Hi all,
>>>
>>> We hope to implement a REST API for WSO2 API manager to expose APIM
>>> statistic data. For that we evaluated WSO2 DAS REST API to check whether we
>>> can use it.
>>> Here are the use cases of the APIM REST API and evaluation result of DAS
>>> REST API.
>>>
>>> *Motivation*:
>>>
>>> Currently we use DAS/BAM to generate summarised data and they are stored
>>> in an RDBMS.
>>> Next, APIM fetch data from the RDBMS and display it on the dashboard.
>>> APIM has implemented Client which fetch the statistic data from the RDBMS,
>>> which is used by the UI to show statistics on the dashboard.
>>>
>>> But we have a new requirement to providing these statistics for a bill
>>> generating application. Since application is separate, we are facing
>>> problems of expose these statistic data to the application.
>>> As a solution we suggest implementing REST API, which can be used to
>>> support above two scenarios (i.e. Current APIM dashboard and the billing
>>> application).
>>>
>>> Other advantages.
>>>
>>>
>>>- Implementing stat dashboard using REST API
>>>   - Client side data processing getting reduced
>>>   - Possibilities to expose statistic data to others
>>>   - Possible to implement appropriate security model
>>>
>>>
>>> *DAS REST API*
>>>
>>> DAS also contain the REST api where we can use it to expose analytics
>>> data of the DAS data analytics layer(DAL). Currently we use DAS to generate
>>> summarised data of the APIM statistics.
>>>
>>> But summarised data are saved in RDBMS since it was easy to fetch data.
>>> But with DAS they have provide the REST API to access DAL data.
>>> Because of this if we used DAS rest api, we can generate and store
>>> summarized data on DAS without using RDBMS.
>>>
>>> This will reduce the APIM configuration for statistics and central point
>>> of DATA storing and processing.
>>>
>>>
>>> Other advantages.
>>>
>>>
>>>- Reduce cost to configure RDBMS
>>>   - APIM will depend only on DAS
>>>
>>>

Re: [Architecture] Basic WSO2 Developer Studio tooling support for BAM

[Sinthuja Ragendran]

On Mon, Jun 24, 2013 at 10:55 AM, Ayashkantha Ramasinghe <
ayashkan...@wso2.com> wrote:

> Hi,
>
> The following features are going to be added in the initial stage of
> having the Developer Studio support to create toolboxes to BAM.
>
> 1. Hive Editor.
> 2. Jaggery Editor.
> 3. UI to create Stream-Definitions.
> 4. Packaging the toolbox as a carbon application and deploy it in the
> server.
>

+1, there is already Jaggery IDE integrated to DevStudio recently. I
believe you will be using that as a Jaggery editor.

Thanks,
Sinthuja.

>
> Initially, the editors are not much functional, but they are just text
> editors with syntax highlighting.
>
> Thank you,
>
> --
> Ayashkantha Ramasinghe
> Software Engineer
>
> Tel: +94 777 487 669
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Software Engineer 
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Need to upgrade to new Cassandra

[Sinthuja Ragendran]

We need to upgrade the hive also. Currently in BAM we are using the hive
0.8.1, but the latest release is 0.11.0.

Latest hive uses the thrift version - 0.9 but the cassandra uses thrfit
version - 0.7 which may cause incompatibility of both working together in
BAM. In past we had similar problem and we could fortunately have same
thrift version by upgrading the cassandra and hive to latest versions.

But the thrift version is upgraded in Cassandra trunk [2], but there wasn't
any relaese with this yet. So if we upgrade the cassandra now, we have to
upgrade again once there is a release with [2], along with the hive upgrade.


[1] http://hive.apache.org/releases.html#Download
[2] https://issues.apache.org/jira/browse/CASSANDRA-3719

Thanks,
Sinthuja.


On Mon, Jul 1, 2013 at 2:33 PM, Isuru Perera  wrote:

> Latest Cassandra release is 1.2.6 [1]. We should be able to directly
> upgrade to 1.2.6, right?
>
> [1] http://cassandra.apache.org/
>
>
> On Mon, Jul 1, 2013 at 1:21 PM, Srinath Perera  wrote:
>
>> $Subject, we are in 1.1.3 vs. Cassandra is now in 1.2.4, and have lot of
>> improvements, and likely switch should just work.
>>
>>
>
>> Apply to MB, BAM
>>
>> --Srinath
>>
>> --
>> 
>> Srinath Perera, Ph.D.
>>   Director, Research, WSO2 Inc.
>>   Visiting Faculty, University of Moratuwa
>>   Member, Apache Software Foundation
>>   Research Scientist, Lanka Software Foundation
>>   Blog: http://srinathsview.blogspot.com/
>>   Photos: http://www.flickr.com/photos/hemapani/
>>Phone: 0772360902
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Isuru Perera
> Senior Software Engineer | WSO2, Inc. | http://wso2.com/
> Lean . Enterprise . Middleware
>
> Twitter: http://twitter.com/chrishantha | LinkedIn:
> http://lk.linkedin.com/in/chrishantha/
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Software Engineer 
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Annotation scheme for Hive scripts

[Sinthuja Ragendran]

Whether the annotation variables' scope is within hive query? or it's
within the hive script? or it's global scope such that all hive scripts can
use it,

For an example there might be situations where user is interested in
defining some variables to the entire script, and some are local to the
hive query. So it is useful to have different scope also for the annotation.

May be we can say,

@foo -> all scripts scope
@script.foo -> scope within the script
@query.foo -> scope within the query.

I believe you will be setting this annotation variables withing the
HiveContext, which basically holds the context for the current tenant.

Thanks,
Sinthuja.


On Tue, Jul 9, 2013 at 11:23 AM, Malith Dhanushka  wrote:

> Hi all,
>
> I have started implementing the $Subject. The idea of having an annotation
> facility is to carryout some pre-processing of Hive queries before they are
> being passed to the Hive engine. Currently we already have a "class
> analyzer" which can be used execute some custom logic as a part of a Hive
> script. But the main use case of annotations is to inject run-time
> properties to Hive execution context before the actual queries are carried
> out by Hive. The annotation facility would be building upon this by having
> set of such common analyzers which can manipulate the Hive queries or Hive
> execution context which it is passed to Hive query engine.
>
> Annotation Syntax,
>
> *@script.foo(bar="value", bar1="value1",*)*
>
>
> Annotation scheme will be externalized by giving *abstract implementation
> of annotation* and *annotation-config.xml* file to provide the annotation
> configuration which allows third party annotations to be included to the
> system.
>
> *annotation-config.xml*
>
> 
> foo
> org.wso2.carbon.analytics.hive.extension.annotation.foo
> org.wso2.carbon.analytics.hive.extension.foo
> 
>
> 
> 
> 
>
>
> Potential use case for this in incremental data processing where any query
> associated with "*@script.incremental(foo="value1", bar="value2",*)*"
> would flag and setup the properties those are required to present in order
> for that particular query to be executed in an incremental manner.There can
> be many other useful additions as well.
>
> Any suggestions, thoughts are welcome.
>
> --
> Malith Dhanushka
>
> Engineer - Data Technologies
> *WSO2, Inc. : wso2.com*
>
> *Mobile*  : +94 716 506 693
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Software Engineer 
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] JMX Agent Discussion Notes

[Sinthuja Ragendran]

On Thu, Jul 18, 2013 at 3:26 PM, Sriskandarajah Suhothayan wrote:

> We can make this a Input Transport Adaptor that returns WSO2Events as
> output, then we can have one agent common for both CEP and BAM,
>
> WDYT?
>

+1. Alerting will be incorporated with KPI monitoring feature for BAM. And
during that effort, we'll use the common components from CEP much as as
possible.

Thanks,
Sinthuja.

>
> Thanks
> Suho
>
>
> On Thu, Jul 18, 2013 at 3:17 PM, Maninda Edirisooriya wrote:
>
>> The most useful scenario I guess is alerting. For example, when one node
>> consumes more than 90% of CPU or 80% of memory it can notify admins via
>> e-mails. This will be possible after integrating CEP in BAM.
>>
>> *
>> Maninda Edirisooriya*
>> Software Engineer
>> *WSO2, Inc.
>> *lean.enterprise.middleware.
>>
>> *Blog* : http://maninda.blogspot.com/
>> *Phone* : +94 777603226
>>
>>
>> On Thu, Jul 18, 2013 at 8:41 AM, Srinath Perera  wrote:
>>
>>> JConsole like view should give a multi node view when applicable.
>>>
>>>  e.g. If it shows CPU use, show all nodes data in a one graph with a
>>> line for each node etc.
>>>
>>> --Srinath
>>>
>>>
>>> On Wed, Jul 17, 2013 at 9:18 PM, Sanjiva Weerawarana 
>>> wrote:
>>>
 Two minor comments to the notes:

 - The receiving is a "JMX Polling Data Receiver" .. so its an "active"
 receiver that polls and pumps in messages
  - For KPIs- for standard MBeans we should have the same level (or
 better) of OOTB analytics and visualizations that say JConsole has.

 Sanjiva.


 On Wed, Jul 17, 2013 at 8:40 PM, Srinath Perera wrote:

>  Participants: Sanjiva, BAM Team, Srinath
>
>1. We will write a JMX receiver in BAM, which will pull the JMX
>properties periodically from given list of servers.
>2. Each JMX MBean property we interested, we provide a name
>3. We will have few JMX properties enabled by default, users can
>add remove these.
>4. Adding a reciver add a Stream definitions
>
> To use the data, we follow following model
>
>1. Users go and add a new KPI
>2. Either through a visual wizard or a input box, we generate or
>write a HIVE/CEP queries to calculate the KPIs
>3. Users can also define Gadgets presenting the KPI and can add
>alerts while defining a KPIs
>
>
> Toolboxes define KPIs , some of which available by default and others
> custom written
>
> --Srinath
>
>
>
>
> --
> 
> Srinath Perera, Ph.D.
>http://people.apache.org/~hemapani/
>http://srinathsview.blogspot.com/
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


 --
 Sanjiva Weerawarana, Ph.D.
 Founder, Chairman & CEO; WSO2, Inc.;  http://wso2.com/
 email: sanj...@wso2.com; phone: +94 11 763 9614; cell: +94 77 787 6880| +1
 650 265 8311
 blog: http://sanjiva.weerawarana.org/

 Lean . Enterprise . Middleware

 ___
 Architecture mailing list
 Architecture@wso2.org
 https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


>>>
>>>
>>> --
>>> 
>>> Srinath Perera, Ph.D.
>>>http://people.apache.org/~hemapani/
>>>http://srinathsview.blogspot.com/
>>>
>>> ___
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> *S. Suhothayan
> *
> Associate Technical Lead,
>  *WSO2 Inc. *http://wso2.com *
>  *
> lean . enterprise . middleware
>
> *cell: (+94) 779 756 757 | blog: http://suhothayan.blogspot.com/
> twitter: http://twitter.com/suhothayan | linked-in:
> http://lk.linkedin.com/in/suhothayan*
> *
> *
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Software Engineer 
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Mock up screens for BAM events & activity search

[Sinthuja Ragendran]

Hi,

+1 for Anjana's suggestion for activity search.

For the event search isn't it better that we make the UI generic for
cassandra data search, rather stream's events search?

Ie, instead of StreamName we can provide the Columnfamily name, so on. And
version field also can be omitted, if it's required to be considered into
the search criteria, then the user can give that in the property list.

WDYT?

Thanks,
Sinthuja.



On Thu, Jul 25, 2013 at 4:30 PM, Anjana Fernando  wrote:

> Hi,
>
> In the search, the part where we give the conditions, this approach
> doesn't handle the ordering of the conditions, like (stream1.prop1="1" or
> stream2.prop2="2") and (stream1.prop3="3") kind of a scenario. So how about
> having a simple search query format instead and just give a search text
> field to do the queries? .. I guess we can use a format like the one I just
> mentioned for the earlier example.
>
> Cheers,
> Anjana.
>
>
> On Thu, Jul 25, 2013 at 4:03 PM, Srinath Perera  wrote:
>
>> 1) For Event search, user need to pick an stream, then he can search.
>> Given we have activity search I think that would do.
>>
>> 2) For activity search, this only handle a small subset. One activity
>> might have events from many streams. We define property search to match if
>> at least one event in an activity matches the given condition (e.g.
>> property). If user need more complex ones (e.g. temporal conditions) they
>> should define and KPI.
>>
>> --Srinath
>>
>>
>> On Thu, Jul 25, 2013 at 3:54 PM, Inosh Goonewardena wrote:
>>
>>> Hi,
>>>
>>> We had a few discussion sessions with Srinath about BAM events search
>>> and activity search.
>>>
>>> Herewith I have attached mock up screens on how the search UIs will be
>>> like. Any feedback is appreciated.
>>>
>>> --
>>> Regards,
>>>
>>> Inosh Goonewardena
>>> Associate Technical Lead- WSO2 Inc.
>>> Mobile: +94779966317
>>>
>>
>>
>>
>> --
>> 
>> Srinath Perera, Ph.D.
>>http://people.apache.org/~hemapani/
>>http://srinathsview.blogspot.com/
>>
>
>
>
> --
> *Anjana Fernando*
> Technical Lead
>  WSO2 Inc. | http://wso2.com
> lean . enterprise . middleware
>



-- 
*Sinthuja Rajendran*
Software Engineer 
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Mock up screens for BAM events & activity search

[Sinthuja Ragendran]

Hi,

On Thu, Jul 25, 2013 at 5:23 PM, Anjana Fernando  wrote:

> Well .. this is not meant to be a Cassandra search .. it's an
> event/activity search, so we can't expose the underlying data store here
> and say, we are searching for raw data that's there ..
>

Fine, then we'll have it as event/streams based search. +1.

Thanks,
Sinthuja.

>
> Cheers,
> Anjana.
>
>
> On Thu, Jul 25, 2013 at 5:18 PM, Sinthuja Ragendran wrote:
>
>> Hi,
>>
>> +1 for Anjana's suggestion for activity search.
>>
>> For the event search isn't it better that we make the UI generic for
>> cassandra data search, rather stream's events search?
>>
>> Ie, instead of StreamName we can provide the Columnfamily name, so on.
>> And version field also can be omitted, if it's required to be considered
>> into the search criteria, then the user can give that in the property list.
>>
>> WDYT?
>>
>> Thanks,
>> Sinthuja.
>>
>>
>>
>> On Thu, Jul 25, 2013 at 4:30 PM, Anjana Fernando  wrote:
>>
>>> Hi,
>>>
>>> In the search, the part where we give the conditions, this approach
>>> doesn't handle the ordering of the conditions, like (stream1.prop1="1" or
>>> stream2.prop2="2") and (stream1.prop3="3") kind of a scenario. So how about
>>> having a simple search query format instead and just give a search text
>>> field to do the queries? .. I guess we can use a format like the one I just
>>> mentioned for the earlier example.
>>>
>>> Cheers,
>>> Anjana.
>>>
>>>
>>> On Thu, Jul 25, 2013 at 4:03 PM, Srinath Perera wrote:
>>>
>>>> 1) For Event search, user need to pick an stream, then he can search.
>>>> Given we have activity search I think that would do.
>>>>
>>>> 2) For activity search, this only handle a small subset. One activity
>>>> might have events from many streams. We define property search to match if
>>>> at least one event in an activity matches the given condition (e.g.
>>>> property). If user need more complex ones (e.g. temporal conditions) they
>>>> should define and KPI.
>>>>
>>>> --Srinath
>>>>
>>>>
>>>> On Thu, Jul 25, 2013 at 3:54 PM, Inosh Goonewardena wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> We had a few discussion sessions with Srinath about BAM events search
>>>>> and activity search.
>>>>>
>>>>> Herewith I have attached mock up screens on how the search UIs will be
>>>>> like. Any feedback is appreciated.
>>>>>
>>>>> --
>>>>> Regards,
>>>>>
>>>>> Inosh Goonewardena
>>>>> Associate Technical Lead- WSO2 Inc.
>>>>> Mobile: +94779966317
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> 
>>>> Srinath Perera, Ph.D.
>>>>http://people.apache.org/~hemapani/
>>>>http://srinathsview.blogspot.com/
>>>>
>>>
>>>
>>>
>>> --
>>> *Anjana Fernando*
>>> Technical Lead
>>>  WSO2 Inc. | http://wso2.com
>>> lean . enterprise . middleware
>>>
>>
>>
>>
>> --
>> *Sinthuja Rajendran*
>> Software Engineer <http://wso2.com/>
>> WSO2, Inc.:http://wso2.com
>>
>> Blog: http://sinthu-rajan.blogspot.com/
>> Mobile: +94774273955
>>
>>
>>
>
>
> --
> *Anjana Fernando*
> Technical Lead
> WSO2 Inc. | http://wso2.com
> lean . enterprise . middleware
>



-- 
*Sinthuja Rajendran*
Software Engineer <http://wso2.com/>
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Car File Deployer - BAM

[Sinthuja Ragendran]

Hi Ayashkantha,

You should use the first approach IMO. We don't need duplicate the code for
deploying the toolboxes. It'll become unmanageable and it won't be
consistent also.

Generally we follow the first method, as cApp will copy the artifacts in
the respective servers' deployment directory, and the specific deployer
will handle the deployment logic for that artefact.

Thanks,
Sinthuja.


On Fri, Aug 2, 2013 at 1:11 PM, Ayashkantha Ramasinghe  wrote:

> Hi,
>
> The "BAM car file deployer" is created to deploy toolboxes in the server
> as a cApp. I have two approaches,
>
> 1. to copy the toolbox to bam-toolbox directly and let the toolbox
> deployer run automatically and deploy the toolbox.
> 2. to deploy the toolbox using the code while deploying cApp without
> copying the toolbox to bam-toolbox directory directly.
>
> In 1st approach, we can go and manually see the toolbox deployed inside
> the directory, bam-toolbox. But, in the 2nd approach, we can't see or
> undeploy it manually, only cApp undeployment does the undeployment of the
> toolbox.
>
> I am also thinking that the 1st approach is good, but, for now, the
> deployer is created based on the 2nd approach.
>
> WDYT???
>
> Thank you
>
> --
> Ayashkantha Ramasinghe
> Software Engineer
>
> Tel: +94 777 487 669
>



-- 
*Sinthuja Rajendran*
Software Engineer 
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [Dev][BAM] Defining stream properties to be indexed for BAM events & activity search

[Sinthuja Ragendran]

Hi Gihan,


On Wed, Aug 7, 2013 at 12:07 PM, Gihan Anuruddha  wrote:

> Hi Inosh,
>
> There can be a situation that we might need to use secondary indexes
> without any toolboxes.  How we define secondary indexes in such a situation
> ?
>

In the initial stages of discussion for this topic, this requirement came
out and whether it's required to have management console UI component where
you can select the stream definitions and mention the fields you wanted to
index. But just only indexes won't make sense without a use-case - toolbox,
and decided to have indicate this in the toolbox properties.

Thanks,
Sinthuja.


Regards,
Gihan

>
>
> On Tue, Aug 6, 2013 at 7:27 PM, Inosh Goonewardena  wrote:
>
>> Hi,
>>
>> I am working on implementing BAM events & activity search feature. To
>> support the search we need to index some predefined stream properties. We
>> can either use cassandra secondary indexes or we can use our own custom
>> indexes(secondary indexes are not recommended for columns having high
>> cardinality values). Anyway we need to defined the index properties
>> somehow, so we thought of define those in BAM toolbox.
>>
>> In BAM toolbox there is already a configuration file named
>> 'streams.properties'.
>> .tbox
>> |-- analytics 
>> .
>> .
>> |-- dashboard 
>> .
>> .
>> |-- streamDefn 
>> |-- defn_1 
>> |-- defn_2 
>> |-- *streams.properties *
>>
>> 'streams.properties' file is currently use to keep meta information about
>> stream definitions. So I thought of adding two new properties to
>> 'stream.properties' to keep indexing information like belows.
>>
>> *
>> streams.definitions.defn1.secondary.indexes=meta_host,payload_service_name,payload_operation_name
>> streams.definitions.defn1.custom.indexes=correlation_bam_activity_id*
>>
>> WDYT???
>>
>> --
>> Regards,
>>
>> Inosh Goonewardena
>> Associate Technical Lead- WSO2 Inc.
>> Mobile: +94779966317
>>
>
>
>
> --
> ---
> W.G. Gihan Anuruddha
> Senior Software Engineer | WSO2, Inc.
> M: +94772272595
>



-- 
*Sinthuja Rajendran*
Software Engineer 
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [Dev][BAM] Defining stream properties to be indexed for BAM events & activity search

[Sinthuja Ragendran]

Hi Gokul,


On Wed, Aug 7, 2013 at 1:46 PM, Balakrishnan Gokulakrishnan
wrote:

> Hi Inosh,
>
> IMO the index properties should ideally be defined at the time of the
> column family creation itself, and as the column family creation takes
> place based on the stream definition is it not logical to have the index
> properties there?
>

Stream definition is common for BAM and CEP. And we can't inject properties
into the common stream definition representation, that are only used by
BAM/Underlying implementation of receiver. Therefore it's not advisable to
add it in the stream definition.

Thanks,
Sinthuja.


>
> @Gayan yes, we will no longer be shipping the current bamdashboard from
> next release onwards.
>
> Thanks,
>
> On 7 August 2013 12:22, Gayan Dhanushka  wrote:
>
>> Hi Inosh & BAM team,
>>
>> With the BAM , UES integration in the picture are we going to continue
>> supporting the BAM dashboards as it was done earlier or are we going to
>> encourage creating dashboards using UES?
>>
>> +1 for using the latter for dashboards.
>>
>> Thanks.
>> Gayan
>>
>>
>>
>>
>> On Wed, Aug 7, 2013 at 12:07 PM, Gihan Anuruddha  wrote:
>>
>>> Hi Inosh,
>>>
>>> There can be a situation that we might need to use secondary indexes
>>> without any toolboxes.  How we define secondary indexes in such a situation
>>> ?
>>>
>>> Regards,
>>> Gihan
>>>
>>>
>>> On Tue, Aug 6, 2013 at 7:27 PM, Inosh Goonewardena wrote:
>>>
 Hi,

 I am working on implementing BAM events & activity search feature. To
 support the search we need to index some predefined stream properties. We
 can either use cassandra secondary indexes or we can use our own custom
 indexes(secondary indexes are not recommended for columns having high
 cardinality values). Anyway we need to defined the index properties
 somehow, so we thought of define those in BAM toolbox.

 In BAM toolbox there is already a configuration file named
 'streams.properties'.
 .tbox
 |-- analytics 
 .
 .
 |-- dashboard 
 .
 .
 |-- streamDefn 
 |-- defn_1 
 |-- defn_2 
 |-- *streams.properties *

 'streams.properties' file is currently use to keep meta information
 about stream definitions. So I thought of adding two new properties to
 'stream.properties' to keep indexing information like belows.

 *
 streams.definitions.defn1.secondary.indexes=meta_host,payload_service_name,payload_operation_name
 streams.definitions.defn1.custom.indexes=correlation_bam_activity_id*

 WDYT???

 --
 Regards,

 Inosh Goonewardena
 Associate Technical Lead- WSO2 Inc.
 Mobile: +94779966317

>>>
>>>
>>>
>>> --
>>> ---
>>> W.G. Gihan Anuruddha
>>> Senior Software Engineer | WSO2, Inc.
>>> M: +94772272595
>>>
>>> ___
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> --
>>
>> Gayan Dhanushka
>> Software Engineer
>> WSO2 Inc.
>> http://wso2.com
>>
>> Mobile : 0716662327
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> *Balakrishnan Gokulakrishnan*
> Software Engineer,
> WSO2, Inc.; http://wso2.com
>
> Twitter:  http://twitter.com/gokulbs
> Mobile: +94775935789
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Software Engineer 
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] BAM Data Archival Feature improvements

[Sinthuja Ragendran]

Hi,


On Wed, Sep 4, 2013 at 11:22 AM, Shariq Muhammed  wrote:

> On Wed, Sep 4, 2013 at 10:58 AM, Bhathiya Jayasekara wrote:
>
>> Hi all,
>>
>> How about deleting archived data? As I remember, this requirement came up
>> recently. IMO it's good to have a archive-delete feature which can
>> be scheduled.
>>
>
> Once the data is archived to an external Cassandra ring, we should have
> the option to delete the archived data from the primary Cassandra cluster
> (where raw stats are being published),
>

That is what happening now also, the data will be removed from the original
column family and archived to another column family. This feature is
basically purging and archiving. But yes, we can also have purging and
archiving seperately, and giving the user to select whether he/she needs,
purging/archiving/both.

so that the new analytics job doesn't have to process the already analyzed
> / old data ...  Or else time taken to complete map-reduce jobs will grow
> with the data growth ...
>

yes and  currently we are working on incremental processing feature for BAM
(refer thread 'Incremental Data Processing for BAM'), which will provide
the capability to process only the unprocessed data.

Thanks,
Sinthuja.

>
>> Thanks,
>> Bhathiya
>>
>>
>> On Wed, Sep 4, 2013 at 10:18 AM, Malith Dhanushka wrote:
>>
>>> Hi folks,
>>>
>>> We(BAM team, Sumedha) had a  discussion about the $Subject and
>>> following are the suggested improvements for the Cassandra data archival
>>> feature in BAM.
>>>
>>> - Remove hive script based archiving and use hector API to directly
>>> issue archive queries to Cassandra  (Current implementation is
>>> based on hive where it generates hive script and archiving process uses
>>> map-reduce jobs to achieve the task and it has a limitation of discarding
>>> custom key value pares in column family)
>>>
>>> - Use Task component for scheduling purposes
>>>
>>> - Archive data to external Cassandra ring
>>>
>>> - Major UI improvements
>>> - List the current archiving tasks
>>> - Edit, Remove and Schedule archiving tasks
>>> - Add new archiving task
>>>
>>> If there is any additional requirements please raise.
>>>
>>> Thanks,
>>> Malith
>>> --
>>> Malith Dhanushka
>>>
>>> Engineer - Data Technologies
>>> *WSO2, Inc. : wso2.com*
>>>
>>> *Mobile*  : +94 716 506 693
>>>
>>> ___
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> --
>> *Bhathiya Jayasekara*
>> *Software Engineer,*
>> *WSO2 inc., http://wso2.com*
>> *
>> *
>> *Phone: +94715478185*
>> *LinkedIn: http://www.linkedin.com/in/bhathiyaj*
>> *Twitter: https://twitter.com/bhathiyax*
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Thanks,
> Shariq.
> Phone: +94 777 202 225
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Software Engineer 
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Hive handler to push events to JMS queue

[Sinthuja Ragendran]

Hi Maninda,

We have output transport adopters which basically converts the streams to
the respective transports such as JMS, etc according the current
architecture. AFAIK there is already a JMS transport adopter. And therefore
we need to write a Hive handler to write to the streams, rather JMS queue.
In this model, we could leverage already existing output transport adopters
to output the hive results, not only for JMS queue.

WDYT?

Thanks,
Sinthuja.


On Tue, Sep 17, 2013 at 1:57 AM, Maninda Edirisooriya wrote:

> I am going to write a Hive handler that will send summarized data to JMS
> queue. This will be used to collect summarized data from Hive and publish
> them to a JMS queue / topic. This JMS topic will be the same topic where
> the CEP events will also be published.
>
> This topic will be visible to dashboards so that they can subscribe to a
> such a JMS topic according to the new BAM architecture.
> *
> Maninda Edirisooriya*
> Software Engineer
> *WSO2, Inc.
> *lean.enterprise.middleware.
>
> *Blog* : http://maninda.blogspot.com/
> *Phone* : +94 777603226
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
*Sinthuja Rajendran*
Software Engineer 
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

[Architecture] Re-factoring BAM/CEP Datapublisher

[Sinthuja Ragendran]

Hi all,

I'm in the process of merging the changes of AsyncDataPublisher with
DataPublisher and hence make our DataPublisher do asynchronous calls on
main operations; not only publish but also in connecting to receiver,
define stream, etc. This effort is for improve performance in client side
and reduce the complexity to chose between what data publisher to use in
users' perspective.

The below shown is the expected bahaviour of the re-factored asynchronous
DataPublisher.

API Synchronous Data Publisher Refactored Datapublisher (Async)
public DataPublisher(receiverURL, username, password);

Connects to the receiver during the object instantiation time. It's
blocking call, and if data receiver is down it halt until it timeout.


Connects to receiver asynchronously, and during the instantiation it
obtains the details and connect to the receiver in separate worker thread.
Therefore regardless of data receiver is up or down, it won't affect the
client and the invocation thread immediately returns back.

public String defineStream(StreamDefinition streamDefinition)






It's also blocking call, which halts until the stream is being defined
successfully. If we consider the BAM scenario, it'll be halted until it
save the stream definition in to cassandra. And if cassandra is
down/slow/network latency affects the client side. But the success based on
successful return of stream id OR failure based on
DifferentStreamDefinitionAlreadyDefinedException,
MalformedStreamDefinitionException, etc will be known to the client during
the invocation since it's synchronous call.
It adds the stream definition to the streamDefnCache and returns the
generated streamId based on stream name and version to invocation. During
the defineStream() invocation time, the stream wouldn't have actually
defined and it'll be only stored in the cache. It'll actually define the
stream, once the connection to the receiver was made successfully, and if
any event was triggered to publish for this stream. Hence client won't be
blocked at any casue during this, but always it'll be success and will not
be able propagate the exception since it's asynchronous.

public String defineStreamNow(StreamDefinition streamDefinition) Not
applicable

New method to do the defineStream() synchronous. Hence if the user want to
define stream as blocking call then they can use this.

public void publish(Event event)



















This is already asynchronouse. Invocation thread will put the event into
the queue and it'll return. Queue worker thread will operate on queue and
publish the events.


















Same as default behaviour here also asynchronous with additional
functionalities. Here the events will be buffered if the connection is
lost/ not established, and it'll automatically reconnect if the data
receiver is available back and send the buffered events which was collected
during that time period. Let say publish was called before the connection
is established to the receiver, then the events will be buffered (The
buffersize is configurable) and will be pusblished by a worker thread once
the connection is established. Also before actually publishing a event, we
need to make sure we have defined the stream. There are two caches used
here; streamDefnCache and streamIdCache. streamDefnCache will be having the
stream defintions which should be defined, and streamIdCache will have
streamIds after actually defining/saving the stream defition with receiver.




* The worker thread will first check whether streamId is existing in the
streamIdCache and if it's existing it'll just call publish the event. If
it's not existing in the streamIdCache, then it'll check whether there is
any stream needs to be defined before publishing the event from
streamDefnCache and if it's there it'll define the stream and remove the
streamDefintion from the streamDefnCache and update the streamId in the
streamIdCache. Here we need to decide what we are going to if the streamId
is not there in streamIdCache, stream defintion is also not there in the
streamDefnCache. The below are the possibilities,1) findStream() to check
whether it's already defined. If success send the events, else drop the
events in the client side it self. (Pros: Reduce unnecessary traffic in
receiver side for wrong events, Cons: findStream() is also expesive call
and untill the stream is defined all the events are going will undergo this
check)2) publish the events regardless of anything, and it'll dropped in
after reaching receiver and validating it before processing.*
public String findStream(String name, String version) Retrieves the
streamId for a stream.
Deprecating this API since streamId is combination streamname and version.

public static validateStreamDefinition(String streamDefn) Not applicable
This will be a synchronous call to validate the stream definition string
and whether there is already stream defined with same name, version.


Above are the major changes in the behaviour of the APIs. An

Re: [Architecture] Re-factoring BAM/CEP Datapublisher

[Sinthuja Ragendran]

Hi lasantha,

Thanks for your suggestions. Please see my comments inline.

On Friday, January 17, 2014, Lasantha Fernando  wrote:

> Hi Sinthuja,
>
> +1 for making an asynchronous defineStream() method as well as keeping a
> synchronous version. Would it make sense to keep the defineStream() method
> as it is and make the new method defineStreamAsync() or similar? Am
> suggesting this because AFAIK, some current implementations depend on
> defineStream() being synchronous and all those places might have to be
> refactored if the change is introduced this way.
>

IMO we should have the default behavior as asynchronous, if the user want
synchronous call then explicitly mention them as defineStreamNow(). Here
I'm hoping to have the method signatures as it's, hence we don't need to
change the current data agent's code.

>
> Regarding the publish() method, can you give an example for a situation
> where the streamId is not streamIdCache as well as stream definition not
> being in streamDefnCache? I mean will it be a common situation?
>

It's not common, but there can be a situation like that. For example, user
could have already defined the stream may be with different data publisher
and then now publish the events for that stream without defineStream() with
current data publisher. In that case we won't have streamID in
streamIdCache and streamDefn won't be existing in streamDefnCache. But the
incoming event is valid and should reach the receiver as the stream has
been already defined.

>
> If this happens in the case where the stream is not yet defined, can't we
> drop the events at client side without calling the findStream() method?
>

If the stream has been already defined as explained above, then we can't
drop the event.

>
> Also, if we are going with approach#2, will there be a buffer at receiver
> side as well, or will the events be dropped immediately?
>

No, there is a buffer in receiver also. And during the event formatting
time, it'll fail if the stream is not defined. So after some processing
it'll drop, not immediately.

Thanks,
Sinthuja

>
> Thanks,
> Lasantha
>
>
> On 17 January 2014 18:08, Sinthuja Ragendran 
> 
> > wrote:
>
>> Hi all,
>>
>> I'm in the process of merging the changes of AsyncDataPublisher with
>> DataPublisher and hence make our DataPublisher do asynchronous calls on
>> main operations; not only publish but also in connecting to receiver,
>> define stream, etc. This effort is for improve performance in client side
>> and reduce the complexity to chose between what data publisher to use in
>> users' perspective.
>>
>> The below shown is the expected bahaviour of the re-factored asynchronous
>> DataPublisher.
>>
>> API Synchronous Data Publisher Refactored Datapublisher (Async)
>> public DataPublisher(receiverURL, username, password);
>>
>> Connects to the receiver during the object instantiation time. It's
>> blocking call, and if data receiver is down it halt until it timeout.
>>
>>
>> Connects to receiver asynchronously, and during the instantiation it
>> obtains the details and connect to the receiver in separate worker thread.
>> Therefore regardless of data receiver is up or down, it won't affect the
>> client and the invocation thread immediately returns back.
>>
>> public String defineStream(StreamDefinition streamDefinition)
>>
>>
>>
>>
>>
>>
>> It's also blocking call, which halts until the stream is being defined
>> successfully. If we consider the BAM scenario, it'll be halted until it
>> save the stream definition in to cassandra. And if cassandra is
>> down/slow/network latency affects the client side. But the success based on
>> successful return of stream id OR failure based on
>> DifferentStreamDefinitionAlreadyDefinedException,
>> MalformedStreamDefinitionException, etc will be known to the client during
>> the invocation since it's synchronous call.
>> It adds the stream definition to the streamDefnCache and returns the
>> generated streamId based on stream name and version to invocation. During
>> the defineStream() invocation time, the stream wouldn't have actually
>> defined and it'll be only stored in the cache. It'll actually define the
>> stream, once the connection to the receiver was made successfully, and if
>> any event was triggered to publish for this stream. Hence client won't be
>> blocked at any casue during this, but always it'll be success and will not
>> be able propagate the exception since it's asynchronous.
>>
>> __

Re: [Architecture] Re-factoring BAM/CEP Datapublisher

[Sinthuja Ragendran]

Hi lasantha,


On Sun, Jan 19, 2014 at 2:16 AM, Lasantha Fernando wrote:

> Hi Sinthuja,
>
> Thanks for the explanations.
>
>
> On 17 January 2014 21:16, Sinthuja Ragendran  wrote:
>
>> Hi lasantha,
>>
>> Thanks for your suggestions. Please see my comments inline.
>>
>>
>> On Friday, January 17, 2014, Lasantha Fernando  wrote:
>>
>>> Hi Sinthuja,
>>>
>>> +1 for making an asynchronous defineStream() method as well as keeping a
>>> synchronous version. Would it make sense to keep the defineStream() method
>>> as it is and make the new method defineStreamAsync() or similar? Am
>>> suggesting this because AFAIK, some current implementations depend on
>>> defineStream() being synchronous and all those places might have to be
>>> refactored if the change is introduced this way.
>>>
>>
>> IMO we should have the default behavior as asynchronous, if the user want
>> synchronous call then explicitly mention them as defineStreamNow(). Here
>> I'm hoping to have the method signatures as it's, hence we don't need to
>> change the current data agent's code.
>>
>
> AFAIR, there were some places where the stream is defined when the first
> event comes and those methods depended on the definition of the stream
> happening immediately. We might have to check those code segments and fix
> them if the defineStream() behaviour is changed from synchronous to
> asynchronous.
>

Yes, during this effort it will be handled. To defining the stream when the
first event comes in the publisher worker thread will be changed to
defineStreamNow() rather defineStream().

>
>
>>> Regarding the publish() method, can you give an example for a situation
>>> where the streamId is not streamIdCache as well as stream definition not
>>> being in streamDefnCache? I mean will it be a common situation?
>>>
>>
>> It's not common, but there can be a situation like that. For example,
>> user could have already defined the stream may be with different data
>> publisher and then now publish the events for that stream without
>> defineStream() with current data publisher. In that case we won't have
>> streamID in streamIdCache and streamDefn won't be existing in
>> streamDefnCache. But the incoming event is valid and should reach the
>> receiver as the stream has been already defined.
>>
>
> Another small question... How will the streamDefnCache and streamIdCache
> of a particular data publisher be eventually updated regarding a stream
> definition that was defined in a different data publisher?
>

If we go with #1 approach, with findStream() we can find the relevant
streamId and update the cache. #1 approach is viable when the scenario
follows the correct path (ie, Already stream is defined and publishing) as
it can't reduce the unnecessary traffic at receiver. Let say a scenario due
to some reason the stream is not defined (may be user fault), then it'll
make findStream() for every events which makes it slow. This won't affect
the invocation client thread blocking as it's asynchronouse, but there is a
high possibility of buffer becomes full and events may get dropped. Note
this problem only happens in the false path and success path will work fine
as I explained above.

If we go with #2 approach, where in the receiver needs to validate the
success and failure events.

IMO also #2 approach is better, but it also has it's cons as it'll fail
after reaching the receiver and increase the processing of unnecessary
events.

Thanks,
Sinthuja.

>
>  Anyway, to me, approach#2 feels better. IMHO, the responsibility of
> filtering and dropping out invalid events should be at the receiver end.
>
> Thanks,
> Lasantha
>
>
>>> If this happens in the case where the stream is not yet defined, can't
>>> we drop the events at client side without calling the findStream() method?
>>>
>>
>> If the stream has been already defined as explained above, then we can't
>> drop the event.
>>
>>>
>>> Also, if we are going with approach#2, will there be a buffer at
>>> receiver side as well, or will the events be dropped immediately?
>>>
>>
>> No, there is a buffer in receiver also. And during the event formatting
>> time, it'll fail if the stream is not defined. So after some processing
>> it'll drop, not immediately.
>>
>> Thanks,
>> Sinthuja
>>
>>>
>>> Thanks,
>>> Lasantha
>>>
>>>
>>> On 17 January 2014 18:08, Sinthuja Ragendran  wrote:
>>>
>>>> Hi all,
>>

Re: [Architecture] Re-factoring BAM/CEP Datapublisher

[Sinthuja Ragendran]

Hi,

Srinath, suho and myself had a discussion on above and agreed to go with
approach #2 for during the publish events.

Thanks,
Sinthuja.



On Mon, Jan 20, 2014 at 10:47 AM, Sinthuja Ragendran wrote:

> Hi lasantha,
>
>
> On Sun, Jan 19, 2014 at 2:16 AM, Lasantha Fernando wrote:
>
>> Hi Sinthuja,
>>
>> Thanks for the explanations.
>>
>>
>> On 17 January 2014 21:16, Sinthuja Ragendran  wrote:
>>
>>> Hi lasantha,
>>>
>>> Thanks for your suggestions. Please see my comments inline.
>>>
>>>
>>> On Friday, January 17, 2014, Lasantha Fernando 
>>> wrote:
>>>
>>>> Hi Sinthuja,
>>>>
>>>> +1 for making an asynchronous defineStream() method as well as keeping
>>>> a synchronous version. Would it make sense to keep the defineStream()
>>>> method as it is and make the new method defineStreamAsync() or similar? Am
>>>> suggesting this because AFAIK, some current implementations depend on
>>>> defineStream() being synchronous and all those places might have to be
>>>> refactored if the change is introduced this way.
>>>>
>>>
>>> IMO we should have the default behavior as asynchronous, if the user
>>> want synchronous call then explicitly mention them as defineStreamNow().
>>> Here I'm hoping to have the method signatures as it's, hence we don't need
>>> to change the current data agent's code.
>>>
>>
>> AFAIR, there were some places where the stream is defined when the first
>> event comes and those methods depended on the definition of the stream
>> happening immediately. We might have to check those code segments and fix
>> them if the defineStream() behaviour is changed from synchronous to
>> asynchronous.
>>
>
> Yes, during this effort it will be handled. To defining the stream when
> the first event comes in the publisher worker thread will be changed to
> defineStreamNow() rather defineStream().
>
>>
>>
>>>> Regarding the publish() method, can you give an example for a situation
>>>> where the streamId is not streamIdCache as well as stream definition not
>>>> being in streamDefnCache? I mean will it be a common situation?
>>>>
>>>
>>> It's not common, but there can be a situation like that. For example,
>>> user could have already defined the stream may be with different data
>>> publisher and then now publish the events for that stream without
>>> defineStream() with current data publisher. In that case we won't have
>>> streamID in streamIdCache and streamDefn won't be existing in
>>> streamDefnCache. But the incoming event is valid and should reach the
>>> receiver as the stream has been already defined.
>>>
>>
>> Another small question... How will the streamDefnCache and streamIdCache
>> of a particular data publisher be eventually updated regarding a stream
>> definition that was defined in a different data publisher?
>>
>
> If we go with #1 approach, with findStream() we can find the relevant
> streamId and update the cache. #1 approach is viable when the scenario
> follows the correct path (ie, Already stream is defined and publishing) as
> it can't reduce the unnecessary traffic at receiver. Let say a scenario due
> to some reason the stream is not defined (may be user fault), then it'll
> make findStream() for every events which makes it slow. This won't affect
> the invocation client thread blocking as it's asynchronouse, but there is a
> high possibility of buffer becomes full and events may get dropped. Note
> this problem only happens in the false path and success path will work fine
> as I explained above.
>
> If we go with #2 approach, where in the receiver needs to validate the
> success and failure events.
>
> IMO also #2 approach is better, but it also has it's cons as it'll fail
> after reaching the receiver and increase the processing of unnecessary
> events.
>
> Thanks,
> Sinthuja.
>
>>
>>  Anyway, to me, approach#2 feels better. IMHO, the responsibility of
>> filtering and dropping out invalid events should be at the receiver end.
>>
>> Thanks,
>> Lasantha
>>
>>
>>>> If this happens in the case where the stream is not yet defined, can't
>>>> we drop the events at client side without calling the findStream() method?
>>>>
>>>
>>> If the stream has been already defined as explained above, then we can't
>>> drop the event.
>>>

Re: [Architecture] Should we have one place to define event streams?

[Sinthuja Ragendran]

Hi,


On Mon, Jan 20, 2014 at 5:36 PM, Maninda Edirisooriya wrote:

> Already BAM can create streams with REST API. To facilitate further for a
> person configuring BAM manually without a toolbox, we should provide a UI
> to create streams.
>

CEP already has UI to create streams [1], I think it's for registry stream
defintion store. I'm not sure whether it'll support cassandra stream
definition store at the moment. We need to test and make it compatible with
Cassandra stream definition store as well for BAM.


(and better if possible to facilitate edit and delete streams) And that UI
> should support defining indexes when required as current implementation of
> defining indexes is not nice.
>

The event streams are common for BAM and CEP hence we should be careful
when adding indexes, etc to that common UI where it's not relevant to CEP
usecase. Here we have to load the attributes dynamically depending on what
stream definition store (cassandra, registry, etc), so that we can add the
relevant configuration like indexes, etc.

[1] http://docs.wso2.org/display/CEP300/Working+with+Event+Streams

Thanks,
Sinthuja.

>
> As REST API fulfils light weight requirements of BAM, we can remove the
> Thrift based stream creation feature from the data publisher side. In high
> performance requirement of BAM, creating a stream from publisher side is
> not valid. They should deploy a toolbox at the beginning instead. Anyway
> this will make it backward incompatible for older BAM versions.
>
> And BTW what about keeping all the toolboxes in a WSO2 Store as discussed
> earlier in the thread?
>
>
> * Maninda Edirisooriya*
> Software Engineer
>
> *WSO2, Inc.*lean.enterprise.middleware.
>
> *Blog* : http://maninda.blogspot.com/
> *Phone* : +94 777603226
>
>
> On Mon, Jan 20, 2014 at 4:58 PM, Paul Fremantle  wrote:
>
>> I'd like to add some requirements in if possible.
>>
>> 1) I'd like some really easy dynamic stuff. For example, if I'm using
>> REST, can I just send a JSON? We could create a stream definition
>> dynamically with the name based on some part of the URL and compare to
>> previous definitions for a version number. This isn't going to be
>> efficient, but in many cases I just want to get on with things.
>>
>> I think we have to be aware we have three potentially different usecases
>> here:
>> a) High performance BAM and CEP - need the stream defined in advance for
>> efficiency
>> b) Ad hoc BAM and logging - maybe we don't want to be so prescriptive /
>> statically defined
>> c) Lightweight CEP / prototyping - being able to use a stream like a
>> dynamic language and do duck typing would be useful.
>>
>> 2) I'd like to support both a Registry model but also being able to point
>> to a stream definition JSON via a URL. This would make this nice and
>> "webby"/ REST-friendly.
>>
>> 3) We need to make it super easy to define a stream, edit stream
>> definitions, find them, etc.
>>
>> Paul
>>
>>
>> On 20 January 2014 11:10, Srinath Perera  wrote:
>>
>>> Hi All,
>>>
>>> I am reviving this thread as now we have support to add remove streams
>>> though CEP UI (and BAM?) and via toolboxes, but we still define the event
>>> streams with the data publisher each time. (this thread discusses about
>>> stop doing that).
>>>
>>> Typical CEP/ BAM usecase would go like following
>>>
>>> 1) first you identify and define streams
>>> 2) then you write queries
>>> 3) then you write/configure the agents and point to streams they should
>>> publish to.
>>>
>>> However, we redefine streams from agents every time we publish. This has
>>> several complications.
>>>
>>> 1) If an agent choose to define stream different from queries,
>>> everything get messed up. (problem was in samples in BAM).
>>> 2) If there are multiple event receivers, we need a way to sync them
>>> with db everytime we update a stream.
>>>
>>> I would like to propose we switch to a model where we write queries and
>>> streams and deploy them as a toolbox. Then, just mention the stream name in
>>> the event publisher to keep things simple.
>>>
>>> --Srinath
>>>
>>>
>>>
>>> On Sat, Jan 19, 2013 at 5:41 PM, Tharindu Mathew wrote:
>>>
 A store discover already supported event streams seems to be a good
 idea. It would be good idea which we can implement through a UI or a store.

 But, what do you mean by defining event streams everywhere?

 Clients can define any stream they want, but it is only defined at the
 stream definition store. If there is an error, it is shown in the stream
 definition store and returned to the client (I hope it is returned at
 least, otherwise it is a bug).

 This model was first proposed to get out of the eventing hell that we
 put ourselves into. Right now, if you want to publish something, you just
 define it and publish. You don't have to switch between multiple servers
 just to define and publish some events, which was extremely annoying.




 On Sat, Jan 19, 2013 a

[Architecture] [Update] Incremental Processing for BAM

[Sinthuja Ragendran]

Hi all,

I was working on providing the incremental processing support for BAM. This
feature was implemented in BAM 2.4.0 [1], but it wasn't well tested in
fully distributed mode during the BAM 2.4.0 release, hence it was marked as
experimental feature. You can find the implementation details of this
feature from [2].

During the last week I was involved with testing this feature with 3 node
hadoop cluster and Cassandra cluster. And all use cases mentioned in [1]
was able to run in the external cluster without any issues.

And also I have implemented incremental average operation (incr_avg())
which incrementally calculates the final value based on the last hive query
execution and the current execution. The intermediate necessary values for
incremental operation, are store in cassandra and used for the current hive
query execution. And once the current hive query execution is completed,
the intermediate results will be again replaced with the new values. I have
tested this incr_avg() operation also with fully distributed setup and it
works well. As the first step, I have only implemented the incr_avg() to
make sure whether the adopted approach will succeed on fully distributed
setup.

The below is the hive script sample which uses the incr_avg() function:















*CREATE EXTERNAL TABLE IF NOT EXISTS PhoneSalesTable   (orderID STRING,
brandName STRING, userName STRING, quantity INT, version STRING) STORED
BY   'org.apache.hadoop.hive.cassandra.CassandraStorageHandler'   WITH
SERDEPROPERTIES ( "wso2.carbon.datasource.name
" = "WSO2BAM_CASSANDRA_DATASOURCE",
"cassandra.cf.name " =
"org_wso2_bam_phone_retail_store_kpi" ,   "cassandra.columns.mapping" =
":key,payload_brand, payload_user, payload_quantity, Version" );
@Incremental(name="avgAnalysis", tables="PhoneSalesTable",
bufferTime="20")select brandName, count(DISTINCT orderID),
incr_avg(quantity, "average_quantity")  from PhoneSalesTablewhere
version= "1.0.0" group by brandName;*

The following are the to-do items on this feature:

   - Do a load test with all cases/scenarios with distributed setup
   - Implement more commonly used hive functions as incremental
   functions(incr_count, incr_sum, etc).
   - Write a sample to explain how to use this feature.


[1] http://docs.wso2.org/pages/viewpage.action?pageId=32345660
[2]
http://wso2-oxygen-tank.10903.n7.nabble.com/Incremental-Data-Processing-for-BAM-td77582.html

Thanks,
Sinthuja.

-- 
*Sinthuja Rajendran*
Software Engineer 
WSO2, Inc.:http://wso2.com

Blog: http://sinthu-rajan.blogspot.com/
Mobile: +94774273955
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [Update] Incremental Processing for BAM

[Sinthuja Ragendran]

Hi Srinath,



On Mon, Feb 10, 2014 at 7:51 PM, Srinath Perera  wrote:

> Hi Sinthuja,
>
> Do we now support processing data for a given time period? e.g. last 24
> hours.
>

Yes we do support that. We can do this by following ways.

1) We can configure the start time and end time of data which needs to be
considered for the current hive query execution with following annotation.

*@Incremental(name="incrSample", tables="cassandraTable1",
fromTime="$NOW-8640", toTime="$NOW")*

Here 24hours = 8640 ms. Hence it will consider last 24hours data.


2) We can also schedule the hive script to run every 24hours and then we
can annotate the hive script with following annotation:

@Incremental(name="incrementalProcess", tables="cassandraReadTable",
bufferTime="0").

Hence the query only will consider the unprocessed last 24hours data.




> I am -1 on doing anything more on this for now (e.g. incr_count).
>

As far as we can confirm incr_avg() sample function is working, the
implementation of other functions also very similar. And also it will be
very useful for the users to use this functionality IMHO.

>
> IMHO, there are more high priority items in the roadmap, please chat with
> Anjana.
>

According to discussion I had previously with Anjana, first we have to
start with sample function (incr_avg()) and then implement that to other
functions. May be now the priorities have been changed and we can push this
for later releases. :)   Anjana, Please confirm.

Thanks,
Sinthuja.

>
> --Srinath
>
>
> On Mon, Feb 10, 2014 at 2:42 PM, Sinthuja Ragendran wrote:
>
>> Hi all,
>>
>> I was working on providing the incremental processing support for BAM.
>> This feature was implemented in BAM 2.4.0 [1], but it wasn't well tested in
>> fully distributed mode during the BAM 2.4.0 release, hence it was marked as
>> experimental feature. You can find the implementation details of this
>> feature from [2].
>>
>> During the last week I was involved with testing this feature with 3 node
>> hadoop cluster and Cassandra cluster. And all use cases mentioned in [1]
>> was able to run in the external cluster without any issues.
>>
>> And also I have implemented incremental average operation (incr_avg())
>> which incrementally calculates the final value based on the last hive query
>> execution and the current execution. The intermediate necessary values for
>> incremental operation, are store in cassandra and used for the current hive
>> query execution. And once the current hive query execution is completed,
>> the intermediate results will be again replaced with the new values. I have
>> tested this incr_avg() operation also with fully distributed setup and it
>> works well. As the first step, I have only implemented the incr_avg() to
>> make sure whether the adopted approach will succeed on fully distributed
>> setup.
>>
>> The below is the hive script sample which uses the incr_avg() function:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *CREATE EXTERNAL TABLE IF NOT EXISTS PhoneSalesTable   (orderID STRING,
>> brandName STRING, userName STRING, quantity INT, version STRING) STORED
>> BY   'org.apache.hadoop.hive.cassandra.CassandraStorageHandler'   WITH
>> SERDEPROPERTIES ( "wso2.carbon.datasource.name
>> <http://wso2.carbon.datasource.name>" = "WSO2BAM_CASSANDRA_DATASOURCE",
>> "cassandra.cf.name <http://cassandra.cf.name>" =
>> "org_wso2_bam_phone_retail_store_kpi" ,   "cassandra.columns.mapping" =
>> ":key,payload_brand, payload_user, payload_quantity, Version" );
>> @Incremental(name="avgAnalysis", tables="PhoneSalesTable",
>> bufferTime="20")select brandName, count(DISTINCT orderID),
>> incr_avg(quantity, "average_quantity")  from PhoneSalesTablewhere
>> version= "1.0.0" group by brandName;*
>>
>> The following are the to-do items on this feature:
>>
>>- Do a load test with all cases/scenarios with distributed setup
>>- Implement more commonly used hive functions as incremental
>>functions(incr_count, incr_sum, etc).
>>- Write a sample to explain how to use this feature.
>>
>>
>> [1] http://docs.wso2.org/pages/viewpage.action?pageId=32345660
>> [2]
>> http://wso2-oxygen-tank.10903.n7.nabble.com/Incremental-Data-Processing-for-BAM-td77582.html
>>
>> Thanks,
>> Sinthuja.
>>
>> --
>> *Si