Re: [Dev] ESB Analytics for MT scenarios?

[Anjana Fernando]

Hi,

On Wed, Jun 22, 2016 at 3:25 PM, Isabelle Mauny  wrote:

> Great! So that's for Spark.. You mentioned CEP as a big issue as well. Any
> hope to fix in CEP too ??
>

As per Suho's reply in this thread, it wouldn't be an issue, it just
consumed a little bit of extra memory, but performance wise, there
shouldn't be a problem.

Cheers,
Anjana.


>
>
> -
> *Isabelle Mauny*
> VP, Product Strategy - WSO2, Inc. - http://wso2.com/
> email: isabe...@wso2.com - mobile (Spain) : +34 616050684 - mobile (Sri
> Lanka) +94 (0)774777663
> Landline:  +1 (650) 745 4499  (USA)  or +94 (11) 214 534 (Sri Lanka)
> Extension : 7302
>
> On Wed, Jun 22, 2016 at 11:50 AM, Srinath Perera 
> wrote:
>
>> Perfect!!
>>
>> On Wed, Jun 22, 2016 at 1:58 PM, Anjana Fernando  wrote:
>>
>>> Hi,
>>>
>>> I've updated the thread "Cross Tenant Data Reading from Spark Queries in
>>> DAS", where I've done the implementation on writing back to the tenant
>>> space also from the super tenant space as well. This basically just
>>> eliminates the current requirement of executing identical Spark scripts in
>>> all the tenants, which would not be desirable performance wise, and now, we
>>> can just use super tenant's single script to process all the tenant's data
>>> and write back the results to their space. Of course, we've to now change
>>> the current ESB analytics scripts to use this feature, but it shouldn't be
>>> a big change. @Gihan/Supun, can we please check on this and get it done.
>>>
>>> Cheers,
>>> Anjana.
>>>
>>> On Tue, Jun 21, 2016 at 12:35 PM, Anjana Fernando 
>>> wrote:
>>>
 Hi,

 Yeah, as Srinath mentioned, we do have MT support there, where each ESB
 analytics data/queryeis/dashboards would be fully isolated. This is the
 usual desirable mode, when different tenants deploy their own analytics
 scripts and all, and there is no connection at all with other tenants, and
 obviously there shouldn't be. But in our solutions like this, where we
 would deploy the exact same solution in each tenant, we can potentially
 optimize the scenario by, only doing the processing in one place. This is
 actually how APIM batch queries are written at the moment. Where they
 altogether stores all the tenant's data in super tenant space and process
 and put the summarized there itself. This model works, but then, if the
 tenant itself wants to do their own processing it doesn't have access to
 the data, that was suppose to be in his space. So a solution for this is
 done, which is explained in detail in the mail "Cross Tenant Data Reading
 from Spark Queries in DAS", where optionally the super tenant can read
 tenant's data which creates a single virtual table view which contains data
 from all the tenants. From this, you can read the data, do the processing
 and store it in a table in super tenant space. In this approach, you will
 still have to have the dashboard in the super tenant space, since the data
 resides in the super tenant space. I will be doing some experiments to see
 on the feasibility of storing back summarized data in tenant space soon.

 Even though we can get the batch scenario working efficiently, there
 would be a problem in the CEP side, which we now use heavily in the product
 analytics scenario processing. At the moment, there is no concept of having
 a single CEP execution plan for all the tenants. When data is published to
 receivers, they will be dispatched to their own streams and execution
 plans, so we will need to find an approach for this, to optimize the
 processing of the CEP side of things. Or else, I'm not sure, since these
 are in-memory processing constructs, if having copies of the same execution
 plans will actually be okay, compared to having a single one, since the
 traffic is anyway split between them.

 Cheers,
 Anjana.

 On Tue, Jun 21, 2016 at 9:35 AM, Srinath Perera 
 wrote:

> We had a chat about this.
>
> This is supported, however, current implementation will run spark jobs
> per each tenant. So it will have performance concerns with a large number
> of tenants.
>
> Anjana, can you give the details and potential solutions? DAS team has
> implemented the support for reading tenant data from super tenant so we 
> can
> run a one Spark Job for all tenants. But we will need to change the 
> queries
> and figure out how tenant dashboards can get access to results in super
> tenant's space.
>
> Can we do a 5.1 addressing this?
>
> --Srinath
>
> On Tue, Jun 21, 2016 at 9:15 AM, Kasun Indrasiri 
> wrote:
>
>> Please note that this will be an essential feature for our
>> 

Re: [Dev] ESB Analytics for MT scenarios?

[Isabelle Mauny]

Great! So that's for Spark.. You mentioned CEP as a big issue as well. Any
hope to fix in CEP too ??

-
*Isabelle Mauny*
VP, Product Strategy - WSO2, Inc. - http://wso2.com/
email: isabe...@wso2.com - mobile (Spain) : +34 616050684 - mobile (Sri
Lanka) +94 (0)774777663
Landline:  +1 (650) 745 4499  (USA)  or +94 (11) 214 534 (Sri Lanka)
Extension : 7302

On Wed, Jun 22, 2016 at 11:50 AM, Srinath Perera  wrote:

> Perfect!!
>
> On Wed, Jun 22, 2016 at 1:58 PM, Anjana Fernando  wrote:
>
>> Hi,
>>
>> I've updated the thread "Cross Tenant Data Reading from Spark Queries in
>> DAS", where I've done the implementation on writing back to the tenant
>> space also from the super tenant space as well. This basically just
>> eliminates the current requirement of executing identical Spark scripts in
>> all the tenants, which would not be desirable performance wise, and now, we
>> can just use super tenant's single script to process all the tenant's data
>> and write back the results to their space. Of course, we've to now change
>> the current ESB analytics scripts to use this feature, but it shouldn't be
>> a big change. @Gihan/Supun, can we please check on this and get it done.
>>
>> Cheers,
>> Anjana.
>>
>> On Tue, Jun 21, 2016 at 12:35 PM, Anjana Fernando 
>> wrote:
>>
>>> Hi,
>>>
>>> Yeah, as Srinath mentioned, we do have MT support there, where each ESB
>>> analytics data/queryeis/dashboards would be fully isolated. This is the
>>> usual desirable mode, when different tenants deploy their own analytics
>>> scripts and all, and there is no connection at all with other tenants, and
>>> obviously there shouldn't be. But in our solutions like this, where we
>>> would deploy the exact same solution in each tenant, we can potentially
>>> optimize the scenario by, only doing the processing in one place. This is
>>> actually how APIM batch queries are written at the moment. Where they
>>> altogether stores all the tenant's data in super tenant space and process
>>> and put the summarized there itself. This model works, but then, if the
>>> tenant itself wants to do their own processing it doesn't have access to
>>> the data, that was suppose to be in his space. So a solution for this is
>>> done, which is explained in detail in the mail "Cross Tenant Data Reading
>>> from Spark Queries in DAS", where optionally the super tenant can read
>>> tenant's data which creates a single virtual table view which contains data
>>> from all the tenants. From this, you can read the data, do the processing
>>> and store it in a table in super tenant space. In this approach, you will
>>> still have to have the dashboard in the super tenant space, since the data
>>> resides in the super tenant space. I will be doing some experiments to see
>>> on the feasibility of storing back summarized data in tenant space soon.
>>>
>>> Even though we can get the batch scenario working efficiently, there
>>> would be a problem in the CEP side, which we now use heavily in the product
>>> analytics scenario processing. At the moment, there is no concept of having
>>> a single CEP execution plan for all the tenants. When data is published to
>>> receivers, they will be dispatched to their own streams and execution
>>> plans, so we will need to find an approach for this, to optimize the
>>> processing of the CEP side of things. Or else, I'm not sure, since these
>>> are in-memory processing constructs, if having copies of the same execution
>>> plans will actually be okay, compared to having a single one, since the
>>> traffic is anyway split between them.
>>>
>>> Cheers,
>>> Anjana.
>>>
>>> On Tue, Jun 21, 2016 at 9:35 AM, Srinath Perera 
>>> wrote:
>>>
 We had a chat about this.

 This is supported, however, current implementation will run spark jobs
 per each tenant. So it will have performance concerns with a large number
 of tenants.

 Anjana, can you give the details and potential solutions? DAS team has
 implemented the support for reading tenant data from super tenant so we can
 run a one Spark Job for all tenants. But we will need to change the queries
 and figure out how tenant dashboards can get access to results in super
 tenant's space.

 Can we do a 5.1 addressing this?

 --Srinath

 On Tue, Jun 21, 2016 at 9:15 AM, Kasun Indrasiri 
 wrote:

> Please note that this will be an essential feature for our integration
> cloud. So, we need to decide how should we proceed with this soon.
>
> On Fri, Jun 17, 2016 at 1:57 AM, Kasun Indrasiri 
> wrote:
>
>> How about the $subject? I don't think there's no any limitation from
>> the mediation engine side on collecting these stats. We need to have this
>> for ESB 5.
>>
>> --
>> Kasun Indrasiri

Re: [Dev] ESB Analytics for MT scenarios?

[Anjana Fernando]

Hi,

On Wed, Jun 22, 2016 at 11:38 AM, Selvaratnam Uthaiyashankar <
shan...@wso2.com> wrote:

>
>
> On Tue, Jun 21, 2016 at 11:31 AM, Buddhima Wijeweera 
> wrote:
>
>> Hi,
>>
>> At the moment ESB can publish tenant-wise. I tried it with a latest ESB
>> and following are the steps need to follow:
>> 1. Enable statistics for global level.
>> 2. Add eventpublisher & eventstream files to tenant's deployment folder
>>
>
> Who has to deploy this? Tenant or Super tenant? how do we deploy this in
> cloud setup? Do we have UI for this?
>

Event stream can come from our CApp itself, the event publisher, the tenant
user has to deploy, and yes, it has an UI.


>
>
>
>
>
>> 3. Create a proxy (or API, inbound EP) and Enable Statistics.
>> 4. Send messages to proxy
>>
>> Depending on the tenant credentials in eventpublisher files(at point 2
>> above), statistics  get published to analytics-esb server. (for this
>> testing I published to super-tenant of analytics-esb).
>>
>
> Also, if point 2 above has to be done by the tenant, tenant will not know
> super tenant credentials to send events.
>

The tenant user should use a tenant user's credentials to send events, not
super tenant, since with our tenant isolation, the data should go to the
tenant's space.

Cheers,
Anjana.


>
>
>
>>
>> Thanks,
>>
>> On Tue, Jun 21, 2016 at 9:37 AM, Srinath Perera  wrote:
>>
>>> We had a chat about this.
>>>
>>> This is supported. However, current implementation will run spark jobs
>>> per each tenant. So it will have performance concerns with a large number
>>> of tenants.
>>>
>>> Anjana, can you give the details and potential solutions?
>>>
>>> Summary is that DAS team has implemented the support for reading tenant
>>> data from super tenant so we can run a one Spark Job for all tenants. But
>>> we will need to change the queries and figure out how tenant dashboards can
>>> get access to results in super tenant's space.
>>>
>>> Can we do a 5.1 addressing this?
>>>
>>> --Srinath
>>>
>>> On Tue, Jun 21, 2016 at 9:15 AM, Kasun Indrasiri  wrote:
>>>
 Please note that this will be an essential feature for our integration
 cloud. So, we need to decide how should we proceed with this soon.

 On Fri, Jun 17, 2016 at 1:57 AM, Kasun Indrasiri 
 wrote:

> How about the $subject? I don't think there's no any limitation from
> the mediation engine side on collecting these stats. We need to have this
> for ESB 5.
>
> --
> Kasun Indrasiri
> Software Architect
> WSO2, Inc.; http://wso2.com
> lean.enterprise.middleware
>
> cell: +94 77 556 5206
> Blog : http://kasunpanorama.blogspot.com/
>



 --
 Kasun Indrasiri
 Software Architect
 WSO2, Inc.; http://wso2.com
 lean.enterprise.middleware

 cell: +94 77 556 5206
 Blog : http://kasunpanorama.blogspot.com/

>>>
>>>
>>>
>>> --
>>> 
>>> Srinath Perera, Ph.D.
>>>http://people.apache.org/~hemapani/
>>>http://srinathsview.blogspot.com/
>>>
>>
>>
>>
>> --
>> Buddhima Wijeweera
>> Software Engineer; WSO2 Inc.; http://wso2.com ,
>>
>> Mobile: +94 71 427 9966
>> Email: buddh...@wso2.com
>> Blog:   https://buddhimawijeweera.wordpress.com
>> GitHub Profile: https://github.com/Buddhima
>>
>> ___
>> Dev mailing list
>> Dev@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
> S.Uthaiyashankar
> VP Engineering
> WSO2 Inc.
> http://wso2.com/ - "lean . enterprise . middleware"
>
> Phone: +94 714897591
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 
*Anjana Fernando*
Senior Technical Lead
WSO2 Inc. | http://wso2.com
lean . enterprise . middleware
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] ESB Analytics for MT scenarios?

[Anjana Fernando]

Hi,

I've updated the thread "Cross Tenant Data Reading from Spark Queries in
DAS", where I've done the implementation on writing back to the tenant
space also from the super tenant space as well. This basically just
eliminates the current requirement of executing identical Spark scripts in
all the tenants, which would not be desirable performance wise, and now, we
can just use super tenant's single script to process all the tenant's data
and write back the results to their space. Of course, we've to now change
the current ESB analytics scripts to use this feature, but it shouldn't be
a big change. @Gihan/Supun, can we please check on this and get it done.

Cheers,
Anjana.

On Tue, Jun 21, 2016 at 12:35 PM, Anjana Fernando  wrote:

> Hi,
>
> Yeah, as Srinath mentioned, we do have MT support there, where each ESB
> analytics data/queryeis/dashboards would be fully isolated. This is the
> usual desirable mode, when different tenants deploy their own analytics
> scripts and all, and there is no connection at all with other tenants, and
> obviously there shouldn't be. But in our solutions like this, where we
> would deploy the exact same solution in each tenant, we can potentially
> optimize the scenario by, only doing the processing in one place. This is
> actually how APIM batch queries are written at the moment. Where they
> altogether stores all the tenant's data in super tenant space and process
> and put the summarized there itself. This model works, but then, if the
> tenant itself wants to do their own processing it doesn't have access to
> the data, that was suppose to be in his space. So a solution for this is
> done, which is explained in detail in the mail "Cross Tenant Data Reading
> from Spark Queries in DAS", where optionally the super tenant can read
> tenant's data which creates a single virtual table view which contains data
> from all the tenants. From this, you can read the data, do the processing
> and store it in a table in super tenant space. In this approach, you will
> still have to have the dashboard in the super tenant space, since the data
> resides in the super tenant space. I will be doing some experiments to see
> on the feasibility of storing back summarized data in tenant space soon.
>
> Even though we can get the batch scenario working efficiently, there would
> be a problem in the CEP side, which we now use heavily in the product
> analytics scenario processing. At the moment, there is no concept of having
> a single CEP execution plan for all the tenants. When data is published to
> receivers, they will be dispatched to their own streams and execution
> plans, so we will need to find an approach for this, to optimize the
> processing of the CEP side of things. Or else, I'm not sure, since these
> are in-memory processing constructs, if having copies of the same execution
> plans will actually be okay, compared to having a single one, since the
> traffic is anyway split between them.
>
> Cheers,
> Anjana.
>
> On Tue, Jun 21, 2016 at 9:35 AM, Srinath Perera 
> wrote:
>
>> We had a chat about this.
>>
>> This is supported, however, current implementation will run spark jobs
>> per each tenant. So it will have performance concerns with a large number
>> of tenants.
>>
>> Anjana, can you give the details and potential solutions? DAS team has
>> implemented the support for reading tenant data from super tenant so we can
>> run a one Spark Job for all tenants. But we will need to change the queries
>> and figure out how tenant dashboards can get access to results in super
>> tenant's space.
>>
>> Can we do a 5.1 addressing this?
>>
>> --Srinath
>>
>> On Tue, Jun 21, 2016 at 9:15 AM, Kasun Indrasiri  wrote:
>>
>>> Please note that this will be an essential feature for our integration
>>> cloud. So, we need to decide how should we proceed with this soon.
>>>
>>> On Fri, Jun 17, 2016 at 1:57 AM, Kasun Indrasiri  wrote:
>>>
 How about the $subject? I don't think there's no any limitation from
 the mediation engine side on collecting these stats. We need to have this
 for ESB 5.

 --
 Kasun Indrasiri
 Software Architect
 WSO2, Inc.; http://wso2.com
 lean.enterprise.middleware

 cell: +94 77 556 5206
 Blog : http://kasunpanorama.blogspot.com/

>>>
>>>
>>>
>>> --
>>> Kasun Indrasiri
>>> Software Architect
>>> WSO2, Inc.; http://wso2.com
>>> lean.enterprise.middleware
>>>
>>> cell: +94 77 556 5206
>>> Blog : http://kasunpanorama.blogspot.com/
>>>
>>
>>
>>
>> --
>> 
>>WSO2 Inc. http://wso2.com
>>Blog: http://srinathsview.blogspot.com/
>>Photos: http://www.flickr.com/photos/hemapani/ twitter:@srinath_perera
>>Site: http://people.apache.org/~hemapani/
>>
>
>
>
> --
> *Anjana Fernando*
> Senior Technical Lead
> WSO2 Inc. | http://wso2.com
> lean . enterprise . middleware
>



-- 
*Anjana Fernando*
Senior Technical Lead
WSO2 Inc. | 

Re: [Dev] ESB Analytics for MT scenarios?

[Selvaratnam Uthaiyashankar]

On Tue, Jun 21, 2016 at 11:31 AM, Buddhima Wijeweera 
wrote:

> Hi,
>
> At the moment ESB can publish tenant-wise. I tried it with a latest ESB
> and following are the steps need to follow:
> 1. Enable statistics for global level.
> 2. Add eventpublisher & eventstream files to tenant's deployment folder
>

Who has to deploy this? Tenant or Super tenant? how do we deploy this in
cloud setup? Do we have UI for this?





> 3. Create a proxy (or API, inbound EP) and Enable Statistics.
> 4. Send messages to proxy
>
> Depending on the tenant credentials in eventpublisher files(at point 2
> above), statistics  get published to analytics-esb server. (for this
> testing I published to super-tenant of analytics-esb).
>

Also, if point 2 above has to be done by the tenant, tenant will not know
super tenant credentials to send events.



>
> Thanks,
>
> On Tue, Jun 21, 2016 at 9:37 AM, Srinath Perera  wrote:
>
>> We had a chat about this.
>>
>> This is supported. However, current implementation will run spark jobs
>> per each tenant. So it will have performance concerns with a large number
>> of tenants.
>>
>> Anjana, can you give the details and potential solutions?
>>
>> Summary is that DAS team has implemented the support for reading tenant
>> data from super tenant so we can run a one Spark Job for all tenants. But
>> we will need to change the queries and figure out how tenant dashboards can
>> get access to results in super tenant's space.
>>
>> Can we do a 5.1 addressing this?
>>
>> --Srinath
>>
>> On Tue, Jun 21, 2016 at 9:15 AM, Kasun Indrasiri  wrote:
>>
>>> Please note that this will be an essential feature for our integration
>>> cloud. So, we need to decide how should we proceed with this soon.
>>>
>>> On Fri, Jun 17, 2016 at 1:57 AM, Kasun Indrasiri  wrote:
>>>
 How about the $subject? I don't think there's no any limitation from
 the mediation engine side on collecting these stats. We need to have this
 for ESB 5.

 --
 Kasun Indrasiri
 Software Architect
 WSO2, Inc.; http://wso2.com
 lean.enterprise.middleware

 cell: +94 77 556 5206
 Blog : http://kasunpanorama.blogspot.com/

>>>
>>>
>>>
>>> --
>>> Kasun Indrasiri
>>> Software Architect
>>> WSO2, Inc.; http://wso2.com
>>> lean.enterprise.middleware
>>>
>>> cell: +94 77 556 5206
>>> Blog : http://kasunpanorama.blogspot.com/
>>>
>>
>>
>>
>> --
>> 
>> Srinath Perera, Ph.D.
>>http://people.apache.org/~hemapani/
>>http://srinathsview.blogspot.com/
>>
>
>
>
> --
> Buddhima Wijeweera
> Software Engineer; WSO2 Inc.; http://wso2.com ,
>
> Mobile: +94 71 427 9966
> Email: buddh...@wso2.com
> Blog:   https://buddhimawijeweera.wordpress.com
> GitHub Profile: https://github.com/Buddhima
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 
S.Uthaiyashankar
VP Engineering
WSO2 Inc.
http://wso2.com/ - "lean . enterprise . middleware"

Phone: +94 714897591
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] ESB Analytics for MT scenarios?

[Sriskandarajah Suhothayan]

I think having separate execution plans for each tenant will not be a
problem. Though this consumes some amount of memory to construct multiple
execution plans it will improve the performance as data is not partitioned
by tenants by default.

Regards
Suho

On Tue, Jun 21, 2016 at 8:05 AM, Anjana Fernando  wrote:

> Hi,
>
> Yeah, as Srinath mentioned, we do have MT support there, where each ESB
> analytics data/queryeis/dashboards would be fully isolated. This is the
> usual desirable mode, when different tenants deploy their own analytics
> scripts and all, and there is no connection at all with other tenants, and
> obviously there shouldn't be. But in our solutions like this, where we
> would deploy the exact same solution in each tenant, we can potentially
> optimize the scenario by, only doing the processing in one place. This is
> actually how APIM batch queries are written at the moment. Where they
> altogether stores all the tenant's data in super tenant space and process
> and put the summarized there itself. This model works, but then, if the
> tenant itself wants to do their own processing it doesn't have access to
> the data, that was suppose to be in his space. So a solution for this is
> done, which is explained in detail in the mail "Cross Tenant Data Reading
> from Spark Queries in DAS", where optionally the super tenant can read
> tenant's data which creates a single virtual table view which contains data
> from all the tenants. From this, you can read the data, do the processing
> and store it in a table in super tenant space. In this approach, you will
> still have to have the dashboard in the super tenant space, since the data
> resides in the super tenant space. I will be doing some experiments to see
> on the feasibility of storing back summarized data in tenant space soon.
>
> Even though we can get the batch scenario working efficiently, there would
> be a problem in the CEP side, which we now use heavily in the product
> analytics scenario processing. At the moment, there is no concept of having
> a single CEP execution plan for all the tenants. When data is published to
> receivers, they will be dispatched to their own streams and execution
> plans, so we will need to find an approach for this, to optimize the
> processing of the CEP side of things. Or else, I'm not sure, since these
> are in-memory processing constructs, if having copies of the same execution
> plans will actually be okay, compared to having a single one, since the
> traffic is anyway split between them.
>
> Cheers,
> Anjana.
>
> On Tue, Jun 21, 2016 at 9:35 AM, Srinath Perera 
> wrote:
>
>> We had a chat about this.
>>
>> This is supported, however, current implementation will run spark jobs
>> per each tenant. So it will have performance concerns with a large number
>> of tenants.
>>
>> Anjana, can you give the details and potential solutions? DAS team has
>> implemented the support for reading tenant data from super tenant so we can
>> run a one Spark Job for all tenants. But we will need to change the queries
>> and figure out how tenant dashboards can get access to results in super
>> tenant's space.
>>
>> Can we do a 5.1 addressing this?
>>
>> --Srinath
>>
>> On Tue, Jun 21, 2016 at 9:15 AM, Kasun Indrasiri  wrote:
>>
>>> Please note that this will be an essential feature for our integration
>>> cloud. So, we need to decide how should we proceed with this soon.
>>>
>>> On Fri, Jun 17, 2016 at 1:57 AM, Kasun Indrasiri  wrote:
>>>
 How about the $subject? I don't think there's no any limitation from
 the mediation engine side on collecting these stats. We need to have this
 for ESB 5.

 --
 Kasun Indrasiri
 Software Architect
 WSO2, Inc.; http://wso2.com
 lean.enterprise.middleware

 cell: +94 77 556 5206
 Blog : http://kasunpanorama.blogspot.com/

>>>
>>>
>>>
>>> --
>>> Kasun Indrasiri
>>> Software Architect
>>> WSO2, Inc.; http://wso2.com
>>> lean.enterprise.middleware
>>>
>>> cell: +94 77 556 5206
>>> Blog : http://kasunpanorama.blogspot.com/
>>>
>>
>>
>>
>> --
>> 
>>WSO2 Inc. http://wso2.com
>>Blog: http://srinathsview.blogspot.com/
>>Photos: http://www.flickr.com/photos/hemapani/ twitter:@srinath_perera
>>Site: http://people.apache.org/~hemapani/
>>
>
>
>
> --
> *Anjana Fernando*
> Senior Technical Lead
> WSO2 Inc. | http://wso2.com
> lean . enterprise . middleware
>



-- 

*S. Suhothayan*
Technical Lead & Team Lead of WSO2 Complex Event Processor
*WSO2 Inc. *http://wso2.com
* *
lean . enterprise . middleware


*cell: (+94) 779 756 757 | blog: http://suhothayan.blogspot.com/
twitter: http://twitter.com/suhothayan
 | linked-in:
http://lk.linkedin.com/in/suhothayan *
___
Dev mailing 

Re: [Dev] ESB Analytics for MT scenarios?

[Anjana Fernando]

Hi,

Yeah, as Srinath mentioned, we do have MT support there, where each ESB
analytics data/queryeis/dashboards would be fully isolated. This is the
usual desirable mode, when different tenants deploy their own analytics
scripts and all, and there is no connection at all with other tenants, and
obviously there shouldn't be. But in our solutions like this, where we
would deploy the exact same solution in each tenant, we can potentially
optimize the scenario by, only doing the processing in one place. This is
actually how APIM batch queries are written at the moment. Where they
altogether stores all the tenant's data in super tenant space and process
and put the summarized there itself. This model works, but then, if the
tenant itself wants to do their own processing it doesn't have access to
the data, that was suppose to be in his space. So a solution for this is
done, which is explained in detail in the mail "Cross Tenant Data Reading
from Spark Queries in DAS", where optionally the super tenant can read
tenant's data which creates a single virtual table view which contains data
from all the tenants. From this, you can read the data, do the processing
and store it in a table in super tenant space. In this approach, you will
still have to have the dashboard in the super tenant space, since the data
resides in the super tenant space. I will be doing some experiments to see
on the feasibility of storing back summarized data in tenant space soon.

Even though we can get the batch scenario working efficiently, there would
be a problem in the CEP side, which we now use heavily in the product
analytics scenario processing. At the moment, there is no concept of having
a single CEP execution plan for all the tenants. When data is published to
receivers, they will be dispatched to their own streams and execution
plans, so we will need to find an approach for this, to optimize the
processing of the CEP side of things. Or else, I'm not sure, since these
are in-memory processing constructs, if having copies of the same execution
plans will actually be okay, compared to having a single one, since the
traffic is anyway split between them.

Cheers,
Anjana.

On Tue, Jun 21, 2016 at 9:35 AM, Srinath Perera  wrote:

> We had a chat about this.
>
> This is supported, however, current implementation will run spark jobs per
> each tenant. So it will have performance concerns with a large number of
> tenants.
>
> Anjana, can you give the details and potential solutions? DAS team has
> implemented the support for reading tenant data from super tenant so we can
> run a one Spark Job for all tenants. But we will need to change the queries
> and figure out how tenant dashboards can get access to results in super
> tenant's space.
>
> Can we do a 5.1 addressing this?
>
> --Srinath
>
> On Tue, Jun 21, 2016 at 9:15 AM, Kasun Indrasiri  wrote:
>
>> Please note that this will be an essential feature for our integration
>> cloud. So, we need to decide how should we proceed with this soon.
>>
>> On Fri, Jun 17, 2016 at 1:57 AM, Kasun Indrasiri  wrote:
>>
>>> How about the $subject? I don't think there's no any limitation from the
>>> mediation engine side on collecting these stats. We need to have this for
>>> ESB 5.
>>>
>>> --
>>> Kasun Indrasiri
>>> Software Architect
>>> WSO2, Inc.; http://wso2.com
>>> lean.enterprise.middleware
>>>
>>> cell: +94 77 556 5206
>>> Blog : http://kasunpanorama.blogspot.com/
>>>
>>
>>
>>
>> --
>> Kasun Indrasiri
>> Software Architect
>> WSO2, Inc.; http://wso2.com
>> lean.enterprise.middleware
>>
>> cell: +94 77 556 5206
>> Blog : http://kasunpanorama.blogspot.com/
>>
>
>
>
> --
> 
>WSO2 Inc. http://wso2.com
>Blog: http://srinathsview.blogspot.com/
>Photos: http://www.flickr.com/photos/hemapani/ twitter:@srinath_perera
>Site: http://people.apache.org/~hemapani/
>



-- 
*Anjana Fernando*
Senior Technical Lead
WSO2 Inc. | http://wso2.com
lean . enterprise . middleware
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] ESB Analytics for MT scenarios?

[Buddhima Wijeweera]

Hi,

At the moment ESB can publish tenant-wise. I tried it with a latest ESB and
following are the steps need to follow:
1. Enable statistics for global level.
2. Add eventpublisher & eventstream files to tenant's deployment folder
3. Create a proxy (or API, inbound EP) and Enable Statistics.
4. Send messages to proxy

Depending on the tenant credentials in eventpublisher files(at point 2
above), statistics  get published to analytics-esb server. (for this
testing I published to super-tenant of analytics-esb).

Thanks,

On Tue, Jun 21, 2016 at 9:37 AM, Srinath Perera  wrote:

> We had a chat about this.
>
> This is supported. However, current implementation will run spark jobs per
> each tenant. So it will have performance concerns with a large number of
> tenants.
>
> Anjana, can you give the details and potential solutions?
>
> Summary is that DAS team has implemented the support for reading tenant
> data from super tenant so we can run a one Spark Job for all tenants. But
> we will need to change the queries and figure out how tenant dashboards can
> get access to results in super tenant's space.
>
> Can we do a 5.1 addressing this?
>
> --Srinath
>
> On Tue, Jun 21, 2016 at 9:15 AM, Kasun Indrasiri  wrote:
>
>> Please note that this will be an essential feature for our integration
>> cloud. So, we need to decide how should we proceed with this soon.
>>
>> On Fri, Jun 17, 2016 at 1:57 AM, Kasun Indrasiri  wrote:
>>
>>> How about the $subject? I don't think there's no any limitation from the
>>> mediation engine side on collecting these stats. We need to have this for
>>> ESB 5.
>>>
>>> --
>>> Kasun Indrasiri
>>> Software Architect
>>> WSO2, Inc.; http://wso2.com
>>> lean.enterprise.middleware
>>>
>>> cell: +94 77 556 5206
>>> Blog : http://kasunpanorama.blogspot.com/
>>>
>>
>>
>>
>> --
>> Kasun Indrasiri
>> Software Architect
>> WSO2, Inc.; http://wso2.com
>> lean.enterprise.middleware
>>
>> cell: +94 77 556 5206
>> Blog : http://kasunpanorama.blogspot.com/
>>
>
>
>
> --
> 
> Srinath Perera, Ph.D.
>http://people.apache.org/~hemapani/
>http://srinathsview.blogspot.com/
>



-- 
Buddhima Wijeweera
Software Engineer; WSO2 Inc.; http://wso2.com ,

Mobile: +94 71 427 9966
Email: buddh...@wso2.com
Blog:   https://buddhimawijeweera.wordpress.com
GitHub Profile: https://github.com/Buddhima
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] ESB Analytics for MT scenarios?

[Srinath Perera]

We had a chat about this.

This is supported. However, current implementation will run spark jobs per
each tenant. So it will have performance concerns with a large number of
tenants.

Anjana, can you give the details and potential solutions?

Summary is that DAS team has implemented the support for reading tenant
data from super tenant so we can run a one Spark Job for all tenants. But
we will need to change the queries and figure out how tenant dashboards can
get access to results in super tenant's space.

Can we do a 5.1 addressing this?

--Srinath

On Tue, Jun 21, 2016 at 9:15 AM, Kasun Indrasiri  wrote:

> Please note that this will be an essential feature for our integration
> cloud. So, we need to decide how should we proceed with this soon.
>
> On Fri, Jun 17, 2016 at 1:57 AM, Kasun Indrasiri  wrote:
>
>> How about the $subject? I don't think there's no any limitation from the
>> mediation engine side on collecting these stats. We need to have this for
>> ESB 5.
>>
>> --
>> Kasun Indrasiri
>> Software Architect
>> WSO2, Inc.; http://wso2.com
>> lean.enterprise.middleware
>>
>> cell: +94 77 556 5206
>> Blog : http://kasunpanorama.blogspot.com/
>>
>
>
>
> --
> Kasun Indrasiri
> Software Architect
> WSO2, Inc.; http://wso2.com
> lean.enterprise.middleware
>
> cell: +94 77 556 5206
> Blog : http://kasunpanorama.blogspot.com/
>



-- 

Srinath Perera, Ph.D.
   http://people.apache.org/~hemapani/
   http://srinathsview.blogspot.com/
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] ESB Analytics for MT scenarios?

[Kasun Indrasiri]

Please note that this will be an essential feature for our integration
cloud. So, we need to decide how should we proceed with this soon.

On Fri, Jun 17, 2016 at 1:57 AM, Kasun Indrasiri  wrote:

> How about the $subject? I don't think there's no any limitation from the
> mediation engine side on collecting these stats. We need to have this for
> ESB 5.
>
> --
> Kasun Indrasiri
> Software Architect
> WSO2, Inc.; http://wso2.com
> lean.enterprise.middleware
>
> cell: +94 77 556 5206
> Blog : http://kasunpanorama.blogspot.com/
>



-- 
Kasun Indrasiri
Software Architect
WSO2, Inc.; http://wso2.com
lean.enterprise.middleware

cell: +94 77 556 5206
Blog : http://kasunpanorama.blogspot.com/
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Dev] ESB Analytics for MT scenarios?

[Kasun Indrasiri]

How about the $subject? I don't think there's no any limitation from the
mediation engine side on collecting these stats. We need to have this for
ESB 5.

-- 
Kasun Indrasiri
Software Architect
WSO2, Inc.; http://wso2.com
lean.enterprise.middleware

cell: +94 77 556 5206
Blog : http://kasunpanorama.blogspot.com/
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev