Hi, On Wed, Jun 22, 2016 at 3:25 PM, Isabelle Mauny <isabe...@wso2.com> wrote:
> Great! So that's for Spark.. You mentioned CEP as a big issue as well. Any > hope to fix in CEP too ?? > As per Suho's reply in this thread, it wouldn't be an issue, it just consumed a little bit of extra memory, but performance wise, there shouldn't be a problem. Cheers, Anjana. > > > ------------------------------------------------------------------------------------- > *Isabelle Mauny* > VP, Product Strategy - WSO2, Inc. - http://wso2.com/ > email: isabe...@wso2.com - mobile (Spain) : +34 616050684 - mobile (Sri > Lanka) +94 (0)774777663 > Landline: +1 (650) 745 4499 (USA) or +94 (11) 214 534 (Sri Lanka) > Extension : 7302 > > On Wed, Jun 22, 2016 at 11:50 AM, Srinath Perera <hemap...@gmail.com> > wrote: > >> Perfect!! >> >> On Wed, Jun 22, 2016 at 1:58 PM, Anjana Fernando <anj...@wso2.com> wrote: >> >>> Hi, >>> >>> I've updated the thread "Cross Tenant Data Reading from Spark Queries in >>> DAS", where I've done the implementation on writing back to the tenant >>> space also from the super tenant space as well. This basically just >>> eliminates the current requirement of executing identical Spark scripts in >>> all the tenants, which would not be desirable performance wise, and now, we >>> can just use super tenant's single script to process all the tenant's data >>> and write back the results to their space. Of course, we've to now change >>> the current ESB analytics scripts to use this feature, but it shouldn't be >>> a big change. @Gihan/Supun, can we please check on this and get it done. >>> >>> Cheers, >>> Anjana. >>> >>> On Tue, Jun 21, 2016 at 12:35 PM, Anjana Fernando <anj...@wso2.com> >>> wrote: >>> >>>> Hi, >>>> >>>> Yeah, as Srinath mentioned, we do have MT support there, where each ESB >>>> analytics data/queryeis/dashboards would be fully isolated. This is the >>>> usual desirable mode, when different tenants deploy their own analytics >>>> scripts and all, and there is no connection at all with other tenants, and >>>> obviously there shouldn't be. But in our solutions like this, where we >>>> would deploy the exact same solution in each tenant, we can potentially >>>> optimize the scenario by, only doing the processing in one place. This is >>>> actually how APIM batch queries are written at the moment. Where they >>>> altogether stores all the tenant's data in super tenant space and process >>>> and put the summarized there itself. This model works, but then, if the >>>> tenant itself wants to do their own processing it doesn't have access to >>>> the data, that was suppose to be in his space. So a solution for this is >>>> done, which is explained in detail in the mail "Cross Tenant Data Reading >>>> from Spark Queries in DAS", where optionally the super tenant can read >>>> tenant's data which creates a single virtual table view which contains data >>>> from all the tenants. From this, you can read the data, do the processing >>>> and store it in a table in super tenant space. In this approach, you will >>>> still have to have the dashboard in the super tenant space, since the data >>>> resides in the super tenant space. I will be doing some experiments to see >>>> on the feasibility of storing back summarized data in tenant space soon. >>>> >>>> Even though we can get the batch scenario working efficiently, there >>>> would be a problem in the CEP side, which we now use heavily in the product >>>> analytics scenario processing. At the moment, there is no concept of having >>>> a single CEP execution plan for all the tenants. When data is published to >>>> receivers, they will be dispatched to their own streams and execution >>>> plans, so we will need to find an approach for this, to optimize the >>>> processing of the CEP side of things. Or else, I'm not sure, since these >>>> are in-memory processing constructs, if having copies of the same execution >>>> plans will actually be okay, compared to having a single one, since the >>>> traffic is anyway split between them. >>>> >>>> Cheers, >>>> Anjana. >>>> >>>> On Tue, Jun 21, 2016 at 9:35 AM, Srinath Perera <hemap...@gmail.com> >>>> wrote: >>>> >>>>> We had a chat about this. >>>>> >>>>> This is supported, however, current implementation will run spark jobs >>>>> per each tenant. So it will have performance concerns with a large number >>>>> of tenants. >>>>> >>>>> Anjana, can you give the details and potential solutions? DAS team has >>>>> implemented the support for reading tenant data from super tenant so we >>>>> can >>>>> run a one Spark Job for all tenants. But we will need to change the >>>>> queries >>>>> and figure out how tenant dashboards can get access to results in super >>>>> tenant's space. >>>>> >>>>> Can we do a 5.1 addressing this? >>>>> >>>>> --Srinath >>>>> >>>>> On Tue, Jun 21, 2016 at 9:15 AM, Kasun Indrasiri <ka...@wso2.com> >>>>> wrote: >>>>> >>>>>> Please note that this will be an essential feature for our >>>>>> integration cloud. So, we need to decide how should we proceed with this >>>>>> soon. >>>>>> >>>>>> On Fri, Jun 17, 2016 at 1:57 AM, Kasun Indrasiri <ka...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> How about the $subject? I don't think there's no any limitation from >>>>>>> the mediation engine side on collecting these stats. We need to have >>>>>>> this >>>>>>> for ESB 5. >>>>>>> >>>>>>> -- >>>>>>> Kasun Indrasiri >>>>>>> Software Architect >>>>>>> WSO2, Inc.; http://wso2.com >>>>>>> lean.enterprise.middleware >>>>>>> >>>>>>> cell: +94 77 556 5206 >>>>>>> Blog : http://kasunpanorama.blogspot.com/ >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Kasun Indrasiri >>>>>> Software Architect >>>>>> WSO2, Inc.; http://wso2.com >>>>>> lean.enterprise.middleware >>>>>> >>>>>> cell: +94 77 556 5206 >>>>>> Blog : http://kasunpanorama.blogspot.com/ >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> ============================ >>>>> WSO2 Inc. http://wso2.com >>>>> Blog: http://srinathsview.blogspot.com/ >>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>> twitter:@srinath_perera >>>>> Site: http://people.apache.org/~hemapani/ >>>>> >>>> >>>> >>>> >>>> -- >>>> *Anjana Fernando* >>>> Senior Technical Lead >>>> WSO2 Inc. | http://wso2.com >>>> lean . enterprise . middleware >>>> >>> >>> >>> >>> -- >>> *Anjana Fernando* >>> Senior Technical Lead >>> WSO2 Inc. | http://wso2.com >>> lean . enterprise . middleware >>> >> >> >> >> -- >> ============================ >> WSO2 Inc. http://wso2.com >> Blog: http://srinathsview.blogspot.com/ >> Photos: http://www.flickr.com/photos/hemapani/ twitter:@srinath_perera >> Site: http://people.apache.org/~hemapani/ >> > > -- *Anjana Fernando* Senior Technical Lead WSO2 Inc. | http://wso2.com lean . enterprise . middleware
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
