Hi all, Thanks for the feedbacks.
We cannot say these queries are bounded. In further these queries are tend to be optimised by introducing more constraints. Also, is there any possibilities that searching and getting Summation by grouping? Thanks and Regards. On Tue, Jul 28, 2015 at 1:24 PM, Sinthuja Ragendran <sinth...@wso2.com> wrote: > Hi Gimantha, > > Thanks for the clarification. > > AFAIU APIM spark queries will be anyhow executed and summarized into DAS > tables, and the above mentioned sample query is something going to be > executed against the summarized data dynamically according to user's input. > For example, if the use slides the time-range between some values in APIM, > the the gadgets in the APIM dashboard needs to be updated with SUM of > requests, AVG, etc by filtering the summarized data further for the given > time range. I don't think issuing an SparkSQL query dynamically for such > dynamic changes is viable solution. > > Thanks, > Sinthuja. > > On Tue, Jul 28, 2015 at 1:15 PM, Gimantha Bandara <giman...@wso2.com> > wrote: > >> [Adding Niranda] >> >> Hi Sinduja/Rukshan, >> >> Yes this can be achieved using facets, There are several facets related >> APIs. One is to get the Facet record count. Default behavior is to return >> the number of matching records given the Facet array. We can override the >> default behavior by defining a score function. So each record will >> represent the value of the score function. So id we query for the facet >> count, it will simply return the sum of score function values. This >> behavior is similar to SUM aggregate. But for Aggregates like AVG we need >> the current number of records to calculate the average. We will need to >> make two REST calls ( to get the record count with default score function >> "1" and to get the SUM another record count with the score function >> "aggregating field") since aggregates like AVG are not supported. >> >> As Gihan said, I think Spark SQL will more suitable for this scenario. It >> support basic aggregates and if a custom function required, we can write a >> UDF. >> >> Thanks, >> >> On Tue, Jul 28, 2015 at 11:27 AM, Sinthuja Ragendran <sinth...@wso2.com> >> wrote: >> >>> Hi, >>> >>> One of the main feature of DAS is being decoupled from underline >>> datastore, basically an user should be able to configure RDBMS, cassandra, >>> Hbase/HDFS datastore, the client applications nor analyzation scripts are >>> not necessarily need to change. >>> >>> In APIM 1.9.x and before, we stored the summarized data back into RDBMS >>> to be supported with both BAM and DAS. But when we are moving forward with >>> APIM, IMHO we need to optimize the design based on DAS. >>> >>> On Tue, Jul 28, 2015 at 11:02 AM, Rukshan Premathunga <ruks...@wso2.com> >>> wrote: >>> >>>> Hi all, >>>> >>>> We hope to implement a REST API for WSO2 API manager to expose APIM >>>> statistic data. For that we evaluated WSO2 DAS REST API to check whether we >>>> can use it. >>>> Here are the use cases of the APIM REST API and evaluation result of >>>> DAS REST API. >>>> >>>> *Motivation*: >>>> >>>> Currently we use DAS/BAM to generate summarised data and they are >>>> stored in an RDBMS. >>>> Next, APIM fetch data from the RDBMS and display it on the dashboard. >>>> APIM has implemented Client which fetch the statistic data from the RDBMS, >>>> which is used by the UI to show statistics on the dashboard. >>>> >>>> But we have a new requirement to providing these statistics for a bill >>>> generating application. Since application is separate, we are facing >>>> problems of expose these statistic data to the application. >>>> As a solution we suggest implementing REST API, which can be used to >>>> support above two scenarios (i.e. Current APIM dashboard and the billing >>>> application). >>>> >>>> Other advantages. >>>> >>>> >>>> - Implementing stat dashboard using REST API >>>> - Client side data processing getting reduced >>>> - Possibilities to expose statistic data to others >>>> - Possible to implement appropriate security model >>>> >>>> >>>> *DAS REST API* >>>> >>>> DAS also contain the REST api where we can use it to expose analytics >>>> data of the DAS data analytics layer(DAL). Currently we use DAS to generate >>>> summarised data of the APIM statistics. >>>> >>>> But summarised data are saved in RDBMS since it was easy to fetch data. >>>> But with DAS they have provide the REST API to access DAL data. >>>> Because of this if we used DAS rest api, we can generate and store >>>> summarized data on DAS without using RDBMS. >>>> >>>> This will reduce the APIM configuration for statistics and central >>>> point of DATA storing and processing. >>>> >>>> >>>> Other advantages. >>>> >>>> >>>> - Reduce cost to configure RDBMS >>>> - APIM will depend only on DAS >>>> >>>> >>>> *Issues with DAS REST API* >>>> >>>> Even though DAS provide the REST API to fetch DAL data, query on DAL >>>> data is limited. It provide search mechanism with apache Lucene, but it is >>>> insufficient for APIM requirements. >>>> >>>> >>>> Here are some query examples we used on mysql. >>>> >>>> SELECT api,version,apiPublisher,context,SUM(total_fault_count) as >>>> total_fault_count FROM `table` WHERE time BETWEEN fromDate AND toDate >>>> GROUP BY api,version,apiPublisher,context; >>>> >>>> >>>> Here there is 3 main sub functions and queries >>>> >>>> >>>> - Time between- searching possible with apache Lucene range >>>> search[1] >>>> - Sum - Not possible to calculate the summation >>>> - Group by - impossible to grouping and make the calculation >>>> like SUM, AVG >>>> >>>> >>> AFAIR you can use FACET type here which connects necssary fields >>> api,version,apiPublisher,context together, and then also you can pass a >>> function which is applied on the FACET search result. @Gimantha, please >>> confirm whether it's possible in DAS. - I know we can get the total rows >>> count of facet search, but I'm not very sure about passing the operation to >>> be done on the search result. >>> >>> Thanks, >>> Sinthuja. >>> >>> >>>> With all of these advantages we hope to use to DAS REST API without >>>> implementing APIM own REST API. But to take the advantages of the >>>> REST API we have to have a REST API with above queries supported. >>>> >>>> Can you please provide a proper way to handle these scenarios? >>>> Any suggestions are appreciated. >>>> >>>> [1] https://lucene.apache.org/core/2_9_4/queryparsersyntax.html#Range >>>> Searches >>>> >>>> Thanks and Regards. >>>> -- >>>> Rukshan Chathuranga. >>>> Software Engineer. >>>> WSO2, Inc. >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> Architecture@wso2.org >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> *Sinthuja Rajendran* >>> Associate Technical Lead >>> WSO2, Inc.:http://wso2.com >>> >>> Blog: http://sinthu-rajan.blogspot.com/ >>> Mobile: +94774273955 >>> >>> >>> >> >> >> -- >> Gimantha Bandara >> Software Engineer >> WSO2. Inc : http://wso2.com >> Mobile : +94714961919 >> > > > > -- > *Sinthuja Rajendran* > Associate Technical Lead > WSO2, Inc.:http://wso2.com > > Blog: http://sinthu-rajan.blogspot.com/ > Mobile: +94774273955 > > > -- Rukshan Chathuranga. Software Engineer. WSO2, Inc.
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
