[Architecture] Changes in road traffic extractor for CEP from Twitter feed
Hi all, We are working on a road traffic extractor from Twitter feed. Here we are extracting useful road traffic information from a public Twitter feed using natural language processing tools and then publishing to WSO2 CEP. Feed was narrowed down to @road_lk Twitter account, popular for posting free crowd sourced traffic alerts. @road_lk feed also was used train NLP models which are utilized for name entity recognition. In real time scenario, users can view/search for road traffic from a web UI and also can subscribe for a location to receive traffic alerts regularly. To generate traffic information available in feed alerts, Twitter feed is processed in two steps. First each tweet will go through the NLP module and location, traffic level will be extracted from it. Then these extractions will be published to CEP as an input event stream. Custom Siddhi queries will further process streams to generate traffic information according to user request which will be published back to web UI or to alerts. We have currently implemented modules given in the diagram. At our last code review it was suggested to make following changes to our current implementation. 1. Use ESB or CEP as the data publishing server instead of a custom server 2. Use existing NLP toolbox for name entity recognition 3. Use ESB Twitter connector for Twitter interactions 4. Integrate implemented extra UI features to existing geo-dashboard Your valuable feedback and suggestions on above changes will be much appreciated. Thanks and best regards. -- *CD Athuraliya* Software Engineering Intern WSO2, Inc. lean . enterprise . middleware Mobile: +94 716288847 LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter <https://twitter.com/cdathuraliya> | Blog <http://cdathuraliya.tumblr.com/> ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] Doing also a Radar Chart for data exploration
Hi all, It was suggested to go for a separate implementation of ParSets [1] using D3 mainly due to possible licence issues in existing JS library [2]. Also due to "provide something significant and lack another something" case. Seems like D3 has a direct implementation Radar Chart. [1] http://www.jasondavies.com/parallel-sets/ [2] https://github.com/jasondavies/d3-parsets/blob/master/LICENSE Thanks. On Sat, Nov 29, 2014 at 7:32 AM, Sumedha Rubasinghe wrote: > This came up during the graph review we had on app manager and API manager > yesterday as well. > In order to get this working, we need to have D3 JS library included. > > We already have gone through multiple graph libs and have stopped at plot. > > So if we need to have this, we need to do a full analysis of what this > provides and not provide over the one we use already. Most of these libs > provide something significant and lack another something that is available > on another lib. So changing from one lib to another is kind of costly > effort. > > Problem with having many libs are due to interdependencies of different > core libs and nonuniformity of graphs. > > Notes at: > "Re: Invitation: Revisit APIManager and AppManager default statistics.." > On Nov 29, 2014 4:41 AM, "Srinath Perera" wrote: > >> Radar charts can do lot of dimensions. Not always very clear with lot of >> data, but can be OK if we use right colours etc. Should also try out IMO. >> >> Radar chart >> *https://github.com/alangrafu/radar-chart-d3* >> <https://github.com/alangrafu/radar-chart-d3> >> http://bl.ocks.org/nbremer/6506614 >> >> -- >> >> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >> Site: http://people.apache.org/~hemapani/ >> Photos: http://www.flickr.com/photos/hemapani/ >> Phone: 0772360902 >> >> ___ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > ___ > Architecture mailing list > Architecture@wso2.org > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *CD Athuraliya* Software Engineering Intern WSO2, Inc. lean . enterprise . middleware Mobile: +94 716288847 LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter <https://twitter.com/cdathuraliya> | Blog <http://cdathuraliya.tumblr.com/> ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
[Architecture] WSO2 ML 1.0.0 Milestone 2 Released
WSO2 ML 1.0.0 Milestone 2 is Released! WSO2 ML team is pleased to announce the 2nd Milestone of WSO2 ML 1.0.0. The distribution is available at [1]. This release includes following features. New features - Initial version of the User Interface ( https://docs.wso2.com/display/ML100/ML+UI+Workflow) - Support analyses execution in an external Spark cluster ( https://docs.wso2.com/display/ML100/Connecting+to+an+External+Apache+Spark+Cluster ) - Support for HDFS storage Pending features - BAM integration - UI - Model comparison, data exploration - ESB - ML mediator - CEP - ML extension Documentation for WSO2 Machine Learner 1.0.0 - M2 - Introduction - https://docs.wso2.com/display/ML100/Introducing+Machine+Learner - Architecture Guide - https://docs.wso2.com/display/ML100/Architecture - REST API guide - https://docs.wso2.com/display/ML100/REST+API+Guide - User Guide - https://docs.wso2.com/display/ML100/User+Guide - ML UI (initial version) Guide - https://docs.wso2.com/display/ML100/ML+UI+Workflow - Sample ML Flows - https://docs.wso2.com/display/ML100/Samples Please report any issues you may find in our JIRA: https://wso2.org/jira/browse/ML GIT tag: https://github.com/wso2/product-ml/releases/tag/1.0.0-m2 [1] *https://github.com/wso2/product-ml/releases/download/1.0.0-m2/wso2ml-1.0.0-SNAPSHOT.zip <https://github.com/wso2/product-ml/releases/download/1.0.0-m2/wso2ml-1.0.0-SNAPSHOT.zip>* Thanks, -- *CD Athuraliya* Software Engineer WSO2, Inc. lean . enterprise . middleware Mobile: +94 716288847 <94716288847> LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter <https://twitter.com/cdathuraliya> | Blog <http://cdathuraliya.tumblr.com/> ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] WSO2 Machine Learner - Model Generation Process
7:30 PM, Nirmal Fernando >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> Please find the model generation process of WSO2 ML as a diagram >>>>>>>>> below [1]. Model Generation is one of the key use-cases of WSO2 ML. >>>>>>>>> Model Generation >>>>>>>>> >>>>>>>>> When you login to the WSO2 ML, you will be redirected to a page >>>>>>>>> where you can see two main entities of WSO2 ML, namely, Datasets and >>>>>>>>> Projects. When you navigate to the Projects, you could see all the >>>>>>>>> Analyses >>>>>>>>> created for that project. Analysis is a workflow created to cover the >>>>>>>>> full >>>>>>>>> machine learning pipeline from data extraction to model generation. >>>>>>>>> When >>>>>>>>> you create a new Analysis, you could see all the available Value-sets >>>>>>>>> for >>>>>>>>> the logged-in user. You pick a Value set and start the process of >>>>>>>>> analysing >>>>>>>>> that Value-set. Then you will be navigated to the data extraction >>>>>>>>> interface, where you are getting a chance, either to pick an existing >>>>>>>>> customized feature-set or to create a new customized feature-set. >>>>>>>>> Once you >>>>>>>>> have picked a feature-set, you could provide the model configuration. >>>>>>>>> Based >>>>>>>>> on the model configuration, now you could calibrate the corresponding >>>>>>>>> hyper >>>>>>>>> parameters. Once you have a value-set, customized feature-set, a model >>>>>>>>> configuration and a calibrated set of hyper-parameters, you could >>>>>>>>> generate >>>>>>>>> a ML model. >>>>>>>>> >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Thanks & regards, >>>>>>>>> Nirmal >>>>>>>>> >>>>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc. >>>>>>>>> Mobile: +94715779733 >>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>> Phone: 0772360902 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Thanks & regards, >>>>>>> Nirmal >>>>>>> >>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc. >>>>>>> Mobile: +94715779733 >>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Dhanuka De Silva >>>>>> m: +94 773 887816 >>>>>> e: dhanu...@wso2.com >>>>>> w: http://www.wso2.com >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Upul Bandara, >>>>> Associate Technical Lead, WSO2, Inc., >>>>> Mob: +94 715 468 345. >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Thanks & regards, >>>> Nirmal >>>> >>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc. >>>> Mobile: +94715779733 >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>>> >>>> >>> >>> >>> -- >>> Dhanuka De Silva >>> m: +94 773 887816 >>> e: dhanu...@wso2.com >>> w: http://www.wso2.com >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Senior Software Engineer- Platform Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > > Blog: http://srinathsview.blogspot.com twitter:@srinath_perera > Site: http://people.apache.org/~hemapani/ > Photos: http://www.flickr.com/photos/hemapani/ > Phone: 0772360902 > -- *CD Athuraliya* Software Engineer WSO2, Inc. lean . enterprise . middleware Mobile: +94 716288847 <94716288847> LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter <https://twitter.com/cdathuraliya> | Blog <http://cdathuraliya.tumblr.com/> ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] [IGVIZ.JS] : D3.js Based Interactive Generic Visualization Library
>>>>>>>>> //chartConfig-minimal >>>>>>>>>>>>>>> set of parameters >>>>>>>>>>>>>>> chart.setXaxis(axisConfig) >>>>>>>>>>>>>>> .setYaxis(yaxisConfig) >>>>>>>>>>>>>>> .setDimension(dimensionConfig) >>>>>>>>>>>>>>> //optional visual customization >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> chart.plot( dataTable.data, callBack); //draw the >>>>>>>>>>>>>>> charts based on the parameters >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> chart.update(newData) >>>>>>>>>>>>>>> //real-time updating >>>>>>>>>>>>>>> chart.resize()//Responsive >>>>>>>>>>>>>>> Handler >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] http://dunithd.github.io/igviz/samples/ >>>>>>>>>>>>>>> [2] https://github.com/dunithd/igviz >>>>>>>>>>>>>>> [3] Generic UI Parameter Names >>>>>>>>>>>>>>> <https://docs.google.com/a/wso2.com/spreadsheets/d/1tCf3lTZ7kkEXdK7T75tKEXqm0QhsZZpiF47saKNMboc/edit#gid=0> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Mar 3, 2015 at 6:02 AM, Tharindu Munasinghe < >>>>>>>>>>>>>>> tharin...@wso2.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> +1 , I will do a bar chart sample using 'Vega' with the >>>>>>>>>>>>>>>> features we have discussed and then we will see how we can >>>>>>>>>>>>>>>> extend that >>>>>>>>>>>>>>>> implementation to the other types of charts :) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, Mar 2, 2015 at 6:05 PM, Dunith Dhanushka < >>>>>>>>>>>>>>>> dun...@wso2.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Today we had a discussion on using Vega [1] library >>>>>>>>>>>>>>>>> underneath igviz. Vega comes with a highlevel visualization >>>>>>>>>>>>>>>>> grammer that >>>>>>>>>>>>>>>>> runs on top of D3. In addition to that it's got some polished >>>>>>>>>>>>>>>>> UI so that >>>>>>>>>>>>>>>>> it'll save more efforts for fine tuning the charts. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Please refer mail thread [RFC: Building a Generic >>>>>>>>>>>>>>>>> Configurable UI Gadget for Analytics] for a background on >>>>>>>>>>>>>>>>> IGviz. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Igviz's main API interface is >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> var chart = igviz.plot(canvas,chartConfig,dataTable); >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Plan is to keep Igviz's plot() method unchanged and >>>>>>>>>>>>>>>>> transform DataTable and chartConfig objects to the format >>>>>>>>>>>>>>>>> that vega expects. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> where canvas is the div element that chart is being drawn, >>>>>>>>>>>>>>>>> chartConfig is the JSON object that contain parameters >>>>>>>>>>>>>>>>> required to draw the chart. (E.g xAxis, yAxis, width, height) >>>>>>>>>>>>>>>>> dataTable is the JSON formatted tabular data structure >>>>>>>>>>>>>>>>> that passes data for the chart >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Following are the discussed action items: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1. Keep the current DataTable structure intact and have >>>>>>>>>>>>>>>>> Igviz trasform it to the format that vega expects. >>>>>>>>>>>>>>>>> 2. Depending on the chartConfig.chartType, create Vega's >>>>>>>>>>>>>>>>> marks[] components. >>>>>>>>>>>>>>>>> 3. Explore the possibility of doing drill downs using >>>>>>>>>>>>>>>>> Vega. (Tharindu has already done that on Igviz) >>>>>>>>>>>>>>>>> 4. Figure out a way of upadting the chart dynamically. We >>>>>>>>>>>>>>>>> agreed to make DataTable's schema fixed during the update. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Tharindu, please add anything I missed here. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [1] http://trifacta.github.io/vega/ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Dunith >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Mon, Mar 2, 2015 at 10:52 AM, Tharindu Munasinghe < >>>>>>>>>>>>>>>>> tharin...@wso2.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>>> Pubudu,Fawsan and Fasna has started this project as >>>>>>>>>>>>>>>>>> their training project and they did the implementations of >>>>>>>>>>>>>>>>>> table, single >>>>>>>>>>>>>>>>>> value , map , line charts . Me and Dunith have been >>>>>>>>>>>>>>>>>> continuing this >>>>>>>>>>>>>>>>>> project to make it a generic library so that any product can >>>>>>>>>>>>>>>>>> use it. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> So far igviz.js library supports bar charts >>>>>>>>>>>>>>>>>> ,scatter plots, tables , maps , single number graph , line >>>>>>>>>>>>>>>>>> and area charts >>>>>>>>>>>>>>>>>> with drill-downing options for hierarchical data sets . >>>>>>>>>>>>>>>>>> Please find the samples hosted at [1] and the >>>>>>>>>>>>>>>>>> implementation at [2]. Documentation for the configuration >>>>>>>>>>>>>>>>>> parameters can >>>>>>>>>>>>>>>>>> be found at [3] . >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [1] igviz.js samples >>>>>>>>>>>>>>>>>> <http://dunithd.github.io/igviz/samples/> >>>>>>>>>>>>>>>>>> [2] igviz.js source code >>>>>>>>>>>>>>>>>> <https://github.com/dunithd/igviz> >>>>>>>>>>>>>>>>>> [3] GenericUIParameterNames >>>>>>>>>>>>>>>>>> <https://docs.google.com/a/wso2.com/spreadsheets/d/1tCf3lTZ7kkEXdK7T75tKEXqm0QhsZZpiF47saKNMboc/edit#gid=0> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> *Tharindu Munasinghe* >>>>>>>>>>>>>>>>>> Software Engineering Intern >>>>>>>>>>>>>>>>>> WSO2 Inc.(http://wso2.com) >>>>>>>>>>>>>>>>>> SRI LANKA >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Mobile: +94 77460887 >>>>>>>>>>>>>>>>>> E-Mail: tharin...@wso2.com >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Dunith Dhanushka, >>>>>>>>>>>>>>>>> Senior Software Engineer - BAM, >>>>>>>>>>>>>>>>> WSO2 Inc, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Mobile - +94 71 8615744 >>>>>>>>>>>>>>>>> Blog - dunithd.wordpress.com <http://blog.dunith.com> >>>>>>>>>>>>>>>>> Twitter - @dunithd <http://twitter.com/dunithd> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> *Tharindu Munasinghe* >>>>>>>>>>>>>>>> Software Engineering Intern >>>>>>>>>>>>>>>> WSO2 Inc.(http://wso2.com) >>>>>>>>>>>>>>>> SRI LANKA >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Mobile: +94 77460887 >>>>>>>>>>>>>>>> E-Mail: tharin...@wso2.com >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> *Tharindu Munasinghe* >>>>>>>>>>>>>>> Software Engineering Intern >>>>>>>>>>>>>>> WSO2 Inc.(http://wso2.com) >>>>>>>>>>>>>>> SRI LANKA >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Mobile: +94 77460887 >>>>>>>>>>>>>>> E-Mail: tharin...@wso2.com >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> *Tharindu Munasinghe* >>>>>>>>>>>>>> Software Engineering Intern >>>>>>>>>>>>>> WSO2 Inc.(http://wso2.com) >>>>>>>>>>>>>> SRI LANKA >>>>>>>>>>>>>> >>>>>>>>>>>>>> Mobile: +94 77460887 >>>>>>>>>>>>>> E-Mail: tharin...@wso2.com >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> *Tharindu Munasinghe* >>>>>>>>>>> Software Engineering Intern >>>>>>>>>>> WSO2 Inc.(http://wso2.com) >>>>>>>>>>> SRI LANKA >>>>>>>>>>> >>>>>>>>>>> Mobile: +94 77460887 >>>>>>>>>>> E-Mail: tharin...@wso2.com >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *Tharindu Munasinghe* >>>>>>>>>> Software Engineering Intern >>>>>>>>>> WSO2 Inc.(http://wso2.com) >>>>>>>>>> SRI LANKA >>>>>>>>>> >>>>>>>>>> Mobile: +94 77460887 >>>>>>>>>> E-Mail: tharin...@wso2.com >>>>>>>>>> >>>>>>>>>> ___ >>>>>>>>>> Architecture mailing list >>>>>>>>>> Architecture@wso2.org >>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Dunith Dhanushka, >>>>>>>>> Senior Software Engineer - BAM, >>>>>>>>> WSO2 Inc, >>>>>>>>> >>>>>>>>> Mobile - +94 71 8615744 >>>>>>>>> Blog - dunithd.wordpress.com <http://blog.dunith.com> >>>>>>>>> Twitter - @dunithd <http://twitter.com/dunithd> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ___ >>>>>>>> Architecture mailing list >>>>>>>> Architecture@wso2.org >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Tharindu Munasinghe* >>>>>>> Software Engineering Intern >>>>>>> WSO2 Inc.(http://wso2.com) >>>>>>> SRI LANKA >>>>>>> >>>>>>> Mobile: +94 77460887 >>>>>>> E-Mail: tharin...@wso2.com >>>>>>> >>>>>>> ___ >>>>>>> Architecture mailing list >>>>>>> Architecture@wso2.org >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> ___ >>>>> Architecture mailing list >>>>> Architecture@wso2.org >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Tharindu Munasinghe* >>>> Software Engineering Intern >>>> WSO2 Inc.(http://wso2.com) >>>> SRI LANKA >>>> >>>> Mobile: +94 77460887 >>>> E-Mail: tharin...@wso2.com >>>> >>> >>> >> >> >> -- >> *Tharindu Munasinghe* >> Software Engineering Intern >> WSO2 Inc.(http://wso2.com) >> SRI LANKA >> >> Mobile: +94 77460887 >> E-Mail: tharin...@wso2.com >> > > > > -- > *Tharindu Munasinghe* > Software Engineering Intern > WSO2 Inc.(http://wso2.com) > SRI LANKA > > Mobile: +94 77460887 > E-Mail: tharin...@wso2.com > > ___ > Architecture mailing list > Architecture@wso2.org > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *CD Athuraliya* Software Engineer WSO2, Inc. lean . enterprise . middleware Mobile: +94 716288847 <94716288847> LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter <https://twitter.com/cdathuraliya> | Blog <http://cdathuraliya.tumblr.com/> ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] ML Model Summary Illustration and Comparison
Hi all, With what chart types and implementations we are going to proceed for alpha? We will be able to finalize comparison and summery views with them. Thanks, CD On Fri, May 1, 2015 at 9:39 AM, Supun Sethunga wrote: > Hi Nirmal, > > During the last discussion, what we decided was to, show some numerical > value (Accuracy / Std error) next to each model to illustrate the > performance in the model listing view, so that user can get an overall idea > at one glance. And in a separate page, have the ROC comparison. Think we > still need to figure out where would the later fit in, in the UI > navigation.. > > Thanks, > Supun > > On Thu, Apr 30, 2015 at 6:51 PM, Nirmal Fernando wrote: > >> Thanks for summarizing Supun. Did we think about how we gonna create the >> cross-model comparisons view? >> >> On Thu, Apr 30, 2015 at 8:33 AM, Supun Sethunga wrote: >> >>> [-strategy@, +architecture@] >>> >>> On Thu, Apr 30, 2015 at 5:58 PM, Srinath Perera >>> wrote: >>> >>>> should go to arch@ >>>> >>>> On Thu, Apr 30, 2015 at 6:28 AM, Srinath Perera >>>> wrote: >>>> >>>>> Thanks Supun!! this looks good. >>>>> >>>>> --Srinath >>>>> >>>>> On Thu, Apr 30, 2015 at 6:25 AM, Supun Sethunga >>>>> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> Following is the break down of the Model Summary illustrations that >>>>>> can be supported by ML at the moment. Initiating this thread to finalize >>>>>> on >>>>>> what we can support and what cannot, with the initial release. Blue >>>>>> colored >>>>>> ones are yet to implement. >>>>>> >>>>>>- Numerical Prediction >>>>>> - Standard Error [1] >>>>>> - Residual Plot [2] >>>>>> - Feature Importance (*Graph containing weights assigned to >>>>>> each of the feature in the model*) >>>>>> >>>>>> >>>>>>- Classification: >>>>>>- Binary >>>>>> - ROC [3] >>>>>> - AUC >>>>>> - Confusion Matrix (*Available on spark as a static metric. >>>>>> But if this was calculated manually, it can be made >>>>>> interactive, so that >>>>>> user can find the optimal threshold*) >>>>>> - Accuracy >>>>>> - Feature Importance >>>>>> - Multi-Class >>>>>> - Confusion Matrix (*Available on spark*) >>>>>> - Accuracy >>>>>> - Feature Importance >>>>>> >>>>>> >>>>>>- Clustering >>>>>> - Scatter plot with clustered points >>>>>> >>>>>> >>>>>> *Cross-comparing Models* >>>>>> >>>>>> As you can see, major limitation we have when cross comparing models >>>>>> within a project is, different categories have different summary >>>>>> statistics/plots, and hence we cannot compare two models in two >>>>>> categories. >>>>>> >>>>>> Following are the possibilities: >>>>>> >>>>>>- ROC can be used to compare Binary classification models. >>>>>>- Cobweb (a radar chart) can be used to compare Multi-Class >>>>>>classification models (This is the possible alternative for ROC >>>>>>in multi-class case. But the drawback is, the graph will be very >>>>>> unclear >>>>>>when there are excess amounts of features in the models). [4] [5] >>>>>>- Accuracy can be used to compare all classification models. >>>>>> >>>>>> Please add if I've missed anything. >>>>>> >>>>>> *Ref:* >>>>>> [1] http://onlinestatbook.com/2/regression/accuracy.html >>>>>> [2] http://stattrek.com/regression/residual-analysis.aspx >>>>>> [3] >>>>>> http://www.sciencedirect.com/science/article/pii/S016786550500303X >>>>>> [4] >>>>>> http://www.academia.edu/2519022/Visualization_and_analysis_of_classifiers_performance_in_multi-class_
Re: [Architecture] ML Model Summary Illustration and Comparison
Hi all, We have implemented model comparison for classification and numerical prediction with following measures. - Binary and multiclass classification - Accuracy - Numerical prediction - Mean squared error We are currently working on a sorted view of models according to their accuracy/MSE. This release will not support cross comparison for clustering algorithms. Thanks, CD On Tue, May 5, 2015 at 5:41 PM, CD Athuraliya wrote: > Hi all, > > With what chart types and implementations we are going to proceed for > alpha? We will be able to finalize comparison and summery views with them. > > Thanks, > CD > > On Fri, May 1, 2015 at 9:39 AM, Supun Sethunga wrote: > >> Hi Nirmal, >> >> During the last discussion, what we decided was to, show some numerical >> value (Accuracy / Std error) next to each model to illustrate the >> performance in the model listing view, so that user can get an overall idea >> at one glance. And in a separate page, have the ROC comparison. Think we >> still need to figure out where would the later fit in, in the UI >> navigation.. >> >> Thanks, >> Supun >> >> On Thu, Apr 30, 2015 at 6:51 PM, Nirmal Fernando wrote: >> >>> Thanks for summarizing Supun. Did we think about how we gonna create the >>> cross-model comparisons view? >>> >>> On Thu, Apr 30, 2015 at 8:33 AM, Supun Sethunga wrote: >>> >>>> [-strategy@, +architecture@] >>>> >>>> On Thu, Apr 30, 2015 at 5:58 PM, Srinath Perera >>>> wrote: >>>> >>>>> should go to arch@ >>>>> >>>>> On Thu, Apr 30, 2015 at 6:28 AM, Srinath Perera >>>>> wrote: >>>>> >>>>>> Thanks Supun!! this looks good. >>>>>> >>>>>> --Srinath >>>>>> >>>>>> On Thu, Apr 30, 2015 at 6:25 AM, Supun Sethunga >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> Following is the break down of the Model Summary illustrations that >>>>>>> can be supported by ML at the moment. Initiating this thread to >>>>>>> finalize on >>>>>>> what we can support and what cannot, with the initial release. Blue >>>>>>> colored >>>>>>> ones are yet to implement. >>>>>>> >>>>>>>- Numerical Prediction >>>>>>> - Standard Error [1] >>>>>>> - Residual Plot [2] >>>>>>> - Feature Importance (*Graph containing weights assigned to >>>>>>> each of the feature in the model*) >>>>>>> >>>>>>> >>>>>>>- Classification: >>>>>>>- Binary >>>>>>> - ROC [3] >>>>>>> - AUC >>>>>>> - Confusion Matrix (*Available on spark as a >>>>>>> static metric. But if this was calculated manually, it can be >>>>>>> made >>>>>>> interactive, so that user can find the optimal threshold*) >>>>>>> - Accuracy >>>>>>> - Feature Importance >>>>>>> - Multi-Class >>>>>>> - Confusion Matrix (*Available on spark*) >>>>>>> - Accuracy >>>>>>> - Feature Importance >>>>>>> >>>>>>> >>>>>>>- Clustering >>>>>>> - Scatter plot with clustered points >>>>>>> >>>>>>> >>>>>>> *Cross-comparing Models* >>>>>>> >>>>>>> As you can see, major limitation we have when cross comparing models >>>>>>> within a project is, different categories have different summary >>>>>>> statistics/plots, and hence we cannot compare two models in two >>>>>>> categories. >>>>>>> >>>>>>> Following are the possibilities: >>>>>>> >>>>>>>- ROC can be used to compare Binary classification models. >>>>>>>- Cobweb (a radar chart) can be used to compare Multi-Class >>>>>>>classification models (This is the possible alternative for ROC >>>>>>>in multi-class case. But the drawback is, th
[Architecture] WSO2 ML 1.0.0 Beta-1 Released
WSO2 ML 1.0.0 Beta-1 Released! WSO2 ML team is pleased to announce the Beta-1 release of WSO2 ML 1.0.0. The distribution is available at [1]. This release includes following bug fixes, improvements. Bug - [ML-8 <https://wso2.org/jira/browse/ML-8>] - [UI] Make dataset versions non-clickable - [ML-9 <https://wso2.org/jira/browse/ML-9>] - [UI] Step 1 of an Analysis generates graphs only for first 10 features - [ML-21 <https://wso2.org/jira/browse/ML-21>] - Create analysis fails immediately after deletion of same name - [ML-22 <https://wso2.org/jira/browse/ML-22>] - Model building fails when column headers of CSV are Quoted - [ML-28 <https://wso2.org/jira/browse/ML-28>] - Make Title in Scatter Plot (Exploration) Prominant - [ML-31 <https://wso2.org/jira/browse/ML-31>] - Exploration UI needs improvements - [ML-44 <https://wso2.org/jira/browse/ML-44>] - [UI] Changing feature type doesn't change the graph type - [ML-53 <https://wso2.org/jira/browse/ML-53>] - [PredictedVsActual Graph] Mix up features of different projects - [ML-54 <https://wso2.org/jira/browse/ML-54>] - Project name should be validated for global uniqueness within a tenant - [ML-55 <https://wso2.org/jira/browse/ML-55>] - Remove all unused JAG files etc. from ML Jaggery App - [ML-62 <https://wso2.org/jira/browse/ML-62>] - [Server Startup] logging of the ML app url dependency is confusing - [ML-63 <https://wso2.org/jira/browse/ML-63>] - Dataset explore page show error when column headers of CSV quoted - [ML-64 <https://wso2.org/jira/browse/ML-64>] - Number format exception when trying to build model - [ML-65 <https://wso2.org/jira/browse/ML-65>] - [UI] Errors in input validation for model prediction - [ML-66 <https://wso2.org/jira/browse/ML-66>] - Model building fails when one or more categorial features are not included - [ML-67 <https://wso2.org/jira/browse/ML-67>] - Summary stats charts are not updated with type selection - [ML-70 <https://wso2.org/jira/browse/ML-70>] - [Windows] Dataset upload failed Improvement - [ML-24 <https://wso2.org/jira/browse/ML-24>] - Implement automated back-end tests - [ML-40 <https://wso2.org/jira/browse/ML-40>] - Suggest an appropriate value for the number of classes in a Decision tree etc. algorithms - [ML-47 <https://wso2.org/jira/browse/ML-47>] - Pagination support to ML model listing page - [ML-56 <https://wso2.org/jira/browse/ML-56>] - Support logistic regression with L-BFGS - [ML-57 <https://wso2.org/jira/browse/ML-57>] - Moving common javascript code to one place - [ML-68 <https://wso2.org/jira/browse/ML-68>] - [Spark Logs] Move spark logs to spark.log file and skip wso2carbon.log - [ML-69 <https://wso2.org/jira/browse/ML-69>] - [UI] Delete confirmations - [ML-71 <https://wso2.org/jira/browse/ML-71>] - [UI] Improvements in Model Comparison UI - [ML-72 <https://wso2.org/jira/browse/ML-72>] - Upload data points for prediction using a file - [ML-73 <https://wso2.org/jira/browse/ML-73>] - [UI] Disable Model Summary page for clustering algorithms - [ML-74 <https://wso2.org/jira/browse/ML-74>] - Downloaded model should have the same name as model name Task - [ML-59 <https://wso2.org/jira/browse/ML-59>] - Fix critical issues reported by Sonar - [ML-60 <https://wso2.org/jira/browse/ML-60>] - Integrate carbon metrics to ML - [ML-61 <https://wso2.org/jira/browse/ML-61>] - Remove unused bam server url element from machine-learner.xml The documentation for ML 1.0.0 Beta-1 can be found at [2]. Please report any issues you may find in our JIRA [3]. [1] https://github.com/wso2/product-ml/releases/tag/1.0.0-beta1 [2] https://docs.wso2.com/display/ML100/WSO2+Machine+Learner+Documentation [3] https://wso2.org/jira/browse/ML - WSO2 ML Team - -- *CD Athuraliya* Software Engineer WSO2, Inc. lean . enterprise . middleware Mobile: +94 716288847 <94716288847> LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter <https://twitter.com/cdathuraliya> | Blog <http://cdathuraliya.tumblr.com/> ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
[Architecture] WSO2 ML 1.0.0 Beta-2 Released
WSO2 ML 1.0.0 Beta-2 Released! WSO2 ML team is pleased to announce the Beta-2 release of WSO2 ML 1.0.0. The distribution is available at [1] and ML P2 repository at [2]. This release includes following bug fixes and improvements. Bug - [ML-77 <https://wso2.org/jira/browse/ML-77>] - Generated model's weight vector should be verified for not null and not infinite - [ML-78 <https://wso2.org/jira/browse/ML-78>] - Canceling a new model created by existing analysis deletes the analysis - [ML-80 <https://wso2.org/jira/browse/ML-80>] - New feature indices should be available via the built model - [ML-82 <https://wso2.org/jira/browse/ML-82>] - Feature Importance graph doesn't show all features sometimes - [ML-85 <https://wso2.org/jira/browse/ML-85>] - rename BAM option -> DAS in dataset upload feature - [ML-89 <https://wso2.org/jira/browse/ML-89>] - Predict Mediator 'Help' page is missing - [ML-90 <https://wso2.org/jira/browse/ML-90>] - [Mediator] Namespaces doesn't get appear in namespace editor - [ML-91 <https://wso2.org/jira/browse/ML-91>] - [Mediator] WARN - MLIOFactory Failed to load/instantiate the class: null getting printed due to improper usage of MLIOFactory - [ML-92 <https://wso2.org/jira/browse/ML-92>] - [Mediator] Updated feature xpath expressions are not getting updated - [ML-96 <https://wso2.org/jira/browse/ML-96>] - Bundle predict mediator feature to p2-repo - [ML-97 <https://wso2.org/jira/browse/ML-97>] - Cluster diagram doesn't work in DAS case. Improvement - [ML-75 <https://wso2.org/jira/browse/ML-75>] - Add favicon to ML UI - [ML-76 <https://wso2.org/jira/browse/ML-76>] - Models of an analysis page should be in reverse chronological order - [ML-79 <https://wso2.org/jira/browse/ML-79>] - [UI] Upload data points for prediction using a file - [ML-81 <https://wso2.org/jira/browse/ML-81>] - [UI] Sanitize dynamically generated content using an encoding function - [ML-83 <https://wso2.org/jira/browse/ML-83>] - In DAS case, we have to extract feature names from the table schema - [ML-84 <https://wso2.org/jira/browse/ML-84>] - Refactor input and output adapters to work with String paths instead of URIs - [ML-86 <https://wso2.org/jira/browse/ML-86>] - Styles for ML Carbon console - [ML-87 <https://wso2.org/jira/browse/ML-87>] - Make default folder configurations OS independent - [ML-88 <https://wso2.org/jira/browse/ML-88>] - Create a Spark Commons feature and use it from ML and DAS - [ML-93 <https://wso2.org/jira/browse/ML-93>] - [DAS] API to retrieve all table names of a tenant - [ML-94 <https://wso2.org/jira/browse/ML-94>] - Confusion matrix for multiclass classification models - [ML-99 <https://wso2.org/jira/browse/ML-99>] - List DAS tables to select a particular table when creating dataset from DAS Task - [ML-95 <https://wso2.org/jira/browse/ML-95>] - Get all carbon kernel patches to ML - [ML-98 <https://wso2.org/jira/browse/ML-98>] - Enable UI tests for Jenkins The documentation for ML 1.0.0 Beta-2 can be found at [3]. Please report any issues you may find in our JIRA [4]. [1] https://github.com/wso2/product-ml/releases/download/1.0.0-beta2/wso2ml-1.0.0-SNAPSHOT.zip [2] https://github.com/wso2/product-ml/releases/download/1.0.0-beta2/p2-repo.zip [3] https://docs.wso2.com/display/ML100/WSO2+Machine+Learner+Documentation [4] https://wso2.org/jira/browse/ML - WSO2 ML Team - -- *CD Athuraliya* Software Engineer WSO2, Inc. lean . enterprise . middleware Mobile: +94 716288847 <94716288847> LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter <https://twitter.com/cdathuraliya> | Blog <http://cdathuraliya.tumblr.com/> ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] [ML] PMML support for Machine Learner
Hi,
I feel that asking user to go through the complete ML workflow for PMML is
too demanding. Computationally this conversion should be less expensive
compared to model training in real world use cases (since it's a mapping of
model parameters from Java objects to XML AFAIK). And model training should
be independent from the model format. Instead can't we support this
conversion on demand? Or save in both formats for now? Once Spark starts
supporting PMML for all algorithms we can go for Method 1 if it looks
consistent through out our ML life cycle.
Thanks
On Mon, Oct 12, 2015 at 11:09 AM, Fazlan Nazeem wrote:
> Hi,
>
> I am working on redmine[1] regarding PMML support for Machine Learner.
> Please provide your opinion on this design.
> [1]https://redmine.wso2.com/issues/4303
>
> *Overview*
>
> Spark 1.5.1(lastest version) supports PMML model export for some of the
> available models in Spark through MLlib.
>
> The table below outlines the MLlib models that can be exported to PMML and
> their equivalent PMML model.
>
>
>
> MLlib model
>
> PMML model
>
> KMeansModel
>
> ClusteringModel
>
> LinearRegressionModel
>
> RegressionModel (functionName="regression")
>
> RidgeRegressionModel
>
> RegressionModel (functionName="regression")
>
> LassoModel
>
> RegressionModel (functionName="regression")
>
> SVMModel
>
> RegressionModel (functionName="classification" normalizationMethod="none")
>
> Binary LogisticRegressionModel
>
> RegressionModel (functionName="classification" normalizationMethod="logit")
>
> Not all models available in MLlib can be exported to PMML as of now.
> Goal
>
>1.
>
>We need to save models generated by WSO2 ML(PMML supported models) in
>PMML format, so that those could be reused from PMML supported tools.
>
> How To
>
> if “clusters” is the trained model, we can do the following with the PMML
> support.
>
> // Export the model to a String in PMML format
> clusters.toPMML
>
> // Export the model to a local file in PMML format
> clusters.toPMML("/tmp/kmeans.xml")
>
> // Export the model to a directory on a distributed file system in PMML
> format
> clusters.toPMML(sc,"/tmp/kmeans")
>
> // Export the model to the OutputStream in PMML format
> clusters.toPMML(System.out)
>
> For unsupported models, either you will not find a .toPMML method or an
> IllegalArgumentException will be thrown.
> Design
>
> In the following diagram models highlighted in green can be exported to
> PMML, but not the models highlighted in red. The diagram illustrates
> algorithms supported by WSO2 Machine Learner.
>
> [image: Inline image 2]
>
>
> Method 1
>
> By default save the models in PMML if PMML export is supported, using one
> of these supported options.
>
> 1. Export the model to a String in PMML format
> 2. Export the model to a local file in PMML format
> 3. Export the model to a directory on a distributed file system in PMML
> format
> 4 . Export the model to the OutputStream in PMML format
>
> Classes need to be modified (apart from UI)
>
>-
>
>SupervisedSparkModelBuilder
>-
>
>UnsupervisedSparkModelBuilder
>
>
> e.g
>
> [image: Inline image 1]
>
> As of now the serialized models are saved in “models” folder. The PMML
> models can also be saved in the same directory with a PMML suffix.
>
> optional:
>
> After the model is generated let the user export the PMML model to a
> chosen location through the UI.
>
> Method 2
>
> Add a *new REST API* to build models with PMML
>
> public Response buildPMMLModel(@PathParam("modelId") long modelId)
>
> in the backend we could add an additional argument to "buildXModel"
> methods to decide whether to save the PMML model or not.
>
> UI modifications also needed (An option for the user to choose whether to
> build the PMML and to choose the path to save it)
>
> Identified classes need to be modified (apart from UI)
>
>-
>
>SupervisedSparkModelBuilder
>-
>
>UnsupervisedSparkModelBuilder
>-
>
>ModelApiV10
>
>
>
> *Conclusion*
>
> Currently we have decided to go with "Method 2" because of the following
> reasons.
>
>- Not all models have PMML support in Spark.
>- If we are to use anything apart from Spark MLlib, such as H2O, we
>will be depending on PMML support from H2O.
>- With Method 1 we might be generating PMML models when users are not
>in need of it (useless computation).
>
> Please let me know if there is a bett
Re: [Architecture] [ML] PMML support for Machine Learner
On Mon, Oct 12, 2015 at 12:36 PM, Vidura Gamini Abhaya wrote: > Hi Fazlan, > > Please see my comments inline in blue. > >> >> No I am not planning to build the model from scratch. Once the serialized >> spark model is built, we can export it to PMML format. In other words, we >> are using the serialized model in order to build the PMML model. >> > > That's great. > > If I have not mistaken what you are suggesting is let the user go through >> the normal workflow of model building and once it is done, give an option >> to the user to export it to PMML format(also for the models that have been >> already built)? >> > Yes exactly! What we should not do IMO is asking the user to go through the whole workflow if he needs to export already created model in PMML. > > Yes, this is exactly what I meant. > > >> @Vidura I will check on the run-time support, if that is possible that >> would be great. >> > > If it's supported, it'll be great. If not we can still do it based on the > model type but I think it'll be a bit messy as the code wouldn't be as > generic. > > > Thanks and Regards, > > Vidura > > > >> >> On Mon, Oct 12, 2015 at 12:10 PM, Vidura Gamini Abhaya >> wrote: >> >>> Hi Fazlan, >>> >>> Are you planning to build a PMML model from the scratch (i.e going >>> through the entire flow to build an ML model) or is this to be used for >>> exporting a PMML out of an already built model? >>> >>> If it's the former, +1 to what CD mentioned on not asking user to go >>> through the entire ML workflow for PMML. My preference is also for >>> saving/exporting a model in PMML to be an option for the user, once a model >>> is built and for models that have already been built. >>> >>> @Fazlan - Can we find out whether the PMML export is possible at runtime >>> through a method or through the inheritance hierarchy? If so, we could only >>> make the export option visible on a UI, only for supported models. >>> >>> Thanks and Regards, >>> >>> Vidura >>> >>> >>> >>> On 12 October 2015 at 11:33, CD Athuraliya wrote: >>> >>>> Hi, >>>> >>>> I feel that asking user to go through the complete ML workflow for PMML >>>> is too demanding. Computationally this conversion should be less expensive >>>> compared to model training in real world use cases (since it's a mapping of >>>> model parameters from Java objects to XML AFAIK). And model training should >>>> be independent from the model format. Instead can't we support this >>>> conversion on demand? Or save in both formats for now? Once Spark starts >>>> supporting PMML for all algorithms we can go for Method 1 if it looks >>>> consistent through out our ML life cycle. >>>> >>>> Thanks >>>> >>>> On Mon, Oct 12, 2015 at 11:09 AM, Fazlan Nazeem >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I am working on redmine[1] regarding PMML support for Machine Learner. >>>>> Please provide your opinion on this design. >>>>> [1]https://redmine.wso2.com/issues/4303 >>>>> >>>>> *Overview* >>>>> >>>>> Spark 1.5.1(lastest version) supports PMML model export for some of >>>>> the available models in Spark through MLlib. >>>>> >>>>> The table below outlines the MLlib models that can be exported to PMML >>>>> and their equivalent PMML model. >>>>> >>>>> >>>>> >>>>> MLlib model >>>>> >>>>> PMML model >>>>> >>>>> KMeansModel >>>>> >>>>> ClusteringModel >>>>> >>>>> LinearRegressionModel >>>>> >>>>> RegressionModel (functionName="regression") >>>>> >>>>> RidgeRegressionModel >>>>> >>>>> RegressionModel (functionName="regression") >>>>> >>>>> LassoModel >>>>> >>>>> RegressionModel (functionName="regression") >>>>> >>>>> SVMModel >>>>> >>>>> RegressionModel (functionName="classification" >>>>> normalizationMethod="none") >>>>> >>>>> Bina
Re: [Architecture] [ML] PMML support for Machine Learner
To me buildPMMLModel(long modelId) sounds more like we are building (or training) the model. ExportPMML or something similar would sound more like the actual action IMO. On Mon, Oct 12, 2015 at 2:02 PM, Nirmal Fernando wrote: > Hi CD/Vidura, > > On Mon, Oct 12, 2015 at 1:56 PM, CD Athuraliya > wrote: > >> >> >> On Mon, Oct 12, 2015 at 12:36 PM, Vidura Gamini Abhaya >> wrote: >> >>> Hi Fazlan, >>> >>> Please see my comments inline in blue. >>> >>>> >>>> No I am not planning to build the model from scratch. Once the >>>> serialized spark model is built, we can export it to PMML format. In other >>>> words, we are using the serialized model in order to build the PMML model. >>>> >>> >>> That's great. >>> >>> If I have not mistaken what you are suggesting is let the user go >>>> through the normal workflow of model building and once it is done, give an >>>> option to the user to export it to PMML format(also for the models that >>>> have been already built)? >>>> >>> >> Yes exactly! What we should not do IMO is asking the user to go through >> the whole workflow if he needs to export already created model in PMML. >> > > Can you please explain from where did you get this idea? If this idea is > there in Fazlan's content, we need to fix it. > > >>> Yes, this is exactly what I meant. >>> >>> >>>> @Vidura I will check on the run-time support, if that is possible that >>>> would be great. >>>> >>> >>> If it's supported, it'll be great. If not we can still do it based on >>> the model type but I think it'll be a bit messy as the code wouldn't be as >>> generic. >>> >>> >>> Thanks and Regards, >>> >>> Vidura >>> >>> >>> >>>> >>>> On Mon, Oct 12, 2015 at 12:10 PM, Vidura Gamini Abhaya >>> > wrote: >>>> >>>>> Hi Fazlan, >>>>> >>>>> Are you planning to build a PMML model from the scratch (i.e going >>>>> through the entire flow to build an ML model) or is this to be used for >>>>> exporting a PMML out of an already built model? >>>>> >>>>> If it's the former, +1 to what CD mentioned on not asking user to go >>>>> through the entire ML workflow for PMML. My preference is also for >>>>> saving/exporting a model in PMML to be an option for the user, once a >>>>> model >>>>> is built and for models that have already been built. >>>>> >>>>> @Fazlan - Can we find out whether the PMML export is possible at >>>>> runtime through a method or through the inheritance hierarchy? If so, we >>>>> could only make the export option visible on a UI, only for supported >>>>> models. >>>>> >>>>> Thanks and Regards, >>>>> >>>>> Vidura >>>>> >>>>> >>>>> >>>>> On 12 October 2015 at 11:33, CD Athuraliya >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I feel that asking user to go through the complete ML workflow for >>>>>> PMML is too demanding. Computationally this conversion should be less >>>>>> expensive compared to model training in real world use cases (since it's >>>>>> a >>>>>> mapping of model parameters from Java objects to XML AFAIK). And model >>>>>> training should be independent from the model format. Instead can't we >>>>>> support this conversion on demand? Or save in both formats for now? Once >>>>>> Spark starts supporting PMML for all algorithms we can go for Method 1 if >>>>>> it looks consistent through out our ML life cycle. >>>>>> >>>>>> Thanks >>>>>> >>>>>> On Mon, Oct 12, 2015 at 11:09 AM, Fazlan Nazeem >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am working on redmine[1] regarding PMML support for Machine >>>>>>> Learner. Please provide your opinion on this design. >>>>>>> [1]https://redmine.wso2.com/issues/4303 >>>>>>> >>>>>>> *Overview* >>>&
Re: [Architecture] [ML] PMML support for Machine Learner
On Mon, Oct 12, 2015 at 2:17 PM, Fazlan Nazeem wrote: > Sure Nirmal. > > Thanks CD for pointing it out! > > On Mon, Oct 12, 2015 at 2:14 PM, Nirmal Fernando wrote: > >> Excellent! Good catch! Fazlan please fix. exportAsPMMLModel may be? >> >> On Mon, Oct 12, 2015 at 2:10 PM, CD Athuraliya >> wrote: >> >>> To me buildPMMLModel(long modelId) sounds more like we are building (or >>> training) the model. ExportPMML or something similar would sound more >>> like the actual action IMO. >>> >>> On Mon, Oct 12, 2015 at 2:02 PM, Nirmal Fernando >>> wrote: >>> >>>> Hi CD/Vidura, >>>> >>>> On Mon, Oct 12, 2015 at 1:56 PM, CD Athuraliya >>>> wrote: >>>> >>>>> >>>>> >>>>> On Mon, Oct 12, 2015 at 12:36 PM, Vidura Gamini Abhaya < >>>>> vid...@wso2.com> wrote: >>>>> >>>>>> Hi Fazlan, >>>>>> >>>>>> Please see my comments inline in blue. >>>>>> >>>>>>> >>>>>>> No I am not planning to build the model from scratch. Once the >>>>>>> serialized spark model is built, we can export it to PMML format. In >>>>>>> other >>>>>>> words, we are using the serialized model in order to build the PMML >>>>>>> model. >>>>>>> >>>>>> >>>>>> That's great. >>>>>> >>>>>> If I have not mistaken what you are suggesting is let the user go >>>>>>> through the normal workflow of model building and once it is done, give >>>>>>> an >>>>>>> option to the user to export it to PMML format(also for the models that >>>>>>> have been already built)? >>>>>>> >>>>>> >>>>> Yes exactly! What we should not do IMO is asking the user to go >>>>> through the whole workflow if he needs to export already created model in >>>>> PMML. >>>>> >>>> >>>> Can you please explain from where did you get this idea? If this idea >>>> is there in Fazlan's content, we need to fix it. >>>> >>>> >>>>>> Yes, this is exactly what I meant. >>>>>> >>>>>> >>>>>>> @Vidura I will check on the run-time support, if that is possible >>>>>>> that would be great. >>>>>>> >>>>>> >>>>>> If it's supported, it'll be great. If not we can still do it based on >>>>>> the model type but I think it'll be a bit messy as the code wouldn't be >>>>>> as >>>>>> generic. >>>>>> >>>>>> >>>>>> Thanks and Regards, >>>>>> >>>>>> Vidura >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> On Mon, Oct 12, 2015 at 12:10 PM, Vidura Gamini Abhaya < >>>>>>> vid...@wso2.com> wrote: >>>>>>> >>>>>>>> Hi Fazlan, >>>>>>>> >>>>>>>> Are you planning to build a PMML model from the scratch (i.e going >>>>>>>> through the entire flow to build an ML model) or is this to be used for >>>>>>>> exporting a PMML out of an already built model? >>>>>>>> >>>>>>>> If it's the former, +1 to what CD mentioned on not asking user to >>>>>>>> go through the entire ML workflow for PMML. My preference is also for >>>>>>>> saving/exporting a model in PMML to be an option for the user, once a >>>>>>>> model >>>>>>>> is built and for models that have already been built. >>>>>>>> >>>>>>>> @Fazlan - Can we find out whether the PMML export is possible at >>>>>>>> runtime through a method or through the inheritance hierarchy? If so, >>>>>>>> we >>>>>>>> could only make the export option visible on a UI, only for supported >>>>>>>> models. >>>>>>>> >>>>>>> This option can be something similar to platform selection in typical software d
Re: [Architecture] VizGrammar chart improvements
when it comes to drawing, plotting and updating. If new >>>>>>>>> chart type is implemented, it needs to follow same structure and code >>>>>>>>> will >>>>>>>>> be in a separate JS file under charts directory. Meanwhile samples for >>>>>>>>> charts can be included the samples directory. >>>>>>>>> >>>>>>>>> |- lib >>>>>>>>> | |- vega.min.js >>>>>>>>> | |- d3.min.js >>>>>>>>> | >>>>>>>>> |- samples >>>>>>>>> | |- bar >>>>>>>>> | |- line >>>>>>>>> | . >>>>>>>>> | . >>>>>>>>> | |- scatter >>>>>>>>> | >>>>>>>>> |- charts >>>>>>>>> |-|- chart.js >>>>>>>>> |-|- chart.bar.js >>>>>>>>> |-|- chart.line.js >>>>>>>>> |-|- chart.area.js >>>>>>>>> |-|- chart.map.js >>>>>>>>> |-|- chart.table.js >>>>>>>>> |-|- chart.arc.js >>>>>>>>> |-|- chart.scatter.js >>>>>>>>> | >>>>>>>>> |- vizgrammar.js >>>>>>>>> |- vizgrammar.min.js >>>>>>>>> >>>>>>>>> >>>>>>>>> Finally chart JS files will be combined to create vizgrammar.js >>>>>>>>> and vizgrammar.min.js. >>>>>>>>> >>>>>>>>> [1] https://github.com/wso2/VizGrammar >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> -- >>>>>>>>> >>>>>>>>> *Tharik Kanaka* >>>>>>>>> >>>>>>>>> WSO2, Inc |#20, Palm Grove, Colombo 03, Sri Lanka >>>>>>>>> >>>>>>>>> Email: tha...@wso2.com | Web: www.wso2.com >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> >>>>>>>> Dunith Dhanushka, >>>>>>>> Senior Software Engineer >>>>>>>> WSO2 Inc, >>>>>>>> >>>>>>>> Mobile - +94 71 8615744 >>>>>>>> Blog - dunithd.wordpress.com <http://blog.dunith.com> >>>>>>>> Twitter - @dunithd <http://twitter.com/dunithd> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> *S. Suhothayan* >>>>>>> Technical Lead & Team Lead of WSO2 Complex Event Processor >>>>>>> *WSO2 Inc. *http://wso2.com >>>>>>> * <http://wso2.com/>* >>>>>>> lean . enterprise . middleware >>>>>>> >>>>>>> >>>>>>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: >>>>>>> http://suhothayan.blogspot.com/ >>>>>>> <http://suhothayan.blogspot.com/>twitter: >>>>>>> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | >>>>>>> linked-in: >>>>>>> http://lk.linkedin.com/in/suhothayan >>>>>>> <http://lk.linkedin.com/in/suhothayan>* >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> >>>>>> Dunith Dhanushka, >>>>>> Senior Software Engineer >>>>>> WSO2 Inc, >>>>>> >>>>>> Mobile - +94 71 8615744 >>>>>> Blog - dunithd.wordpress.com <http://blog.dunith.com> >>>>>> Twitter - @dunithd <http://twitter.com/dunithd> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> *S. Suhothayan* >>>>> Technical Lead & Team Lead of WSO2 Complex Event Processor >>>>> *WSO2 Inc. *http://wso2.com >>>>> * <http://wso2.com/>* >>>>> lean . enterprise . middleware >>>>> >>>>> >>>>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: >>>>> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/>twitter: >>>>> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | linked-in: >>>>> http://lk.linkedin.com/in/suhothayan >>>>> <http://lk.linkedin.com/in/suhothayan>* >>>>> >>>>> ___ >>>>> Architecture mailing list >>>>> Architecture@wso2.org >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Isuru Perera >>>> Associate Technical Lead | WSO2, Inc. | http://wso2.com/ >>>> Lean . Enterprise . Middleware >>>> >>>> about.me/chrishantha >>>> Contact: +IsuruPereraWSO2 >>>> <https://www.google.com/+IsuruPereraWSO2/about> >>>> >>>> ___ >>>> Architecture mailing list >>>> Architecture@wso2.org >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> >>> Srinath Perera, Ph.D. >>>http://people.apache.org/~hemapani/ >>>http://srinathsview.blogspot.com/ >>> >>> ___ >>> Architecture mailing list >>> Architecture@wso2.org >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> >> *Tharik Kanaka* >> >> WSO2, Inc |#20, Palm Grove, Colombo 03, Sri Lanka >> >> Email: tha...@wso2.com | Web: www.wso2.com >> >> ___ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > > *S. Suhothayan* > Technical Lead & Team Lead of WSO2 Complex Event Processor > *WSO2 Inc. *http://wso2.com > * <http://wso2.com/>* > lean . enterprise . middleware > > > *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: > http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/>twitter: > http://twitter.com/suhothayan <http://twitter.com/suhothayan> | linked-in: > http://lk.linkedin.com/in/suhothayan <http://lk.linkedin.com/in/suhothayan>* > > ___ > Architecture mailing list > Architecture@wso2.org > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *CD Athuraliya* Software Engineer WSO2, Inc. lean . enterprise . middleware Mobile: +94 716288847 <94716288847> LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter <https://twitter.com/cdathuraliya> | Blog <https://cdathuraliya.wordpress.com/> ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
