Hi Maheshakya, great.thank you.i already have ML and CEP and working more towards it. does that Scala API is with your current product or repo?. thank you. BR, Mahesh.
On Sun, Mar 6, 2016 at 5:49 PM, Maheshakya Wijewardena <mahesha...@wso2.com> wrote: > Hi Mahesh, > > Please find the comments inline. > > does data stream is taken to ML as the event publisher's format through >> event publisher. Or we can use direct traffic that comes to event >> receiver, or else as streams >> > We intend to use the direct data as even streams. > > 1.) Those data coming from wso2 DAS to ML are coming as streams? >> > No, WSO2 ML doesn't use any even stream. The data stored in tables in DAS > is loaded into ML. > > 2.) Are there any incremental learning algorithms currently active in >> ML?you mentioned that there are and they are with scala API. So there is a >> streaming support with that Scala API. In that API which format the data is >> aquired to ML? >> > No, there are no incremental learning algorithms in ML. The scala API is > about Spark MLLib. MLLib supports streaming k-means and other generalized > linear models (linear regression variants and logistic regression) with > Scala API. What they basically do in those implementations is retraining > the trained models with mini batches when data sequentially arrives. There, > the breaking of streaming data into mini batches is done with the help of > Spark Streaming. But we do not intend to use Spark streaming in our > implementation. What we need to do is implement a similar behavior for > event streams using the Java API. The Java API has the following methods: > > - *createModel > > <http://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/regression/LinearRegressionWithSGD.html#createModel%28org.apache.spark.mllib.linalg.Vector,%20double%29>* > (Vector > > <http://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/linalg/Vector.html> > weights, > double intercept) - for GLMs > - *setInitialModel > > <http://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/clustering/KMeans.html#setInitialModel%28org.apache.spark.mllib.clustering.KMeansModel%29>* > (KMeansModel > > <http://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/clustering/KMeansModel.html> > model) > - for K means > > With the help of these methods, we can train models again with newly > arriving data, keeping the characteristics learned with the previous data. > When implementing this, we need to pay attention to other parameters of > incremental learning such as data horizon and data obsolescence (indicated > in the project ideas page). > We need to discuss on how to add these with CEP event streams. I have > added Suho into the thread for more clarification. > > Best regards. > > > On Sat, Mar 5, 2016 at 5:15 PM, Mahesh Dananjaya < > dananjayamah...@gmail.com> wrote: > >> Hi maheshakya, >> as we concerned to use WSO2 CEP to handle streaming data and implement >> the machine learning algorithms with Spark MLLib, does data stream is taken >> to ML as the event publisher's format through event publisher. Or we can >> use direct traffic that comes to event receiver, or else as streams. >> referring to https://docs.wso2.com/display/CEP410/User+Guide >> 1.) Those data coming from wso2 DAS to ML are coming as streams? >> 2.) Are there any incremental learning algorithms currently active in >> ML?you mentioned that there are and they are with scala API. So there is a >> streaming support with that Scala API. In that API which format the data is >> aquired to ML? >> >> thank you. >> BR, >> Mahesh. >> >> On Fri, Mar 4, 2016 at 2:03 PM, Maheshakya Wijewardena < >> mahesha...@wso2.com> wrote: >> >>> Hi Mahesh, >>> >>> We had to modify a the project scope a little to suit best for the >>> requirements. We will update the project idea with those concerns soon and >>> let you know. >>> >>> We do not support streaming data in WSO2 Machine learner at the moment. >>> The new concern is to use WSO2 CEP to handle streaming data and implement >>> the machine learning algorithms with Spark MLLib. You can look at the >>> streaming k-means and streaming linear regression implementations in MLLib. >>> Currently, the API is only for scala. Our need is to get the Java APIs of >>> k-means and generalized linear models to support incremental learning with >>> streaming data. This has to be done as mini-batch learning since these >>> algorithms operates as stochastic gradient descents so that any learning >>> with new data can be done on top of the previously learned models. So >>> please go through the those APIs[1][2][3] and try to get an idea. >>> Also please try to understand how event streams work in WSO2 CEP [4][5]. >>> >>> Best regards. >>> >>> [1] >>> http://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/regression/LinearRegressionWithSGD.html >>> [2] >>> http://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/clustering/KMeans.html >>> [3] >>> http://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/classification/LogisticRegressionWithSGD.html >>> [4] https://docs.wso2.com/display/CEP310/Working+with+Event+Streams >>> [5] https://docs.wso2.com/display/CEP310/Working+with+Execution+Plans >>> >>> On Fri, Mar 4, 2016 at 11:26 AM, Mahesh Dananjaya < >>> dananjayamah...@gmail.com> wrote: >>> >>>> Hi maheshakya, >>>> give me sometime to go through your ML package. Do current product have >>>> any stream data support?. i did some university projects related to machine >>>> learning with regressions,modelling, factor analysis, cluster analysis and >>>> classification problems (Discriminant Analysis) with SVM (Support Vector >>>> machines), Neural networks, LS classification and ML(Maximum likelihood). >>>> give me sometime to see how wso2 architecture works.then i can come up with >>>> good architecture.thank you. >>>> BR, >>>> Mahesh. >>>> >>>> On Wed, Mar 2, 2016 at 2:41 PM, Mahesh Dananjaya < >>>> dananjayamah...@gmail.com> wrote: >>>> >>>>> Hi Maheshakya, >>>>> Thank you for the resources. I will go through this and looking >>>>> forward to this proposed project.Thank you. >>>>> BR, >>>>> Mahesh. >>>>> >>>>> On Wed, Mar 2, 2016 at 1:52 PM, Maheshakya Wijewardena < >>>>> mahesha...@wso2.com> wrote: >>>>> >>>>>> Hi Mahesh, >>>>>> >>>>>> Thank you for the interest for this project. >>>>>> >>>>>> We would like to know what type of similar projects you have worked >>>>>> on. You may have seen that WSO2 Machine Learner supports several learning >>>>>> algorithms at the moment[1]. This project intends to leverage the >>>>>> existing >>>>>> algorithms in WSO2 Machine Learner to support streaming data. As an >>>>>> initiative, first you can get an idea about what WSO2 Machine Learner >>>>>> does >>>>>> and how it operates. You can download WSO2 Machine Learner from product >>>>>> page[2] and the the source code [3]. ML is using Apache Spark MLLib[4] >>>>>> for >>>>>> its' algorithms so it's better to read and understand what it does as >>>>>> well. >>>>>> >>>>>> In order to get an idea about the deliverables and the scope of this >>>>>> project, try to understand how Spark streaming[5] (see examples) handles >>>>>> streaming data. Also, have a look in the streaming algorithms[6][7] >>>>>> supported by MLLib. There are two approaches discussed to employ >>>>>> incremental learning in ML in the project proposals page. These streaming >>>>>> algorithms can be directly used in the first approach. For the other >>>>>> approach, the your implementation should contain a procedure to create >>>>>> mini >>>>>> batches from streaming data with relevant sizes (i.e. a moving window) >>>>>> and >>>>>> do periodic retraining of the same algorithm. >>>>>> >>>>>> To start with the project, you will need to come up with a suitable >>>>>> plan and an architecture first. >>>>>> >>>>>> Please watch the video referenced in the proposal (reference: 5). It >>>>>> will help you getting a better idea about machine learning algorithms >>>>>> with >>>>>> streaming data. >>>>>> >>>>>> Let us know if you need any help with these. >>>>>> >>>>>> Best regards >>>>>> >>>>>> [1] https://docs.wso2.com/display/ML110/Machine+Learner+Algorithms >>>>>> [2] http://wso2.com/products/machine-learner/ >>>>>> [3] >>>>>> https://docs.wso2.com/display/ML110/Building+from+Source#BuildingfromSource-Downloadingthesourcecheckout >>>>>> [4] https://spark.apache.org/docs/1.4.1/mllib-guide.html >>>>>> [5] >>>>>> https://spark.apache.org/docs/1.4.1/streaming-programming-guide.html >>>>>> [6] >>>>>> https://spark.apache.org/docs/1.4.1/mllib-linear-methods.html#streaming-linear-regression >>>>>> [7] >>>>>> https://spark.apache.org/docs/1.4.1/mllib-clustering.html#streaming-k-means >>>>>> >>>>>> On Wed, Mar 2, 2016 at 1:19 PM, Mahesh Dananjaya < >>>>>> dananjayamah...@gmail.com> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> I am interesting on contribute to proposal 6: "Predictive analytic >>>>>>> with online data for WSO2 Machine Learner" for GSOC2 this time. Since i >>>>>>> have been engaging with some similar projects i think it will be a great >>>>>>> experience for me. Please let me know what you think and what you >>>>>>> suggest. >>>>>>> I have been going through your documents.thank you. >>>>>>> regards, >>>>>>> Mahesh Dananjaya. >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Dev mailing list >>>>>>> Dev@wso2.org >>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Pruthuvi Maheshakya Wijewardena >>>>>> mahesha...@wso2.com >>>>>> +94711228855 >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> >>> -- >>> Pruthuvi Maheshakya Wijewardena >>> mahesha...@wso2.com >>> +94711228855 >>> >>> >>> >> > > > -- > Pruthuvi Maheshakya Wijewardena > mahesha...@wso2.com > +94711228855 > > >
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
