Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

>>>> This is the full query i used. >>>>>>>> >>>>>>>> @Import('LinRegInput:1.0.0') >>>>>>>> >>>>>>>> define stream LinRegInput (salary double, rbi double, walks double, >>>>>>

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

double, >>>>>>> strikeouts double, errors double); >>>>>>> >>>>>>> @Export('LinRegOutput:1.0.0') >>>>>>> >>>>>>> define stream LinregOutput (mse double); >>>>>>> >>>>>>> from LinRegInput#ml:streamlinr

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

ry, rbi, walks, strikeouts, errors) >>>>>> >>>>>> select * >>>>>> insert into mse; >>>>>> >>>>>> but i am sending [mse,intercept,beta1....betap] as a outputData >>>>>> Object[]. SO how can

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

t;>> regards, >>>>> Mahesh. >>>>> >>>>> On Tue, Jun 21, 2016 at 6:10 PM, Nirmal Fernando <nir...@wso2.com> >>>>> wrote: >>>>> >>>>>> Hi Mahesh, >>>>>> >>>>>&g

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

2, 2016 at 10:24 AM, Maheshakya Wijewardena < >>>>>> mahesha...@wso2.com> wrote: >>>>>> >>>>>>> Hi Mahesh, >>>>>>> >>>>>>> In your output stream, you need to list all the attrib

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

>>>>> >>>>> Can you summarize the work we have done so far and the remaining work >>>>> items please? >>>>> >>>>> Thanks. >>>>> >>>>> On Tue, Jun 21, 2016 at 5:56 PM, Mahesh Dananjaya

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

t;>>>> I have updated the repo [2] and upto date documents can be found at >>>>> [1].thank you. >>>>> regards, >>>>> Mahesh. >>>>> [1] >>>>> https://github.com/dananjayamahesh/GSOC2016/tree/master/gsoc/siddhi/extension/stream

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

pdated the repo [2] and upto date documents can be found at >>>> [1].thank you. >>>> regards, >>>> Mahesh. >>>> [1] >>>> https://github.com/dananjayamahesh/GSOC2016/tree/master/gsoc/siddhi/extension/streaming >>>> [2] ht

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

>> https://github.com/dananjayamahesh/GSOC2016/tree/master/gsoc/siddhi/extension/streaming >>> [2] https://github.com/dananjayamahesh/carbon-ml/tree/wso2_gsoc_ml6_cml >>> >>> >>> On Tue, Jun 21, 2016 at 5:08 PM, Mahesh Dananjaya < >>> dananjayam

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

arbon-ml/tree/wso2_gsoc_ml6_cml >> >> >> On Tue, Jun 21, 2016 at 5:08 PM, Mahesh Dananjaya < >> dananjayamah...@gmail.com> wrote: >> >>> >>> -- Forwarded message ---------- >>> From: Mahesh Dananjaya <dananjayamah...@gmail.c

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

ya < > dananjayamah...@gmail.com> wrote: > >> >> -- Forwarded message -- >> From: Mahesh Dananjaya <dananjayamah...@gmail.com> >> Date: Tue, Jun 21, 2016 at 5:08 PM >> Subject: Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with >> online

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

, 2016 at 5:08 PM, Mahesh Dananjaya <dananjayamah...@gmail.com > wrote: > > -- Forwarded message -- > From: Mahesh Dananjaya <dananjayamah...@gmail.com> > Date: Tue, Jun 21, 2016 at 5:08 PM > Subject: Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Mahesh, If you are installing features from new p2 repo into a new CEP pack, then you wont need to replace those jars. If you have already installed those in the CEP from a previous p2-repo, then you have to un-install those features and reinstall with new p2 repo. But you don't need to do

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshakya, If i built the carbon-ml then product-ml and point new p2 repository to cep features, do i need to copy that org.wso2.carbon.ml.siddhi.extension1.1. thing into cep_home/repository/component/... place. regards, Mahesh. On Thu, Jun 16, 2016 at 6:39 PM, Mahesh Dananjaya

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshakya, I pushed the StreamingLinearRegression modules into my forked carbon-ml repo at branch wso2_gsoc_ml6_cml [1]. I am working on persisting model.thank you. Mahesh. [1] https://github.com/dananjayamahesh/carbon-ml On Tue, Jun 14, 2016 at 5:56 PM, Mahesh Dananjaya

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshakya, the above error is due to a simple mistake of not providing my local p2 repo.Now it is working and i debugged the StreamingLinearRegression model cep. regards, Mahesh. On Tue, Jun 14, 2016 at 3:19 PM, Mahesh Dananjaya wrote: > Hi Maheshakya, > I did

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshakya, I did what you recommend. But when i am adding the query the following error is appearing. No extension exist for StreamFunctionExtension{namespace='ml'} in execution plan "NewExecutionPlan" *My query is as follows, @Import('LinRegInput:1.0.0') define stream LinRegInput (salary

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Mahesh, You don't need to add new p2 repo. In the /repository/components/plugins folder, you will find org.wso2.carbon.ml.siddhi.extension_some_version.jar. Replace this with

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshakya, Do i need to add p2 local repos of ML into CEP after i made changes to ml extensions. Or will it be automatically updated. I am trying to debug my extension with the cep.thank you. regards, Mahesh. On Tue, Jun 14, 2016 at 1:57 PM, Maheshakya Wijewardena

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Mahesh, You can add a new folder for streaming algorithms in the siddhi extension. There, keep stream processors and the algorithms classes separately. We can arrange a hangout tomorrow. Best regards. On Tue, Jun 14, 2016 at 12:22 PM, Mahesh Dananjaya < dananjayamah...@gmail.com> wrote: >

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshakya, May i seperately put the classes to ml and extensions in carbon-core. I can put Streaming Extensions to extensions and Algorithms/StreamingLinear Regression and StreamingKMeans in ml core. what is the suitable format. I will commit my changes today as seperate branch in my forked

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi maheshakya, ok.these couple of days i have spent on implementing streamin clustering in a efficient way.i have found couple of methods.intially i am developing k batch k means for streaming.i will let you know the progress within next couple of days.i have already added paramter in query for

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Mahesh, Regarding your question: my outputData Object[]array is in the format of > [mse,beta0,beta1,betap].But seems to be that cep does not understand it. Did you create an output stream first for the publisher? You need to create a stream with attributes: mse double, beta1 double,

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshakya, in the last one mentioned example query for streaming linear regression should be, *insert into regResults; from LinRegInputStream#streaming:streaminglr(0, 2, 100, 0.0001, 1.0, 0.95, salary, rbi, walks, strikeouts, errors)select *insert into regResults*; miniBatchFraction

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshakya, you can find the details of the queries in this ReadMe [1]. i have add some changes . so previous querirs may not valid.please use these new queries in the README. *1.Streaming Linear regression* from LinRegInputStream#streaming:streaminglr((learnType), (batchSize/timeFrame),

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Mahesh, Great work so far. Regarding the queries: streamingkm(0, 2,2,20,1,0.95 salary, rbi, walks, strikeouts, errors) Can you give me the definitions of the first few entities in the order. Also in previous supervised cases (linear regression), what is the response variable, etc. I'll go

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshkya, I have changed the siddhi query for our StreamingKMeansClustering by adding Alpha into the picture which we can use to make data horizon (how quickly a most recent data point becomes a part of the model) and data obsolescence (how long does it take a past data point to become

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshakya, As we have discussed the architecture of the project i have already developed a couple of essential components for our project. During last week i completed the writing cep siddhi extension for our streaming algorithms which are developed to learn incrementally with past

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshkya, If you want to run it please use following queries. 1. StreamingLInearRegression from Stream4InputStream#streaming:streaminglr(0, 2, 0.95, salary, rbi, walks, strikeouts, errors) select * insert into regResults; from Stream8Input#streaming:streamingkm(0, 2, 0.95,2,20, salary,

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshkaya, I have added the moving window method and update the previos StreamingLinearRegression [1] which only performed batch processing with streaming data. and also i added the StreamingKMeansClustering [1] for our purposes and debugged them.thank you. regards, Mahesh. [1]

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Thanks Mahesh! The graphs look promising! :) So by looking at graph, LR with SGD can train a model within 60 secs (6*10^10 nano sec), using about 900,000 data points . Means, this online training can handle events/data points coming at rate of 15,000 per second (or more) , if the batch size is

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshakya, I have looked into the spark streaming fundamentals and k mean clustering to develop the streaming k mean clustering for stream data. those can be found at [1] and [2].I will commit new changes to my repo today including the basic implementation of streaming k mean clustering.thank

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshakya, As you requested i can change other parameters as well such as feature size(p). Initially i did it with p=3;sure thing. Anyway you can see and run the code if you want. source is at [1]. the test timing is called with random data as you requested if you set args[0] to 1. And you can

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi supun, Though i pushed it yesterday, there was some problems with the network. now you can see them in the repo location [1].I added some Matlab plot you can see the patter there.you can use ml also. Ok sure thing. I can prepare a report or else blog if you want. files are as follows. The y

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Mahesh, I have added those timing reports to my repo [1]. Whats the file name? :) Btw, can you compile simple doc (gdoc) with the above results, and bring everything to one place? That way it is easy to compare, and keep track. Thanks, Supun On Fri, Jun 3, 2016 at 7:23 PM, Mahesh Dananjaya

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshkya, I have added those timing reports to my repo [1].please have a look at. three files are there. one is using incremet as 1000 for batch sizes (lr_timing_1000). Otherone is using incremet by 1 (lr_timing_1) upto 1 million in both scenarios.you can see the reports and figures in

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Mahesh, Thank you for the update. I will look into your implementation. And i will be able to send you the timing/performances analysis report > tomorrow for the SGD functions > Great. Sent those asap so that we can proceed. Best regards. On Sun, May 29, 2016 at 6:56 PM, Mahesh Dananjaya

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi maheshkaya, i have written some siddhi extension and trying to develop a one for my one. In time series example in the [1], can you please explain me the input format and query lines in that example for my understanding. from baseballData#timeseries:regress(2, 1, 0.95, salary, rbi, walks,

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshakya, today i got the siddhi and debug the math extention. then did some changes and check. Now i am trying to write same kind of extension in my code base. so i add dependencies and it was built fine. Now i am trying to debug my extension and i did the same thing as i did in previous

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Also note that there is a calculation interval in the siddhi time series regression function[1]. You maybe able get some insight for this from that as well. [1] https://docs.wso2.com/display/CEP400/Regression On Tue, May 24, 2016 at 11:03 AM, Maheshakya Wijewardena < mahesha...@wso2.com> wrote:

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Mahesh, As we discussed offline, we can use similar mechanism to train linear regression models, logistic regression models and k-means clustering models. It is very interesting that i have found that somethings that can make use > of our work. In the cep 4.0 documentation there is a Custom

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshkya, shall we use [1] for our work? i am checking the possibility. BR, Mahesh. [1] https://docs.wso2.com/display/CEP400/Writing+a+Custom+Stream+Processor+Extension [2] https://docs.wso2.com/display/CEP400/Inbuilt+Windows#InbuiltWindows-lengthlength

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshakya, It is very interesting that i have found that somethings that can make use of our work. In the cep 4.0 documentation there is a Custom Stream Processor Extention program [1]. There is a example of LinearRegressionStreamProcessor [1] and also i saw private int batchSize =

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi maheshakya, anyway how can test any siddhi extention after write it without integrating it to cep.can you please explain me the procedure. i am referring to [1] [2] [3] [4]. thank you. BR, Mahesh. [1] https://docs.wso2.com/display/CEP310/Writing+Extensions+to+Siddhi [2]

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshakya, thank you for the feedback. I have add data-sets into repo. data-sets/lr. I am all right with next week.Now i am writing some examples to collect samples and build mini batches and run the algorithms on those mini-batches. thank you. will add those into repo soon.I am still working

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Mahesh, I've look into your code sample of streaming linear regression. Looks good to me, apart from few issues in coding practices which we can improve when you're doing the implementations in carbon-ml and during the code reviews. You are using a set of files as mini-batches of data, right?

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshakya, Ok. I will check it.you have sent me those relevant references and i am working on that thing.thank you. does that cep problem is now all right that we were trying to fix. I am still using those pre-build versions. If so i can merge with the latest one.thanks. BR, Mahesh. On Wed,

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Mahesh, You don't actually have to implement anything in spark streaming. Try to understand how streaming data is handled in and the specifics of the underlying algorithms in streaming. What we want to do is having the similar algorithms that support CEP event streams with siddhi. Best

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Mahesh, I'll review your code sample and give you our feedback asap. In the meantime, please go through the documentation for writing siddhi extensions and get some idea. It's better if you can try writing some simple siddhi extensions your self and test them to get a good understanding. Best

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshakya, Did you check the repo. I will add recent works today.And also i was going through the Java docs related to spark streaming work. It is with that scala API. thank you. regards, Mahesh. On Tue, May 17, 2016 at 10:11 AM, Mahesh Dananjaya < dananjayamah...@gmail.com> wrote: > Hi

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshakya, I have gone through the Java Docs and run some of the Spark examples on spark shell which are paramount improtant for our work. Then i have been writing my codes to check the Linear regression, K means for streaming. please check my git repo [1]. I think now i have to ask on dev

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Maheshakya, thank you. i will update the repo today.thank you.i changed the carbon ml siddhi extention and see how the changes are effecting. i will update the progress as soon as possible.thank you. i had some problem in spark mllib dependency. i was fixing that. regards, Mahesh. p.s: do i

Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi Mahesh, Sorry for replying late. Thank you for the update. I believe you have done some implementations with with Spark MLLIb algorithms in streaming fashion as we have discussed. If so, can you please share your code in a Github repo. Now i want to implements some machine learning

[Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

-- Forwarded message -- From: Mahesh Dananjaya Date: Wed, May 11, 2016 at 1:43 PM Subject: Re: [Dev] GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner To: Maheshakya Wijewardena Hi Maheshakya, sorry

[Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner

Hi maheshakya, I have installed them correctly.now I am trying to debug the siddhi extention with the cep as the [1] describes. But when i created an input stream and a predictionStream (output stream). when i was trying to create new execution plan with above streams i got error when i clicked