Hi Maheshakya, today i got the siddhi and debug the math extention. then did some changes and check. Now i am trying to write same kind of extension in my code base. so i add dependencies and it was built fine. Now i am trying to debug my extension and i did the same thing as i did in previous case. Cep is sending data, bu my extension is not firing in relevant break point. 1. So how can i debug the siddhi extension in my new extension.(you can see it in my example repoo)
I think if i do it correctly we can built the extension for our purpose. And i will send the relevant timing report of SGD algorithms very soon as supun was asking me. thank you. regards, Mahesh. On Tue, May 24, 2016 at 11:07 AM, Maheshakya Wijewardena < mahesha...@wso2.com> wrote: > Also note that there is a calculation interval in the siddhi time series > regression function[1]. You maybe able get some insight for this from that > as well. > > [1] https://docs.wso2.com/display/CEP400/Regression > > On Tue, May 24, 2016 at 11:03 AM, Maheshakya Wijewardena < > mahesha...@wso2.com> wrote: > >> Hi Mahesh, >> >> As we discussed offline, we can use similar mechanism to train linear >> regression models, logistic regression models and k-means clustering models. >> >> It is very interesting that i have found that somethings that can make >>> use of our work. In the cep 4.0 documentation there is a Custom Stream >>> Processor Extention program [1]. There is a example of >>> LinearRegressionStreamProcessor [1]. >>> >> >> As we have to train predictive models with Spark, you can write wrappers >> around regression/clustering models of Spark. Refer to Siddhi time series >> regression source codes[1][2]. You can write a streaming linear regression >> class for ML in a similar fashion by wrapping Spark mllib implementations. >> You can use the methods "addEvent", "removeEvent", etc. (may have to be >> changed according to requirements) for the similar purpose. You can >> introduce trainLinearRegression/LogisticRegression/Kmeans which does a >> similar thing as in createLinearRegression in those time series functions. >> In the processData method you can use Spark mllib classes to actually train >> models and return the model weights, evaluation metrics. So, converting >> streams into RDDs and retrieving information from the trained models shall >> happen in this method. >> >> In the stream processor extension example, you can retrieve those values >> then use them to train new models with new batches. Weights/cluster centers >> maybe passed as initialization parameters for the wrappers. >> >> Please note that we have to figure out the best siddhi extension type for >> this process. In the siddhi query, we define batch size, type of algorithm >> and number of features (there can be more). After batch size number of >> events received, train a model and save parameters, return evaluation >> metric. With the next batch, retrain the model initialized with previously >> learned parameters. >> >> We also may need to test the same scenario with a moving window, but I >> suspect that that approach may become so slow as a model is trained each >> time an event is received. So, we may have to change the number of slots >> the moving window moves at a time (eg: not one by one, but ten by ten). >> >> Once this is resolved, majority of the research part will be finished and >> all we will be left to do is implementing wrappers around the 3 learning >> algorithms we consider. >> >> Best regards. >> >> [1] >> https://github.com/wso2/siddhi/blob/master/modules/siddhi-extensions/timeseries/src/main/java/org/wso2/siddhi/extension/timeseries/linreg/RegressionCalculator.java >> [2] >> https://github.com/wso2/siddhi/blob/master/modules/siddhi-extensions/timeseries/src/main/java/org/wso2/siddhi/extension/timeseries/linreg/SimpleLinearRegressionCalculator.java >> >> >> On Sat, May 21, 2016 at 2:55 PM, Mahesh Dananjaya < >> dananjayamah...@gmail.com> wrote: >> >>> Hi Maheshkya, >>> shall we use [1] for our work? i am checking the possibility. >>> BR, >>> Mahesh. >>> [1] >>> https://docs.wso2.com/display/CEP400/Writing+a+Custom+Stream+Processor+Extension >>> [2] >>> https://docs.wso2.com/display/CEP400/Inbuilt+Windows#InbuiltWindows-lengthlength >>> [3] >>> https://docs.wso2.com/display/CEP400/Writing+a+Custom+Aggregate+Function >>> >>> On Sat, May 21, 2016 at 2:44 PM, Mahesh Dananjaya < >>> dananjayamah...@gmail.com> wrote: >>> >>>> Hi Maheshakya, >>>> It is very interesting that i have found that somethings that can make >>>> use of our work. In the cep 4.0 documentation there is a Custom Stream >>>> Processor Extention program [1]. There is a example of >>>> LinearRegressionStreamProcessor [1] and also i saw >>>> private int batchSize = 1000000000; i am going through this one. >>>> Please check whether we can use. WIll there be any compatibility or >>>> support issue? >>>> regards, >>>> Mahesh. >>>> >>>> >>>> [1] >>>> https://docs.wso2.com/display/CEP400/Writing+a+Custom+Stream+Processor+Extension >>>> >>>> On Sat, May 21, 2016 at 11:52 AM, Mahesh Dananjaya < >>>> dananjayamah...@gmail.com> wrote: >>>> >>>>> Hi maheshakya, >>>>> anyway how can test any siddhi extention after write it without >>>>> integrating it to cep.can you please explain me the procedure. i am >>>>> referring to [1] [2] [3] [4]. thank you. >>>>> BR, >>>>> Mahesh. >>>>> >>>>> [1] https://docs.wso2.com/display/CEP310/Writing+Extensions+to+Siddhi >>>>> [2] https://docs.wso2.com/display/CEP310/Writing+a+Custom+Function >>>>> [3] https://docs.wso2.com/display/CEP310/Writing+a+Custom+Window >>>>> [4] https://docs.wso2.com/display/CEP400/Writing+Extensions+to+Siddhi >>>>> >>>>> On Thu, May 19, 2016 at 12:08 PM, Mahesh Dananjaya < >>>>> dananjayamah...@gmail.com> wrote: >>>>> >>>>>> Hi Maheshakya, >>>>>> thank you for the feedback. I have add data-sets into repo. >>>>>> data-sets/lr. I am all right with next week.Now i am writing some >>>>>> examples >>>>>> to collect samples and build mini batches and run the algorithms on those >>>>>> mini-batches. thank you. will add those into repo soon.I am still working >>>>>> on that siddhi extention.i will let you know the progress. >>>>>> BR, >>>>>> mahesh. >>>>>> >>>>>> On Thu, May 19, 2016 at 11:10 AM, Maheshakya Wijewardena < >>>>>> mahesha...@wso2.com> wrote: >>>>>> >>>>>>> Hi Mahesh, >>>>>>> >>>>>>> I've look into your code sample of streaming linear regression. >>>>>>> Looks good to me, apart from few issues in coding practices which we can >>>>>>> improve when you're doing the implementations in carbon-ml and during >>>>>>> the >>>>>>> code reviews. You are using a set of files as mini-batches of data, >>>>>>> right? >>>>>>> Can you also send us the datasets you've been using. I'd like to run >>>>>>> this. >>>>>>> >>>>>>> does that cep problem is now all right that we were trying to fix. I >>>>>>>> am still using those pre-build versions. If so i can merge with the >>>>>>>> latest >>>>>>>> one. >>>>>>> >>>>>>> >>>>>>> I'll check this and let you know. >>>>>>> >>>>>>> Can we arrange a meeting (preferably in WSO2 offices) in next week >>>>>>> with ML team members as well. Coding period begins on next Monday, so >>>>>>> it's >>>>>>> better to get overall feedback from others and discuss more about the >>>>>>> project. Let me know convenient time slots for you. I'll arrange a >>>>>>> meeting >>>>>>> with ML team. >>>>>>> >>>>>>> Best regards. >>>>>>> >>>>>>> On Wed, May 18, 2016 at 9:53 AM, Mahesh Dananjaya < >>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Maheshakya, >>>>>>>> Ok. I will check it.you have sent me those relevant references and >>>>>>>> i am working on that thing.thank you. does that cep problem is now all >>>>>>>> right that we were trying to fix. I am still using those pre-build >>>>>>>> versions. If so i can merge with the latest one.thanks. >>>>>>>> BR, >>>>>>>> Mahesh. >>>>>>>> >>>>>>>> On Wed, May 18, 2016 at 9:44 AM, Maheshakya Wijewardena < >>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>> >>>>>>>>> Hi Mahesh, >>>>>>>>> >>>>>>>>> You don't actually have to implement anything in spark streaming. >>>>>>>>> Try to understand how streaming data is handled in and the specifics >>>>>>>>> of the >>>>>>>>> underlying algorithms in streaming. >>>>>>>>> What we want to do is having the similar algorithms that support >>>>>>>>> CEP event streams with siddhi. >>>>>>>>> >>>>>>>>> Best regards. >>>>>>>>> >>>>>>>>> On Wed, May 18, 2016 at 9:38 AM, Mahesh Dananjaya < >>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Maheshakya, >>>>>>>>>> Did you check the repo. I will add recent works today.And also i >>>>>>>>>> was going through the Java docs related to spark streaming work. It >>>>>>>>>> is with >>>>>>>>>> that scala API. thank you. >>>>>>>>>> regards, >>>>>>>>>> Mahesh. >>>>>>>>>> >>>>>>>>>> On Tue, May 17, 2016 at 10:11 AM, Mahesh Dananjaya < >>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>> I have gone through the Java Docs and run some of the Spark >>>>>>>>>>> examples on spark shell which are paramount improtant for our work. >>>>>>>>>>> Then i >>>>>>>>>>> have been writing my codes to check the Linear regression, K means >>>>>>>>>>> for >>>>>>>>>>> streaming. please check my git repo [1]. I think now i have to ask >>>>>>>>>>> on dev >>>>>>>>>>> regarding the capturing event streams for our work. I will update >>>>>>>>>>> the >>>>>>>>>>> recent things on git. check the park-example directory for java. >>>>>>>>>>> examples >>>>>>>>>>> run on git shell is not included there. In my case i think i have >>>>>>>>>>> to build >>>>>>>>>>> mini batches from data streams that comes as individual samples. >>>>>>>>>>> Now i am >>>>>>>>>>> working on some coding to collect mini batches from data >>>>>>>>>>> streams.thank you. >>>>>>>>>>> regards, >>>>>>>>>>> Mahesh. >>>>>>>>>>> [1]https://github.com/dananjayamahesh/GSOC2016 >>>>>>>>>>> >>>>>>>>>>> On Tue, May 17, 2016 at 10:10 AM, Mahesh Dananjaya < >>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>> I have gone through the Java Docs and run some of the Spark >>>>>>>>>>>> examples on spark shell which are paramount improtant for our >>>>>>>>>>>> work. Then i >>>>>>>>>>>> have been writing my codes to check the Linear regression, K means >>>>>>>>>>>> for >>>>>>>>>>>> streaming. please check my git repo [1]. I think now i have to ask >>>>>>>>>>>> on dev >>>>>>>>>>>> regarding the capturing event streams for our work. I will update >>>>>>>>>>>> the >>>>>>>>>>>> recent things on git. check the park-example directory for java. >>>>>>>>>>>> examples >>>>>>>>>>>> run on git shell is not included there. In my case i think i have >>>>>>>>>>>> to build >>>>>>>>>>>> mini batches from data streams that comes as individual samples. >>>>>>>>>>>> Now i am >>>>>>>>>>>> working on some coding to collect mini batches from data >>>>>>>>>>>> streams.thank you. >>>>>>>>>>>> regards, >>>>>>>>>>>> Mahesh. >>>>>>>>>>>> [1]https://github.com/dananjayamahesh/GSOC2016 >>>>>>>>>>>> >>>>>>>>>>>> On Mon, May 16, 2016 at 1:19 PM, Mahesh Dananjaya < >>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>> thank you. i will update the repo today.thank you.i changed >>>>>>>>>>>>> the carbon ml siddhi extention and see how the changes are >>>>>>>>>>>>> effecting. i >>>>>>>>>>>>> will update the progress as soon as possible.thank you. i had >>>>>>>>>>>>> some problem >>>>>>>>>>>>> in spark mllib dependency. i was fixing that. >>>>>>>>>>>>> regards, >>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>> p.s: do i need to maintain a blog? >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, May 16, 2016 at 10:02 AM, Maheshakya Wijewardena < >>>>>>>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sorry for replying late. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you for the update. I believe you have done some >>>>>>>>>>>>>> implementations with with Spark MLLIb algorithms in streaming >>>>>>>>>>>>>> fashion as we >>>>>>>>>>>>>> have discussed. If so, can you please share your code in a >>>>>>>>>>>>>> Github repo. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Now i want to implements some machine learning algorithms >>>>>>>>>>>>>>> with importing mllib and want to run within your code base >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> For the moment you can try out editing the same class >>>>>>>>>>>>>> PredictStreamProcessor in the siddhi extension in carbon-ml. >>>>>>>>>>>>>> Later we will >>>>>>>>>>>>>> add this separately. You should be able to add >>>>>>>>>>>>>> org.apache.spark.mllib. >>>>>>>>>>>>>> classes to there. >>>>>>>>>>>>>> >>>>>>>>>>>>>> And i want to see how event streams are coming from cep. As i >>>>>>>>>>>>>>> think it is not in a RDD format since it is arriving as the >>>>>>>>>>>>>>> individual >>>>>>>>>>>>>>> samples. I will send a email to dev asking about how to get the >>>>>>>>>>>>>>> streams. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Please pay attention to length[1] and lengthbatch[1] inbuilt >>>>>>>>>>>>>> windows in siddhi. What you need to write are functions similar >>>>>>>>>>>>>> to a custom >>>>>>>>>>>>>> aggregate function[2]. >>>>>>>>>>>>>> When you send the email to dev list, explain your >>>>>>>>>>>>>> requirement. You need to get a set of event with from a stream >>>>>>>>>>>>>> with a >>>>>>>>>>>>>> specified window size (number of events). Then build a model >>>>>>>>>>>>>> within that >>>>>>>>>>>>>> function. You also need to retain the data (learned weights, >>>>>>>>>>>>>> cluster >>>>>>>>>>>>>> centers, etc.) from the previous window to use in the current >>>>>>>>>>>>>> window. Ask >>>>>>>>>>>>>> what can be the most suitable option for this among the set of >>>>>>>>>>>>>> siddhi >>>>>>>>>>>>>> extensions given. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] >>>>>>>>>>>>>> https://docs.wso2.com/display/CEP400/Inbuilt+Windows#InbuiltWindows-lengthlength >>>>>>>>>>>>>> [2] >>>>>>>>>>>>>> https://docs.wso2.com/display/CEP400/Writing+a+Custom+Aggregate+Function >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, May 11, 2016 at 1:43 PM, Mahesh Dananjaya < >>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ---------- Forwarded message ---------- >>>>>>>>>>>>>>> From: Mahesh Dananjaya <dananjayamah...@gmail.com> >>>>>>>>>>>>>>> Date: Wed, May 11, 2016 at 1:43 PM >>>>>>>>>>>>>>> Subject: Re: [Dev] GSOC2016: [ML][CEP] Predictive analytic >>>>>>>>>>>>>>> with online data for WSO2 Machine Learner >>>>>>>>>>>>>>> To: Maheshakya Wijewardena <mahesha...@wso2.com> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>>> sorry for not updating. I did what you wanted me to do. I >>>>>>>>>>>>>>> checked the code base and train functions. I went through those >>>>>>>>>>>>>>> java docs. >>>>>>>>>>>>>>> I went through the carbon-ml current implementation of LG and >>>>>>>>>>>>>>> K-Mean. And i >>>>>>>>>>>>>>> had Apache Spark and i tried with several examples. Now i want >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> implements some machine learning algorithms with importing >>>>>>>>>>>>>>> mllib and want >>>>>>>>>>>>>>> to run within your code base. Can you help me with that. >>>>>>>>>>>>>>> And i want to see how event streams are coming from cep. As >>>>>>>>>>>>>>> i think it is not in a RDD format since it is arriving as the >>>>>>>>>>>>>>> individual >>>>>>>>>>>>>>> samples. I will send a email to dev asking about how to get the >>>>>>>>>>>>>>> streams. I >>>>>>>>>>>>>>> debugged many of those functions in the code base. So need >>>>>>>>>>>>>>> further >>>>>>>>>>>>>>> instructions to proceed.thank you. >>>>>>>>>>>>>>> regards, >>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, May 11, 2016 at 10:32 AM, Maheshakya Wijewardena < >>>>>>>>>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Any update on your progress? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, May 4, 2016 at 8:35 PM, Maheshakya Wijewardena < >>>>>>>>>>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> is that "Put break points in train methods in Linear >>>>>>>>>>>>>>>>>> Regression class" means the spark/algorithms/ >>>>>>>>>>>>>>>>>> LinearRegrassion.java class >>>>>>>>>>>>>>>>>> in the org.wso2.carbon.ml.core? is that the correct file? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Yes, this is the correct place. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> You can refer to spark programming guide[1][2] as well as >>>>>>>>>>>>>>>>> our ML code base when you try those algorithms out. Please >>>>>>>>>>>>>>>>> try to do rough >>>>>>>>>>>>>>>>> implementations of the streaming versions of linear >>>>>>>>>>>>>>>>> regression, logistic >>>>>>>>>>>>>>>>> regression and k-means clustering as we have discussed in the >>>>>>>>>>>>>>>>> proposal in >>>>>>>>>>>>>>>>> plain Java. It's better if you can create a git repo and >>>>>>>>>>>>>>>>> share your code >>>>>>>>>>>>>>>>> once you have made some progress. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Were you able debug and understand the flow of the ML >>>>>>>>>>>>>>>>> siddhi extension? I hope you haven't encountered more errors >>>>>>>>>>>>>>>>> after >>>>>>>>>>>>>>>>> switching the released version of CEP. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Is this Friday okay for you? Afternoon at 2:00 pm? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>> http://spark.apache.org/docs/latest/programming-guide.html >>>>>>>>>>>>>>>>> [2] http://spark.apache.org/docs/latest/mllib-guide.html >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Wed, May 4, 2016 at 1:07 PM, Mahesh Dananjaya < >>>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>>>>>> I have been looking into some algorithms related to >>>>>>>>>>>>>>>>>> stochastic gradient descent based algorithms.anything i >>>>>>>>>>>>>>>>>> should focus please >>>>>>>>>>>>>>>>>> let me know.Ans also i will be available for calling this >>>>>>>>>>>>>>>>>> week and next >>>>>>>>>>>>>>>>>> week.thank you. >>>>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 5:05 PM, Mahesh Dananjaya < >>>>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>>>>>>> thank you.that's good. i have been trying to fix that >>>>>>>>>>>>>>>>>>> for couple of days. please inform me when it will be >>>>>>>>>>>>>>>>>>> fixed.now i have been >>>>>>>>>>>>>>>>>>> testing the ML algorithms and trying to identify the flow >>>>>>>>>>>>>>>>>>> and the >>>>>>>>>>>>>>>>>>> hierarchy. is that "Put break points in train methods in >>>>>>>>>>>>>>>>>>> Linear Regression >>>>>>>>>>>>>>>>>>> class" means the spark/algorithms/ LinearRegrassion.java >>>>>>>>>>>>>>>>>>> class in the >>>>>>>>>>>>>>>>>>> org.wso2.carbon.ml.core? is that the correct file? >>>>>>>>>>>>>>>>>>> And also i am planning to write some programs to use >>>>>>>>>>>>>>>>>>> apache spark mllib algorithms. and i refer to [1] and some >>>>>>>>>>>>>>>>>>> wso2 >>>>>>>>>>>>>>>>>>> documentations to get some idea about ML structure.thank >>>>>>>>>>>>>>>>>>> you. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [1]nirmalfdo.blogspot.com >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 4:36 PM, Maheshakya Wijewardena < >>>>>>>>>>>>>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I have checked. It seems the issue you have encountered >>>>>>>>>>>>>>>>>>>> is cause only in the current development branch of the >>>>>>>>>>>>>>>>>>>> product-cep. It >>>>>>>>>>>>>>>>>>>> doesn't identify the ML siddhi extension as an extension. >>>>>>>>>>>>>>>>>>>> ML siddhi >>>>>>>>>>>>>>>>>>>> extension works fine in the latest release of CEP (4.1.0) >>>>>>>>>>>>>>>>>>>> [1]. >>>>>>>>>>>>>>>>>>>> Until we figure out the reason and come up with a >>>>>>>>>>>>>>>>>>>> solution, can you use the latest CEP release for your >>>>>>>>>>>>>>>>>>>> work. It's fine to >>>>>>>>>>>>>>>>>>>> use that since you haven't started actual development yet. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [1] http://wso2.com/products/complex-event-processor/ >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 3:19 PM, Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>>> <mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Is is vital to use those local repo in my upcoming >>>>>>>>>>>>>>>>>>>>>> implementation? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Yes. The remote p2-repo contains the p2-repos of >>>>>>>>>>>>>>>>>>>>> released versions. What you have to develop on is the >>>>>>>>>>>>>>>>>>>>> current master of the >>>>>>>>>>>>>>>>>>>>> carbon-ml and product-ml. You can try out with the >>>>>>>>>>>>>>>>>>>>> modification I have >>>>>>>>>>>>>>>>>>>>> suggested. In the meantime, I'll verify whether the >>>>>>>>>>>>>>>>>>>>> current repos are >>>>>>>>>>>>>>>>>>>>> working as expected. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> And also i am trying to debug the carbon-ml >>>>>>>>>>>>>>>>>>>>>> org.wso2.carbon.ml.core by putting some break point in >>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> spark/algorithms/Linear Regression >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> It's great that you have started looking at the >>>>>>>>>>>>>>>>>>>>> implementation of linear regression as well. Put break >>>>>>>>>>>>>>>>>>>>> points in train >>>>>>>>>>>>>>>>>>>>> methods in LinearRegression class. This is being used >>>>>>>>>>>>>>>>>>>>> when you run linear >>>>>>>>>>>>>>>>>>>>> regression from UI. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I can see some comments left behind for streaming algo >>>>>>>>>>>>>>>>>>>>>> as well.thank you >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> You may be referring to the linear regression with SGD >>>>>>>>>>>>>>>>>>>>> model. Here, there's no retraining with streaming data >>>>>>>>>>>>>>>>>>>>> involved. The SGD >>>>>>>>>>>>>>>>>>>>> with minibatches is used to train the model with the data >>>>>>>>>>>>>>>>>>>>> set only once. >>>>>>>>>>>>>>>>>>>>> What you have to do is create a similar mechanism to >>>>>>>>>>>>>>>>>>>>> involve streaming data and retrain models. We will get to >>>>>>>>>>>>>>>>>>>>> that part once >>>>>>>>>>>>>>>>>>>>> you get comfortable with siddhi extensions. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> BTW, is it possible for you to join a call on this >>>>>>>>>>>>>>>>>>>>> Friday or in the next week. We'll try to resolve your >>>>>>>>>>>>>>>>>>>>> current issues and >>>>>>>>>>>>>>>>>>>>> discuss further on project. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 1:03 PM, Mahesh Dananjaya < >>>>>>>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hi maheshakya, >>>>>>>>>>>>>>>>>>>>>> Is it ok to go with p2 repo at >>>>>>>>>>>>>>>>>>>>>> http://product-dist.wso2.com/p2/carbon/releases/wilkes/features/ >>>>>>>>>>>>>>>>>>>>>> rather than the P2-repo at >>>>>>>>>>>>>>>>>>>>>> product-ml/modules/p2-profile/target/p2-repo in >>>>>>>>>>>>>>>>>>>>>> local repo.What is the impact?.Is is vital to use those >>>>>>>>>>>>>>>>>>>>>> local repo in my >>>>>>>>>>>>>>>>>>>>>> upcoming implementation?so i was trying to give remote >>>>>>>>>>>>>>>>>>>>>> p2 repo to cep built >>>>>>>>>>>>>>>>>>>>>> by source and trying to debug the cep ml extension and >>>>>>>>>>>>>>>>>>>>>> got the same error >>>>>>>>>>>>>>>>>>>>>> as yesterday. But pre-built product is working fine. >>>>>>>>>>>>>>>>>>>>>> Therefore now i am >>>>>>>>>>>>>>>>>>>>>> trying as you described in the last email. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> And also i am trying to debug the carbon-ml >>>>>>>>>>>>>>>>>>>>>> org.wso2.carbon.ml.core by putting some break point in >>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> spark/algorithms/Linear Regression. I am trying to >>>>>>>>>>>>>>>>>>>>>> trigger it with >>>>>>>>>>>>>>>>>>>>>> product-ml project with data set. Does that Linear >>>>>>>>>>>>>>>>>>>>>> Regression also in UI is >>>>>>>>>>>>>>>>>>>>>> consuming that spark algorithms or is it in another >>>>>>>>>>>>>>>>>>>>>> place? I can see some >>>>>>>>>>>>>>>>>>>>>> comments left behind for streaming algo as well.thank >>>>>>>>>>>>>>>>>>>>>> you. >>>>>>>>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 9:35 AM, Maheshakya >>>>>>>>>>>>>>>>>>>>>> Wijewardena <mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> The earlier error you have mentioned may occur due >>>>>>>>>>>>>>>>>>>>>>> to the incompatible Siddhi versions in ML p2-repo and >>>>>>>>>>>>>>>>>>>>>>> CEP, when you add the >>>>>>>>>>>>>>>>>>>>>>> p2-repo from prodcut-ml you built. >>>>>>>>>>>>>>>>>>>>>>> Current siddhi version in product-cep is >>>>>>>>>>>>>>>>>>>>>>> 3.0.6-SNAPSHOT[1], but in ML it's 3.0.2. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Can you try changing the siddhi.version in >>>>>>>>>>>>>>>>>>>>>>> carbon-ml/pom.xml to 3.0.6-SNAPSHOT, build carbon-ml, >>>>>>>>>>>>>>>>>>>>>>> then build product-ml >>>>>>>>>>>>>>>>>>>>>>> again. After this, add p2-repo as a local repository >>>>>>>>>>>>>>>>>>>>>>> again to fresh CEP >>>>>>>>>>>>>>>>>>>>>>> pack and try it out. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Mon, May 2, 2016 at 7:02 PM, Mahesh Dananjaya < >>>>>>>>>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>>>>>>>>>>>> now i remotely debug the CEP extension for ML >>>>>>>>>>>>>>>>>>>>>>>> Prediction. What i did was, i have all the pre-build >>>>>>>>>>>>>>>>>>>>>>>> version of CEP and ML. >>>>>>>>>>>>>>>>>>>>>>>> Therefore i used that built CEP and did the same thing >>>>>>>>>>>>>>>>>>>>>>>> that i was doing >>>>>>>>>>>>>>>>>>>>>>>> with the source code. I think the only change i did >>>>>>>>>>>>>>>>>>>>>>>> was install those >>>>>>>>>>>>>>>>>>>>>>>> packages from remote p2 repo. This was work fine and i >>>>>>>>>>>>>>>>>>>>>>>> debugged the >>>>>>>>>>>>>>>>>>>>>>>> carbon-ml org.wso2.carbon.ml.siddhi.extension as >>>>>>>>>>>>>>>>>>>>>>>> described in the [1]. So >>>>>>>>>>>>>>>>>>>>>>>> now i have to try same thing with the build by source >>>>>>>>>>>>>>>>>>>>>>>> content.thank you. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>>>> https://docs.wso2.com/display/ML110/WSO2+CEP+Extension+for+ML+Predictions#WSO2CEPExtensionforMLPredictions-Siddhisyntaxfortheextension >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>>>>>>>>>> Mahesh >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 25, 2016 at 5:49 PM, Maheshakya >>>>>>>>>>>>>>>>>>>>>>>> Wijewardena <mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Congratulations and welcome to GSoC 2016. You did >>>>>>>>>>>>>>>>>>>>>>>>> a great job in preparing the proposal. Now it's time >>>>>>>>>>>>>>>>>>>>>>>>> to dig deep and get >>>>>>>>>>>>>>>>>>>>>>>>> started with the project. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> First of all you need to familiarize with the code >>>>>>>>>>>>>>>>>>>>>>>>> base. We have agreed to implement this with CEP event >>>>>>>>>>>>>>>>>>>>>>>>> streams. We already >>>>>>>>>>>>>>>>>>>>>>>>> have a CEP extension for predictions [1][2]. Go >>>>>>>>>>>>>>>>>>>>>>>>> through this implementation >>>>>>>>>>>>>>>>>>>>>>>>> and familiarize your self with that. You need to >>>>>>>>>>>>>>>>>>>>>>>>> understand how: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> 1. Even streams are consumed >>>>>>>>>>>>>>>>>>>>>>>>> 2. predictions are made from individual event >>>>>>>>>>>>>>>>>>>>>>>>> 3. Results are sent back >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Get WSO2 ML and CEP sources (You may use latest >>>>>>>>>>>>>>>>>>>>>>>>> released version of CEP) and build the products. Get >>>>>>>>>>>>>>>>>>>>>>>>> both carbon-ml[3] and >>>>>>>>>>>>>>>>>>>>>>>>> product-ml[4] masters and create new branches for >>>>>>>>>>>>>>>>>>>>>>>>> your work from masters. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> After you build the products, you may need to do >>>>>>>>>>>>>>>>>>>>>>>>> remote debugging[5] to understand the flow. So please >>>>>>>>>>>>>>>>>>>>>>>>> follow an example of >>>>>>>>>>>>>>>>>>>>>>>>> real time prediction with ML with debugging and get >>>>>>>>>>>>>>>>>>>>>>>>> some idea. The >>>>>>>>>>>>>>>>>>>>>>>>> component you need to debug is >>>>>>>>>>>>>>>>>>>>>>>>> org.wso2.carbon.ml.siddhi.extension. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Next tasks would be implementing online learning >>>>>>>>>>>>>>>>>>>>>>>>> algorithms in plain java with spark ml lib and >>>>>>>>>>>>>>>>>>>>>>>>> integrating those to ML. We >>>>>>>>>>>>>>>>>>>>>>>>> also need to come up with a proper and detailed >>>>>>>>>>>>>>>>>>>>>>>>> architecture to employ >>>>>>>>>>>>>>>>>>>>>>>>> those algorithms in ML. Getting familiar with the >>>>>>>>>>>>>>>>>>>>>>>>> aforementioned sections >>>>>>>>>>>>>>>>>>>>>>>>> would give you some insight on how this should be >>>>>>>>>>>>>>>>>>>>>>>>> implemented. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> So please try to get a quick grasp then you can >>>>>>>>>>>>>>>>>>>>>>>>> start the implementation. Let us know if you have any >>>>>>>>>>>>>>>>>>>>>>>>> questions or you get >>>>>>>>>>>>>>>>>>>>>>>>> stuck somewhere. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Also, please always add WSO2 developer's list as >>>>>>>>>>>>>>>>>>>>>>>>> well when you communicate with us regarding the >>>>>>>>>>>>>>>>>>>>>>>>> project so that you can get >>>>>>>>>>>>>>>>>>>>>>>>> opinions and feedback from others as well. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>>>>> https://docs.wso2.com/display/ML110/WSO2+CEP+Extension+for+ML+Predictions#WSO2CEPExtensionforMLPredictions-Siddhisyntaxfortheextension >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> [2] >>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/wso2/carbon-ml/tree/master/components/extensions/org.wso2.carbon.ml.siddhi.extension >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> [3] https://github.com/wso2/carbon-ml >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> [4] https://github.com/wso2/product-ml >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> [5] >>>>>>>>>>>>>>>>>>>>>>>>> https://dzone.com/articles/how-debug-wso2-carbon-kernel >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 25, 2016 at 3:33 PM, Mahesh Dananjaya >>>>>>>>>>>>>>>>>>>>>>>>> <dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>>>> thank you for accepting my GSOC 2016 proposal and >>>>>>>>>>>>>>>>>>>>>>>>>> i am looking forward for the further instruction and >>>>>>>>>>>>>>>>>>>>>>>>>> project continuation. >>>>>>>>>>>>>>>>>>>>>>>>>> thank you very much. >>>>>>>>>>>>>>>>>>>>>>>>>> regards, >>>>>>>>>>>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>>>> Dev@wso2.org >>>>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>> mahesha...@wso2.com >>>>>>>>> +94711228855 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>> mahesha...@wso2.com >>>>>>> +94711228855 >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >> >> -- >> Pruthuvi Maheshakya Wijewardena >> mahesha...@wso2.com >> +94711228855 >> >> >> > > > -- > Pruthuvi Maheshakya Wijewardena > mahesha...@wso2.com > +94711228855 > > >
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
