Hi Mahesh, As we discussed offline, we can use similar mechanism to train linear regression models, logistic regression models and k-means clustering models.
It is very interesting that i have found that somethings that can make use > of our work. In the cep 4.0 documentation there is a Custom Stream > Processor Extention program [1]. There is a example of > LinearRegressionStreamProcessor [1]. > As we have to train predictive models with Spark, you can write wrappers around regression/clustering models of Spark. Refer to Siddhi time series regression source codes[1][2]. You can write a streaming linear regression class for ML in a similar fashion by wrapping Spark mllib implementations. You can use the methods "addEvent", "removeEvent", etc. (may have to be changed according to requirements) for the similar purpose. You can introduce trainLinearRegression/LogisticRegression/Kmeans which does a similar thing as in createLinearRegression in those time series functions. In the processData method you can use Spark mllib classes to actually train models and return the model weights, evaluation metrics. So, converting streams into RDDs and retrieving information from the trained models shall happen in this method. In the stream processor extension example, you can retrieve those values then use them to train new models with new batches. Weights/cluster centers maybe passed as initialization parameters for the wrappers. Please note that we have to figure out the best siddhi extension type for this process. In the siddhi query, we define batch size, type of algorithm and number of features (there can be more). After batch size number of events received, train a model and save parameters, return evaluation metric. With the next batch, retrain the model initialized with previously learned parameters. We also may need to test the same scenario with a moving window, but I suspect that that approach may become so slow as a model is trained each time an event is received. So, we may have to change the number of slots the moving window moves at a time (eg: not one by one, but ten by ten). Once this is resolved, majority of the research part will be finished and all we will be left to do is implementing wrappers around the 3 learning algorithms we consider. Best regards. [1] https://github.com/wso2/siddhi/blob/master/modules/siddhi-extensions/timeseries/src/main/java/org/wso2/siddhi/extension/timeseries/linreg/RegressionCalculator.java [2] https://github.com/wso2/siddhi/blob/master/modules/siddhi-extensions/timeseries/src/main/java/org/wso2/siddhi/extension/timeseries/linreg/SimpleLinearRegressionCalculator.java On Sat, May 21, 2016 at 2:55 PM, Mahesh Dananjaya <dananjayamah...@gmail.com > wrote: > Hi Maheshkya, > shall we use [1] for our work? i am checking the possibility. > BR, > Mahesh. > [1] > https://docs.wso2.com/display/CEP400/Writing+a+Custom+Stream+Processor+Extension > [2] > https://docs.wso2.com/display/CEP400/Inbuilt+Windows#InbuiltWindows-lengthlength > [3] > https://docs.wso2.com/display/CEP400/Writing+a+Custom+Aggregate+Function > > On Sat, May 21, 2016 at 2:44 PM, Mahesh Dananjaya < > dananjayamah...@gmail.com> wrote: > >> Hi Maheshakya, >> It is very interesting that i have found that somethings that can make >> use of our work. In the cep 4.0 documentation there is a Custom Stream >> Processor Extention program [1]. There is a example of >> LinearRegressionStreamProcessor [1] and also i saw >> private int batchSize = 1000000000; i am going through this one. >> Please check whether we can use. WIll there be any compatibility or >> support issue? >> regards, >> Mahesh. >> >> >> [1] >> https://docs.wso2.com/display/CEP400/Writing+a+Custom+Stream+Processor+Extension >> >> On Sat, May 21, 2016 at 11:52 AM, Mahesh Dananjaya < >> dananjayamah...@gmail.com> wrote: >> >>> Hi maheshakya, >>> anyway how can test any siddhi extention after write it without >>> integrating it to cep.can you please explain me the procedure. i am >>> referring to [1] [2] [3] [4]. thank you. >>> BR, >>> Mahesh. >>> >>> [1] https://docs.wso2.com/display/CEP310/Writing+Extensions+to+Siddhi >>> [2] https://docs.wso2.com/display/CEP310/Writing+a+Custom+Function >>> [3] https://docs.wso2.com/display/CEP310/Writing+a+Custom+Window >>> [4] https://docs.wso2.com/display/CEP400/Writing+Extensions+to+Siddhi >>> >>> On Thu, May 19, 2016 at 12:08 PM, Mahesh Dananjaya < >>> dananjayamah...@gmail.com> wrote: >>> >>>> Hi Maheshakya, >>>> thank you for the feedback. I have add data-sets into repo. >>>> data-sets/lr. I am all right with next week.Now i am writing some examples >>>> to collect samples and build mini batches and run the algorithms on those >>>> mini-batches. thank you. will add those into repo soon.I am still working >>>> on that siddhi extention.i will let you know the progress. >>>> BR, >>>> mahesh. >>>> >>>> On Thu, May 19, 2016 at 11:10 AM, Maheshakya Wijewardena < >>>> mahesha...@wso2.com> wrote: >>>> >>>>> Hi Mahesh, >>>>> >>>>> I've look into your code sample of streaming linear regression. Looks >>>>> good to me, apart from few issues in coding practices which we can improve >>>>> when you're doing the implementations in carbon-ml and during the code >>>>> reviews. You are using a set of files as mini-batches of data, right? Can >>>>> you also send us the datasets you've been using. I'd like to run this. >>>>> >>>>> does that cep problem is now all right that we were trying to fix. I >>>>>> am still using those pre-build versions. If so i can merge with the >>>>>> latest >>>>>> one. >>>>> >>>>> >>>>> I'll check this and let you know. >>>>> >>>>> Can we arrange a meeting (preferably in WSO2 offices) in next week >>>>> with ML team members as well. Coding period begins on next Monday, so it's >>>>> better to get overall feedback from others and discuss more about the >>>>> project. Let me know convenient time slots for you. I'll arrange a meeting >>>>> with ML team. >>>>> >>>>> Best regards. >>>>> >>>>> On Wed, May 18, 2016 at 9:53 AM, Mahesh Dananjaya < >>>>> dananjayamah...@gmail.com> wrote: >>>>> >>>>>> Hi Maheshakya, >>>>>> Ok. I will check it.you have sent me those relevant references and i >>>>>> am working on that thing.thank you. does that cep problem is now all >>>>>> right >>>>>> that we were trying to fix. I am still using those pre-build versions. If >>>>>> so i can merge with the latest one.thanks. >>>>>> BR, >>>>>> Mahesh. >>>>>> >>>>>> On Wed, May 18, 2016 at 9:44 AM, Maheshakya Wijewardena < >>>>>> mahesha...@wso2.com> wrote: >>>>>> >>>>>>> Hi Mahesh, >>>>>>> >>>>>>> You don't actually have to implement anything in spark streaming. >>>>>>> Try to understand how streaming data is handled in and the specifics of >>>>>>> the >>>>>>> underlying algorithms in streaming. >>>>>>> What we want to do is having the similar algorithms that support CEP >>>>>>> event streams with siddhi. >>>>>>> >>>>>>> Best regards. >>>>>>> >>>>>>> On Wed, May 18, 2016 at 9:38 AM, Mahesh Dananjaya < >>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Maheshakya, >>>>>>>> Did you check the repo. I will add recent works today.And also i >>>>>>>> was going through the Java docs related to spark streaming work. It is >>>>>>>> with >>>>>>>> that scala API. thank you. >>>>>>>> regards, >>>>>>>> Mahesh. >>>>>>>> >>>>>>>> On Tue, May 17, 2016 at 10:11 AM, Mahesh Dananjaya < >>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi Maheshakya, >>>>>>>>> I have gone through the Java Docs and run some of the Spark >>>>>>>>> examples on spark shell which are paramount improtant for our work. >>>>>>>>> Then i >>>>>>>>> have been writing my codes to check the Linear regression, K means for >>>>>>>>> streaming. please check my git repo [1]. I think now i have to ask on >>>>>>>>> dev >>>>>>>>> regarding the capturing event streams for our work. I will update the >>>>>>>>> recent things on git. check the park-example directory for java. >>>>>>>>> examples >>>>>>>>> run on git shell is not included there. In my case i think i have to >>>>>>>>> build >>>>>>>>> mini batches from data streams that comes as individual samples. Now >>>>>>>>> i am >>>>>>>>> working on some coding to collect mini batches from data >>>>>>>>> streams.thank you. >>>>>>>>> regards, >>>>>>>>> Mahesh. >>>>>>>>> [1]https://github.com/dananjayamahesh/GSOC2016 >>>>>>>>> >>>>>>>>> On Tue, May 17, 2016 at 10:10 AM, Mahesh Dananjaya < >>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Maheshakya, >>>>>>>>>> I have gone through the Java Docs and run some of the Spark >>>>>>>>>> examples on spark shell which are paramount improtant for our work. >>>>>>>>>> Then i >>>>>>>>>> have been writing my codes to check the Linear regression, K means >>>>>>>>>> for >>>>>>>>>> streaming. please check my git repo [1]. I think now i have to ask >>>>>>>>>> on dev >>>>>>>>>> regarding the capturing event streams for our work. I will update the >>>>>>>>>> recent things on git. check the park-example directory for java. >>>>>>>>>> examples >>>>>>>>>> run on git shell is not included there. In my case i think i have to >>>>>>>>>> build >>>>>>>>>> mini batches from data streams that comes as individual samples. Now >>>>>>>>>> i am >>>>>>>>>> working on some coding to collect mini batches from data >>>>>>>>>> streams.thank you. >>>>>>>>>> regards, >>>>>>>>>> Mahesh. >>>>>>>>>> [1]https://github.com/dananjayamahesh/GSOC2016 >>>>>>>>>> >>>>>>>>>> On Mon, May 16, 2016 at 1:19 PM, Mahesh Dananjaya < >>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>> thank you. i will update the repo today.thank you.i changed the >>>>>>>>>>> carbon ml siddhi extention and see how the changes are effecting. i >>>>>>>>>>> will >>>>>>>>>>> update the progress as soon as possible.thank you. i had some >>>>>>>>>>> problem in >>>>>>>>>>> spark mllib dependency. i was fixing that. >>>>>>>>>>> regards, >>>>>>>>>>> Mahesh. >>>>>>>>>>> p.s: do i need to maintain a blog? >>>>>>>>>>> >>>>>>>>>>> On Mon, May 16, 2016 at 10:02 AM, Maheshakya Wijewardena < >>>>>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>> >>>>>>>>>>>> Sorry for replying late. >>>>>>>>>>>> >>>>>>>>>>>> Thank you for the update. I believe you have done some >>>>>>>>>>>> implementations with with Spark MLLIb algorithms in streaming >>>>>>>>>>>> fashion as we >>>>>>>>>>>> have discussed. If so, can you please share your code in a Github >>>>>>>>>>>> repo. >>>>>>>>>>>> >>>>>>>>>>>> Now i want to implements some machine learning algorithms with >>>>>>>>>>>>> importing mllib and want to run within your code base >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> For the moment you can try out editing the same class >>>>>>>>>>>> PredictStreamProcessor in the siddhi extension in carbon-ml. Later >>>>>>>>>>>> we will >>>>>>>>>>>> add this separately. You should be able to add >>>>>>>>>>>> org.apache.spark.mllib. >>>>>>>>>>>> classes to there. >>>>>>>>>>>> >>>>>>>>>>>> And i want to see how event streams are coming from cep. As i >>>>>>>>>>>>> think it is not in a RDD format since it is arriving as the >>>>>>>>>>>>> individual >>>>>>>>>>>>> samples. I will send a email to dev asking about how to get the >>>>>>>>>>>>> streams. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Please pay attention to length[1] and lengthbatch[1] inbuilt >>>>>>>>>>>> windows in siddhi. What you need to write are functions similar to >>>>>>>>>>>> a custom >>>>>>>>>>>> aggregate function[2]. >>>>>>>>>>>> When you send the email to dev list, explain your requirement. >>>>>>>>>>>> You need to get a set of event with from a stream with a specified >>>>>>>>>>>> window >>>>>>>>>>>> size (number of events). Then build a model within that function. >>>>>>>>>>>> You also >>>>>>>>>>>> need to retain the data (learned weights, cluster centers, etc.) >>>>>>>>>>>> from the >>>>>>>>>>>> previous window to use in the current window. Ask what can be the >>>>>>>>>>>> most >>>>>>>>>>>> suitable option for this among the set of siddhi extensions given. >>>>>>>>>>>> >>>>>>>>>>>> Best regards. >>>>>>>>>>>> >>>>>>>>>>>> [1] >>>>>>>>>>>> https://docs.wso2.com/display/CEP400/Inbuilt+Windows#InbuiltWindows-lengthlength >>>>>>>>>>>> [2] >>>>>>>>>>>> https://docs.wso2.com/display/CEP400/Writing+a+Custom+Aggregate+Function >>>>>>>>>>>> >>>>>>>>>>>> On Wed, May 11, 2016 at 1:43 PM, Mahesh Dananjaya < >>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ---------- Forwarded message ---------- >>>>>>>>>>>>> From: Mahesh Dananjaya <dananjayamah...@gmail.com> >>>>>>>>>>>>> Date: Wed, May 11, 2016 at 1:43 PM >>>>>>>>>>>>> Subject: Re: [Dev] GSOC2016: [ML][CEP] Predictive analytic >>>>>>>>>>>>> with online data for WSO2 Machine Learner >>>>>>>>>>>>> To: Maheshakya Wijewardena <mahesha...@wso2.com> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>> sorry for not updating. I did what you wanted me to do. I >>>>>>>>>>>>> checked the code base and train functions. I went through those >>>>>>>>>>>>> java docs. >>>>>>>>>>>>> I went through the carbon-ml current implementation of LG and >>>>>>>>>>>>> K-Mean. And i >>>>>>>>>>>>> had Apache Spark and i tried with several examples. Now i want to >>>>>>>>>>>>> implements some machine learning algorithms with importing mllib >>>>>>>>>>>>> and want >>>>>>>>>>>>> to run within your code base. Can you help me with that. >>>>>>>>>>>>> And i want to see how event streams are coming from cep. As i >>>>>>>>>>>>> think it is not in a RDD format since it is arriving as the >>>>>>>>>>>>> individual >>>>>>>>>>>>> samples. I will send a email to dev asking about how to get the >>>>>>>>>>>>> streams. I >>>>>>>>>>>>> debugged many of those functions in the code base. So need further >>>>>>>>>>>>> instructions to proceed.thank you. >>>>>>>>>>>>> regards, >>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, May 11, 2016 at 10:32 AM, Maheshakya Wijewardena < >>>>>>>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Any update on your progress? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, May 4, 2016 at 8:35 PM, Maheshakya Wijewardena < >>>>>>>>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> is that "Put break points in train methods in Linear >>>>>>>>>>>>>>>> Regression class" means the spark/algorithms/ >>>>>>>>>>>>>>>> LinearRegrassion.java class >>>>>>>>>>>>>>>> in the org.wso2.carbon.ml.core? is that the correct file? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes, this is the correct place. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> You can refer to spark programming guide[1][2] as well as >>>>>>>>>>>>>>> our ML code base when you try those algorithms out. Please try >>>>>>>>>>>>>>> to do rough >>>>>>>>>>>>>>> implementations of the streaming versions of linear regression, >>>>>>>>>>>>>>> logistic >>>>>>>>>>>>>>> regression and k-means clustering as we have discussed in the >>>>>>>>>>>>>>> proposal in >>>>>>>>>>>>>>> plain Java. It's better if you can create a git repo and share >>>>>>>>>>>>>>> your code >>>>>>>>>>>>>>> once you have made some progress. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Were you able debug and understand the flow of the ML siddhi >>>>>>>>>>>>>>> extension? I hope you haven't encountered more errors after >>>>>>>>>>>>>>> switching the >>>>>>>>>>>>>>> released version of CEP. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Is this Friday okay for you? Afternoon at 2:00 pm? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>> http://spark.apache.org/docs/latest/programming-guide.html >>>>>>>>>>>>>>> [2] http://spark.apache.org/docs/latest/mllib-guide.html >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, May 4, 2016 at 1:07 PM, Mahesh Dananjaya < >>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>>>> I have been looking into some algorithms related to >>>>>>>>>>>>>>>> stochastic gradient descent based algorithms.anything i should >>>>>>>>>>>>>>>> focus please >>>>>>>>>>>>>>>> let me know.Ans also i will be available for calling this week >>>>>>>>>>>>>>>> and next >>>>>>>>>>>>>>>> week.thank you. >>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 5:05 PM, Mahesh Dananjaya < >>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>>>>> thank you.that's good. i have been trying to fix that for >>>>>>>>>>>>>>>>> couple of days. please inform me when it will be fixed.now i >>>>>>>>>>>>>>>>> have been >>>>>>>>>>>>>>>>> testing the ML algorithms and trying to identify the flow and >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> hierarchy. is that "Put break points in train methods in >>>>>>>>>>>>>>>>> Linear Regression >>>>>>>>>>>>>>>>> class" means the spark/algorithms/ LinearRegrassion.java >>>>>>>>>>>>>>>>> class in the >>>>>>>>>>>>>>>>> org.wso2.carbon.ml.core? is that the correct file? >>>>>>>>>>>>>>>>> And also i am planning to write some programs to use >>>>>>>>>>>>>>>>> apache spark mllib algorithms. and i refer to [1] and some >>>>>>>>>>>>>>>>> wso2 >>>>>>>>>>>>>>>>> documentations to get some idea about ML structure.thank you. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [1]nirmalfdo.blogspot.com >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 4:36 PM, Maheshakya Wijewardena < >>>>>>>>>>>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I have checked. It seems the issue you have encountered >>>>>>>>>>>>>>>>>> is cause only in the current development branch of the >>>>>>>>>>>>>>>>>> product-cep. It >>>>>>>>>>>>>>>>>> doesn't identify the ML siddhi extension as an extension. ML >>>>>>>>>>>>>>>>>> siddhi >>>>>>>>>>>>>>>>>> extension works fine in the latest release of CEP (4.1.0) >>>>>>>>>>>>>>>>>> [1]. >>>>>>>>>>>>>>>>>> Until we figure out the reason and come up with a >>>>>>>>>>>>>>>>>> solution, can you use the latest CEP release for your work. >>>>>>>>>>>>>>>>>> It's fine to >>>>>>>>>>>>>>>>>> use that since you haven't started actual development yet. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [1] http://wso2.com/products/complex-event-processor/ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 3:19 PM, Maheshakya Wijewardena < >>>>>>>>>>>>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Is is vital to use those local repo in my upcoming >>>>>>>>>>>>>>>>>>>> implementation? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Yes. The remote p2-repo contains the p2-repos of >>>>>>>>>>>>>>>>>>> released versions. What you have to develop on is the >>>>>>>>>>>>>>>>>>> current master of the >>>>>>>>>>>>>>>>>>> carbon-ml and product-ml. You can try out with the >>>>>>>>>>>>>>>>>>> modification I have >>>>>>>>>>>>>>>>>>> suggested. In the meantime, I'll verify whether the current >>>>>>>>>>>>>>>>>>> repos are >>>>>>>>>>>>>>>>>>> working as expected. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> And also i am trying to debug the carbon-ml >>>>>>>>>>>>>>>>>>>> org.wso2.carbon.ml.core by putting some break point in the >>>>>>>>>>>>>>>>>>>> spark/algorithms/Linear Regression >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> It's great that you have started looking at the >>>>>>>>>>>>>>>>>>> implementation of linear regression as well. Put break >>>>>>>>>>>>>>>>>>> points in train >>>>>>>>>>>>>>>>>>> methods in LinearRegression class. This is being used when >>>>>>>>>>>>>>>>>>> you run linear >>>>>>>>>>>>>>>>>>> regression from UI. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I can see some comments left behind for streaming algo >>>>>>>>>>>>>>>>>>>> as well.thank you >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> You may be referring to the linear regression with SGD >>>>>>>>>>>>>>>>>>> model. Here, there's no retraining with streaming data >>>>>>>>>>>>>>>>>>> involved. The SGD >>>>>>>>>>>>>>>>>>> with minibatches is used to train the model with the data >>>>>>>>>>>>>>>>>>> set only once. >>>>>>>>>>>>>>>>>>> What you have to do is create a similar mechanism to >>>>>>>>>>>>>>>>>>> involve streaming data and retrain models. We will get to >>>>>>>>>>>>>>>>>>> that part once >>>>>>>>>>>>>>>>>>> you get comfortable with siddhi extensions. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> BTW, is it possible for you to join a call on this >>>>>>>>>>>>>>>>>>> Friday or in the next week. We'll try to resolve your >>>>>>>>>>>>>>>>>>> current issues and >>>>>>>>>>>>>>>>>>> discuss further on project. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 1:03 PM, Mahesh Dananjaya < >>>>>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi maheshakya, >>>>>>>>>>>>>>>>>>>> Is it ok to go with p2 repo at >>>>>>>>>>>>>>>>>>>> http://product-dist.wso2.com/p2/carbon/releases/wilkes/features/ >>>>>>>>>>>>>>>>>>>> rather than the P2-repo at >>>>>>>>>>>>>>>>>>>> product-ml/modules/p2-profile/target/p2-repo in >>>>>>>>>>>>>>>>>>>> local repo.What is the impact?.Is is vital to use those >>>>>>>>>>>>>>>>>>>> local repo in my >>>>>>>>>>>>>>>>>>>> upcoming implementation?so i was trying to give remote p2 >>>>>>>>>>>>>>>>>>>> repo to cep built >>>>>>>>>>>>>>>>>>>> by source and trying to debug the cep ml extension and got >>>>>>>>>>>>>>>>>>>> the same error >>>>>>>>>>>>>>>>>>>> as yesterday. But pre-built product is working fine. >>>>>>>>>>>>>>>>>>>> Therefore now i am >>>>>>>>>>>>>>>>>>>> trying as you described in the last email. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> And also i am trying to debug the carbon-ml >>>>>>>>>>>>>>>>>>>> org.wso2.carbon.ml.core by putting some break point in the >>>>>>>>>>>>>>>>>>>> spark/algorithms/Linear Regression. I am trying to trigger >>>>>>>>>>>>>>>>>>>> it with >>>>>>>>>>>>>>>>>>>> product-ml project with data set. Does that Linear >>>>>>>>>>>>>>>>>>>> Regression also in UI is >>>>>>>>>>>>>>>>>>>> consuming that spark algorithms or is it in another place? >>>>>>>>>>>>>>>>>>>> I can see some >>>>>>>>>>>>>>>>>>>> comments left behind for streaming algo as well.thank you. >>>>>>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 9:35 AM, Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>>> <mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> The earlier error you have mentioned may occur due to >>>>>>>>>>>>>>>>>>>>> the incompatible Siddhi versions in ML p2-repo and CEP, >>>>>>>>>>>>>>>>>>>>> when you add the >>>>>>>>>>>>>>>>>>>>> p2-repo from prodcut-ml you built. >>>>>>>>>>>>>>>>>>>>> Current siddhi version in product-cep is >>>>>>>>>>>>>>>>>>>>> 3.0.6-SNAPSHOT[1], but in ML it's 3.0.2. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Can you try changing the siddhi.version in >>>>>>>>>>>>>>>>>>>>> carbon-ml/pom.xml to 3.0.6-SNAPSHOT, build carbon-ml, >>>>>>>>>>>>>>>>>>>>> then build product-ml >>>>>>>>>>>>>>>>>>>>> again. After this, add p2-repo as a local repository >>>>>>>>>>>>>>>>>>>>> again to fresh CEP >>>>>>>>>>>>>>>>>>>>> pack and try it out. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Mon, May 2, 2016 at 7:02 PM, Mahesh Dananjaya < >>>>>>>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>>>>>>>>>> now i remotely debug the CEP extension for ML >>>>>>>>>>>>>>>>>>>>>> Prediction. What i did was, i have all the pre-build >>>>>>>>>>>>>>>>>>>>>> version of CEP and ML. >>>>>>>>>>>>>>>>>>>>>> Therefore i used that built CEP and did the same thing >>>>>>>>>>>>>>>>>>>>>> that i was doing >>>>>>>>>>>>>>>>>>>>>> with the source code. I think the only change i did was >>>>>>>>>>>>>>>>>>>>>> install those >>>>>>>>>>>>>>>>>>>>>> packages from remote p2 repo. This was work fine and i >>>>>>>>>>>>>>>>>>>>>> debugged the >>>>>>>>>>>>>>>>>>>>>> carbon-ml org.wso2.carbon.ml.siddhi.extension as >>>>>>>>>>>>>>>>>>>>>> described in the [1]. So >>>>>>>>>>>>>>>>>>>>>> now i have to try same thing with the build by source >>>>>>>>>>>>>>>>>>>>>> content.thank you. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>> https://docs.wso2.com/display/ML110/WSO2+CEP+Extension+for+ML+Predictions#WSO2CEPExtensionforMLPredictions-Siddhisyntaxfortheextension >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>>>>>>>> Mahesh >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 25, 2016 at 5:49 PM, Maheshakya >>>>>>>>>>>>>>>>>>>>>> Wijewardena <mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Congratulations and welcome to GSoC 2016. You did a >>>>>>>>>>>>>>>>>>>>>>> great job in preparing the proposal. Now it's time to >>>>>>>>>>>>>>>>>>>>>>> dig deep and get >>>>>>>>>>>>>>>>>>>>>>> started with the project. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> First of all you need to familiarize with the code >>>>>>>>>>>>>>>>>>>>>>> base. We have agreed to implement this with CEP event >>>>>>>>>>>>>>>>>>>>>>> streams. We already >>>>>>>>>>>>>>>>>>>>>>> have a CEP extension for predictions [1][2]. Go through >>>>>>>>>>>>>>>>>>>>>>> this implementation >>>>>>>>>>>>>>>>>>>>>>> and familiarize your self with that. You need to >>>>>>>>>>>>>>>>>>>>>>> understand how: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> 1. Even streams are consumed >>>>>>>>>>>>>>>>>>>>>>> 2. predictions are made from individual event >>>>>>>>>>>>>>>>>>>>>>> 3. Results are sent back >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Get WSO2 ML and CEP sources (You may use latest >>>>>>>>>>>>>>>>>>>>>>> released version of CEP) and build the products. Get >>>>>>>>>>>>>>>>>>>>>>> both carbon-ml[3] and >>>>>>>>>>>>>>>>>>>>>>> product-ml[4] masters and create new branches for your >>>>>>>>>>>>>>>>>>>>>>> work from masters. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> After you build the products, you may need to do >>>>>>>>>>>>>>>>>>>>>>> remote debugging[5] to understand the flow. So please >>>>>>>>>>>>>>>>>>>>>>> follow an example of >>>>>>>>>>>>>>>>>>>>>>> real time prediction with ML with debugging and get >>>>>>>>>>>>>>>>>>>>>>> some idea. The >>>>>>>>>>>>>>>>>>>>>>> component you need to debug is >>>>>>>>>>>>>>>>>>>>>>> org.wso2.carbon.ml.siddhi.extension. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Next tasks would be implementing online learning >>>>>>>>>>>>>>>>>>>>>>> algorithms in plain java with spark ml lib and >>>>>>>>>>>>>>>>>>>>>>> integrating those to ML. We >>>>>>>>>>>>>>>>>>>>>>> also need to come up with a proper and detailed >>>>>>>>>>>>>>>>>>>>>>> architecture to employ >>>>>>>>>>>>>>>>>>>>>>> those algorithms in ML. Getting familiar with the >>>>>>>>>>>>>>>>>>>>>>> aforementioned sections >>>>>>>>>>>>>>>>>>>>>>> would give you some insight on how this should be >>>>>>>>>>>>>>>>>>>>>>> implemented. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> So please try to get a quick grasp then you can >>>>>>>>>>>>>>>>>>>>>>> start the implementation. Let us know if you have any >>>>>>>>>>>>>>>>>>>>>>> questions or you get >>>>>>>>>>>>>>>>>>>>>>> stuck somewhere. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Also, please always add WSO2 developer's list as >>>>>>>>>>>>>>>>>>>>>>> well when you communicate with us regarding the project >>>>>>>>>>>>>>>>>>>>>>> so that you can get >>>>>>>>>>>>>>>>>>>>>>> opinions and feedback from others as well. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>>> https://docs.wso2.com/display/ML110/WSO2+CEP+Extension+for+ML+Predictions#WSO2CEPExtensionforMLPredictions-Siddhisyntaxfortheextension >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> [2] >>>>>>>>>>>>>>>>>>>>>>> https://github.com/wso2/carbon-ml/tree/master/components/extensions/org.wso2.carbon.ml.siddhi.extension >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> [3] https://github.com/wso2/carbon-ml >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> [4] https://github.com/wso2/product-ml >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> [5] >>>>>>>>>>>>>>>>>>>>>>> https://dzone.com/articles/how-debug-wso2-carbon-kernel >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 25, 2016 at 3:33 PM, Mahesh Dananjaya < >>>>>>>>>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>> thank you for accepting my GSOC 2016 proposal and i >>>>>>>>>>>>>>>>>>>>>>>> am looking forward for the further instruction and >>>>>>>>>>>>>>>>>>>>>>>> project continuation. >>>>>>>>>>>>>>>>>>>>>>>> thank you very much. >>>>>>>>>>>>>>>>>>>>>>>> regards, >>>>>>>>>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>> Dev@wso2.org >>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>> +94711228855 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>> mahesha...@wso2.com >>>>>>> +94711228855 >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Pruthuvi Maheshakya Wijewardena >>>>> mahesha...@wso2.com >>>>> +94711228855 >>>>> >>>>> >>>>> >>>> >>> >> > -- Pruthuvi Maheshakya Wijewardena mahesha...@wso2.com +94711228855
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
