Yes path is valid, I explicitly set the path here from the MLModelHandler persistModel method.
On 21 August 2015 at 10:26, Nirmal Fernando <nir...@wso2.com> wrote: > > > On Thu, Aug 20, 2015 at 9:21 PM, Madawa Soysa <madawa...@cse.mrt.ac.lk> > wrote: > >> Hi All, >> >> There an issue with serializing Spark's MatrixFactorizationModel object. >> The object contains a huge RDD and as I have read in many blogs, this model >> cannot be serialized as a java object. Therefore when retrieving the model >> I get the same exception as above; >> >> *Caused by: java.lang.ClassNotFoundException: >> org.apache.spark.OneToOneDependency cannot be found by >> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >> >> I have asked this question in Spark mailing lists and they recommended me >> to use the built in save and load functions other than using Java >> serializing. So I have used following method to persist the model, >> >> model.save(MLCoreServiceValueHolder.*getInstance()*.getSparkContext().sc(), >> outPath);[1] >> <https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9> >> >> Then nothing happens when this line executes. No error is thrown as well. >> Any solution for this? >> > > Can you print outPath and see whether it's a valid file path? > > >> >> [1] - >> https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9 >> >> On 16 August 2015 at 18:06, Madawa Soysa <madawa...@cse.mrt.ac.lk> wrote: >> >>> Yes I was able to resolve the issue by removing RDD fields from the >>> SummaryModel object as @Mano pointed out. Still I have the same exception >>> when retrieving the model. Trying to fix that issue. >>> >>> On 14 August 2015 at 10:43, Nirmal Fernando <nir...@wso2.com> wrote: >>> >>>> Thanks Niranda, this doc is useful. >>>> >>>> On Fri, Aug 14, 2015 at 10:36 AM, Niranda Perera <nira...@wso2.com> >>>> wrote: >>>> >>>>> From what I know, OneToOneDependancy come into play when spark tries >>>>> to create the RDD dependency tree. >>>>> >>>>> Just thought of sharing that. this would be a good resource [1] :-) >>>>> >>>>> [1] >>>>> https://databricks-training.s3.amazonaws.com/slides/advanced-spark-training.pdf >>>>> >>>>> On Thu, Aug 13, 2015 at 12:09 AM, Nirmal Fernando <nir...@wso2.com> >>>>> wrote: >>>>> >>>>>> What is *org.apache.spark.OneToOneDependency ? Is it something you >>>>>> use?* >>>>>> >>>>>> On Wed, Aug 12, 2015 at 11:30 PM, Madawa Soysa < >>>>>> madawa...@cse.mrt.ac.lk> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I created a model summary in order to show the model data in the >>>>>>> analysis.jag page. >>>>>>> But when refreshing the page after building the model I get the >>>>>>> following error. >>>>>>> >>>>>>> org.wso2.carbon.ml.core.exceptions.MLAnalysisHandlerException: An >>>>>>> error has occurred while extracting all the models of analysis id: 13 >>>>>>> at >>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:245) >>>>>>> at >>>>>>> org.wso2.carbon.ml.rest.api.AnalysisApiV10.getAllModelsOfAnalysis(AnalysisApiV10.java:517) >>>>>>> Caused by: >>>>>>> org.wso2.carbon.ml.database.exceptions.DatabaseHandlerException: An >>>>>>> error >>>>>>> has occurred while extracting all the models of analysis id: 13 >>>>>>> at >>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1797) >>>>>>> at >>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:243) >>>>>>> ... 52 more >>>>>>> >>>>>>> *Caused by: java.lang.ClassNotFoundException: >>>>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>>>> at >>>>>>> org.wso2.carbon.ml.database.util.MLDBUtil.getModelSummaryFromInputStream(MLDBUtil.java:54) >>>>>>> at >>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1790) >>>>>>> ... 53 more >>>>>>> >>>>>>> I guess there is an error in persistence of the model summary >>>>>>> object, what should be the cause for this error? [1] >>>>>>> <https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22> >>>>>>> contains >>>>>>> the commit which I introduced the model summary. >>>>>>> >>>>>>> [1] - >>>>>>> https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22 >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> *_________________________________**Madawa Soysa* >>>>>>> >>>>>>> Undergraduate, >>>>>>> >>>>>>> Department of Computer Science and Engineering, >>>>>>> >>>>>>> University of Moratuwa. >>>>>>> >>>>>>> >>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>> madawa...@cse.mrt.ac.lk >>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>> <http://madawas.tumblr.com/> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Thanks & regards, >>>>>> Nirmal >>>>>> >>>>>> Team Lead - WSO2 Machine Learner >>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>> Mobile: +94715779733 >>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Dev mailing list >>>>>> Dev@wso2.org >>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Niranda Perera* >>>>> Software Engineer, WSO2 Inc. >>>>> Mobile: +94-71-554-8430 >>>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>>> https://pythagoreanscript.wordpress.com/ >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Thanks & regards, >>>> Nirmal >>>> >>>> Team Lead - WSO2 Machine Learner >>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>> Mobile: +94715779733 >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>>> >>>> >>> >>> >>> -- >>> >>> *_________________________________**Madawa Soysa* >>> >>> Undergraduate, >>> >>> Department of Computer Science and Engineering, >>> >>> University of Moratuwa. >>> >>> >>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>> madawa...@cse.mrt.ac.lk >>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>> >> >> >> >> -- >> >> *_________________________________**Madawa Soysa* >> >> Undergraduate, >> >> Department of Computer Science and Engineering, >> >> University of Moratuwa. >> >> >> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >> madawa...@cse.mrt.ac.lk >> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >> > > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- *_________________________________**Madawa Soysa* Undergraduate, Department of Computer Science and Engineering, University of Moratuwa. Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: madawa...@cse.mrt.ac.lk LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/>
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
