Hi Madawa, According to Spark API [1], outPath shouldn't be exist.
[1] https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.scala#L200 On Fri, Aug 21, 2015 at 1:59 PM, Niranda Perera <[email protected]> wrote: > I don't think it's correct. Scala version is 2.10.4 even in the mvn repo > > On Fri, Aug 21, 2015, 13:46 Madawa Soysa <[email protected]> wrote: > >> Also I asked this question in StackOverflow[1] >> <http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java> >> and there they have mentioned a version incompatibility between Scala and >> Spark versions >> >> [1] - >> http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java >> >> On 21 August 2015 at 13:31, Madawa Soysa <[email protected]> wrote: >> >>> Yes path is valid, I explicitly set the path here from the >>> MLModelHandler persistModel method. >>> >>> On 21 August 2015 at 10:26, Nirmal Fernando <[email protected]> wrote: >>> >>>> >>>> >>>> On Thu, Aug 20, 2015 at 9:21 PM, Madawa Soysa <[email protected]> >>>> wrote: >>>> >>>>> Hi All, >>>>> >>>>> There an issue with serializing Spark's MatrixFactorizationModel >>>>> object. The object contains a huge RDD and as I have read in many blogs, >>>>> this model cannot be serialized as a java object. Therefore when >>>>> retrieving >>>>> the model I get the same exception as above; >>>>> >>>>> *Caused by: java.lang.ClassNotFoundException: >>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>> >>>>> I have asked this question in Spark mailing lists and they recommended >>>>> me to use the built in save and load functions other than using Java >>>>> serializing. So I have used following method to persist the model, >>>>> >>>>> model.save(MLCoreServiceValueHolder.*getInstance()*.getSparkContext().sc(), >>>>> outPath);[1] >>>>> <https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9> >>>>> >>>>> Then nothing happens when this line executes. No error is thrown as >>>>> well. Any solution for this? >>>>> >>>> >>>> Can you print outPath and see whether it's a valid file path? >>>> >>>> >>>>> >>>>> [1] - >>>>> https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9 >>>>> >>>>> On 16 August 2015 at 18:06, Madawa Soysa <[email protected]> >>>>> wrote: >>>>> >>>>>> Yes I was able to resolve the issue by removing RDD fields from the >>>>>> SummaryModel object as @Mano pointed out. Still I have the same exception >>>>>> when retrieving the model. Trying to fix that issue. >>>>>> >>>>>> On 14 August 2015 at 10:43, Nirmal Fernando <[email protected]> wrote: >>>>>> >>>>>>> Thanks Niranda, this doc is useful. >>>>>>> >>>>>>> On Fri, Aug 14, 2015 at 10:36 AM, Niranda Perera <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> From what I know, OneToOneDependancy come into play when spark >>>>>>>> tries to create the RDD dependency tree. >>>>>>>> >>>>>>>> Just thought of sharing that. this would be a good resource [1] :-) >>>>>>>> >>>>>>>> >>>>>>>> [1] >>>>>>>> https://databricks-training.s3.amazonaws.com/slides/advanced-spark-training.pdf >>>>>>>> >>>>>>>> On Thu, Aug 13, 2015 at 12:09 AM, Nirmal Fernando <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> What is *org.apache.spark.OneToOneDependency ? Is it something >>>>>>>>> you use?* >>>>>>>>> >>>>>>>>> On Wed, Aug 12, 2015 at 11:30 PM, Madawa Soysa < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I created a model summary in order to show the model data in the >>>>>>>>>> analysis.jag page. >>>>>>>>>> But when refreshing the page after building the model I get the >>>>>>>>>> following error. >>>>>>>>>> >>>>>>>>>> org.wso2.carbon.ml.core.exceptions.MLAnalysisHandlerException: >>>>>>>>>> An error has occurred while extracting all the models of analysis >>>>>>>>>> id: 13 >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:245) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.ml.rest.api.AnalysisApiV10.getAllModelsOfAnalysis(AnalysisApiV10.java:517) >>>>>>>>>> Caused by: >>>>>>>>>> org.wso2.carbon.ml.database.exceptions.DatabaseHandlerException: An >>>>>>>>>> error >>>>>>>>>> has occurred while extracting all the models of analysis id: 13 >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1797) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:243) >>>>>>>>>> ... 52 more >>>>>>>>>> >>>>>>>>>> *Caused by: java.lang.ClassNotFoundException: >>>>>>>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>>>>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.ml.database.util.MLDBUtil.getModelSummaryFromInputStream(MLDBUtil.java:54) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1790) >>>>>>>>>> ... 53 more >>>>>>>>>> >>>>>>>>>> I guess there is an error in persistence of the model summary >>>>>>>>>> object, what should be the cause for this error? [1] >>>>>>>>>> <https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22> >>>>>>>>>> contains >>>>>>>>>> the commit which I introduced the model summary. >>>>>>>>>> >>>>>>>>>> [1] - >>>>>>>>>> https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22 >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>>> >>>>>>>>>> Undergraduate, >>>>>>>>>> >>>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>>> >>>>>>>>>> University of Moratuwa. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>>> [email protected] >>>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Thanks & regards, >>>>>>>>> Nirmal >>>>>>>>> >>>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>> Mobile: +94715779733 >>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Dev mailing list >>>>>>>>> [email protected] >>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Niranda Perera* >>>>>>>> Software Engineer, WSO2 Inc. >>>>>>>> Mobile: +94-71-554-8430 >>>>>>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>>>>>> https://pythagoreanscript.wordpress.com/ >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Thanks & regards, >>>>>>> Nirmal >>>>>>> >>>>>>> Team Lead - WSO2 Machine Learner >>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>> Mobile: +94715779733 >>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> *_________________________________**Madawa Soysa* >>>>>> >>>>>> Undergraduate, >>>>>> >>>>>> Department of Computer Science and Engineering, >>>>>> >>>>>> University of Moratuwa. >>>>>> >>>>>> >>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>> [email protected] >>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> *_________________________________**Madawa Soysa* >>>>> >>>>> Undergraduate, >>>>> >>>>> Department of Computer Science and Engineering, >>>>> >>>>> University of Moratuwa. >>>>> >>>>> >>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>> [email protected] >>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Thanks & regards, >>>> Nirmal >>>> >>>> Team Lead - WSO2 Machine Learner >>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>> Mobile: +94715779733 >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>>> >>>> >>> >>> >>> -- >>> >>> *_________________________________**Madawa Soysa* >>> >>> Undergraduate, >>> >>> Department of Computer Science and Engineering, >>> >>> University of Moratuwa. >>> >>> >>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>> [email protected] >>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>> >> >> >> >> -- >> >> *_________________________________**Madawa Soysa* >> >> Undergraduate, >> >> Department of Computer Science and Engineering, >> >> University of Moratuwa. >> >> >> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >> [email protected] >> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >> > -- Thanks & regards, Nirmal Team Lead - WSO2 Machine Learner Associate Technical Lead - Data Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
