Madawa, hope the issue is resolved now after the instructions given offline.
On Mon, Aug 24, 2015 at 8:34 PM, Madawa Soysa <madawa...@cse.mrt.ac.lk> wrote: > I have tested the model.save() with a simple Java program. It works fine. > > I have noticed that scala-library:2.11.6 is a dependency of > spark:spark-core_2.11:1.4.1 [1] > <http://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-core_2.11%7C1.4.1%7Cjar> > In ML scala version is explicitly specified as 2.10.4. Is there a specific > reason to use scala 2.10.4? I guess this version incompatibility could be > the reason for this issue. > > [1] - > http://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-core_2.11%7C1.4.1%7Cjar > > On 24 August 2015 at 10:24, Nirmal Fernando <nir...@wso2.com> wrote: > >> Madawa, seems this is a Spark issue :-( can you try a simple Java program >> and see whether model.save() works? >> >> On Sat, Aug 22, 2015 at 8:19 AM, Madawa Soysa <madawa...@cse.mrt.ac.lk> >> wrote: >> >>> Hi Nirmal, >>> >>> outPath is correct and the directory gets created, but the process >>> becomes idle after that. Attached the only part that was written to a file. >>> >>> Also the method doesn't throw an exception as well. >>> >>> On 21 August 2015 at 21:31, Nirmal Fernando <nir...@wso2.com> wrote: >>> >>>> Hi Madawa, >>>> >>>> According to Spark API [1], outPath shouldn't be exist. >>>> >>>> [1] >>>> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.scala#L200 >>>> >>>> On Fri, Aug 21, 2015 at 1:59 PM, Niranda Perera <nira...@wso2.com> >>>> wrote: >>>> >>>>> I don't think it's correct. Scala version is 2.10.4 even in the mvn >>>>> repo >>>>> >>>>> On Fri, Aug 21, 2015, 13:46 Madawa Soysa <madawa...@cse.mrt.ac.lk> >>>>> wrote: >>>>> >>>>>> Also I asked this question in StackOverflow[1] >>>>>> <http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java> >>>>>> and there they have mentioned a version incompatibility between Scala and >>>>>> Spark versions >>>>>> >>>>>> [1] - >>>>>> http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java >>>>>> >>>>>> On 21 August 2015 at 13:31, Madawa Soysa <madawa...@cse.mrt.ac.lk> >>>>>> wrote: >>>>>> >>>>>>> Yes path is valid, I explicitly set the path here from the >>>>>>> MLModelHandler persistModel method. >>>>>>> >>>>>>> On 21 August 2015 at 10:26, Nirmal Fernando <nir...@wso2.com> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Aug 20, 2015 at 9:21 PM, Madawa Soysa < >>>>>>>> madawa...@cse.mrt.ac.lk> wrote: >>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> There an issue with serializing Spark's MatrixFactorizationModel >>>>>>>>> object. The object contains a huge RDD and as I have read in many >>>>>>>>> blogs, >>>>>>>>> this model cannot be serialized as a java object. Therefore when >>>>>>>>> retrieving >>>>>>>>> the model I get the same exception as above; >>>>>>>>> >>>>>>>>> *Caused by: java.lang.ClassNotFoundException: >>>>>>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>>>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>>>>>> >>>>>>>>> I have asked this question in Spark mailing lists and they >>>>>>>>> recommended me to use the built in save and load functions other than >>>>>>>>> using >>>>>>>>> Java serializing. So I have used following method to persist the >>>>>>>>> model, >>>>>>>>> >>>>>>>>> model.save(MLCoreServiceValueHolder.*getInstance()*.getSparkContext().sc(), >>>>>>>>> outPath);[1] >>>>>>>>> <https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9> >>>>>>>>> >>>>>>>>> Then nothing happens when this line executes. No error is thrown >>>>>>>>> as well. Any solution for this? >>>>>>>>> >>>>>>>> >>>>>>>> Can you print outPath and see whether it's a valid file path? >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> [1] - >>>>>>>>> https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9 >>>>>>>>> >>>>>>>>> On 16 August 2015 at 18:06, Madawa Soysa <madawa...@cse.mrt.ac.lk> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Yes I was able to resolve the issue by removing RDD fields from >>>>>>>>>> the SummaryModel object as @Mano pointed out. Still I have the same >>>>>>>>>> exception when retrieving the model. Trying to fix that issue. >>>>>>>>>> >>>>>>>>>> On 14 August 2015 at 10:43, Nirmal Fernando <nir...@wso2.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks Niranda, this doc is useful. >>>>>>>>>>> >>>>>>>>>>> On Fri, Aug 14, 2015 at 10:36 AM, Niranda Perera < >>>>>>>>>>> nira...@wso2.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> From what I know, OneToOneDependancy come into play when spark >>>>>>>>>>>> tries to create the RDD dependency tree. >>>>>>>>>>>> >>>>>>>>>>>> Just thought of sharing that. this would be a good resource [1] >>>>>>>>>>>> :-) >>>>>>>>>>>> >>>>>>>>>>>> [1] >>>>>>>>>>>> https://databricks-training.s3.amazonaws.com/slides/advanced-spark-training.pdf >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Aug 13, 2015 at 12:09 AM, Nirmal Fernando < >>>>>>>>>>>> nir...@wso2.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> What is *org.apache.spark.OneToOneDependency ? Is it >>>>>>>>>>>>> something you use?* >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Aug 12, 2015 at 11:30 PM, Madawa Soysa < >>>>>>>>>>>>> madawa...@cse.mrt.ac.lk> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I created a model summary in order to show the model data in >>>>>>>>>>>>>> the analysis.jag page. >>>>>>>>>>>>>> But when refreshing the page after building the model I get >>>>>>>>>>>>>> the following error. >>>>>>>>>>>>>> >>>>>>>>>>>>>> org.wso2.carbon.ml.core.exceptions.MLAnalysisHandlerException: >>>>>>>>>>>>>> An error has occurred while extracting all the models of >>>>>>>>>>>>>> analysis id: 13 >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:245) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.ml.rest.api.AnalysisApiV10.getAllModelsOfAnalysis(AnalysisApiV10.java:517) >>>>>>>>>>>>>> Caused by: >>>>>>>>>>>>>> org.wso2.carbon.ml.database.exceptions.DatabaseHandlerException: >>>>>>>>>>>>>> An error >>>>>>>>>>>>>> has occurred while extracting all the models of analysis id: 13 >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1797) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:243) >>>>>>>>>>>>>> ... 52 more >>>>>>>>>>>>>> >>>>>>>>>>>>>> *Caused by: java.lang.ClassNotFoundException: >>>>>>>>>>>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>>>>>>>>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.ml.database.util.MLDBUtil.getModelSummaryFromInputStream(MLDBUtil.java:54) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1790) >>>>>>>>>>>>>> ... 53 more >>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess there is an error in persistence of the model summary >>>>>>>>>>>>>> object, what should be the cause for this error? [1] >>>>>>>>>>>>>> <https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22> >>>>>>>>>>>>>> contains >>>>>>>>>>>>>> the commit which I introduced the model summary. >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] - >>>>>>>>>>>>>> https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22 >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>>>>>>> >>>>>>>>>>>>>> Undergraduate, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>>>>>>> >>>>>>>>>>>>>> University of Moratuwa. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>>>>>>> madawa...@cse.mrt.ac.lk >>>>>>>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks & regards, >>>>>>>>>>>>> Nirmal >>>>>>>>>>>>> >>>>>>>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>> Dev@wso2.org >>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> *Niranda Perera* >>>>>>>>>>>> Software Engineer, WSO2 Inc. >>>>>>>>>>>> Mobile: +94-71-554-8430 >>>>>>>>>>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>>>>>>>>>> https://pythagoreanscript.wordpress.com/ >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> Thanks & regards, >>>>>>>>>>> Nirmal >>>>>>>>>>> >>>>>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>>> >>>>>>>>>> Undergraduate, >>>>>>>>>> >>>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>>> >>>>>>>>>> University of Moratuwa. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>>> madawa...@cse.mrt.ac.lk >>>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>> >>>>>>>>> Undergraduate, >>>>>>>>> >>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>> >>>>>>>>> University of Moratuwa. >>>>>>>>> >>>>>>>>> >>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>> madawa...@cse.mrt.ac.lk >>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Thanks & regards, >>>>>>>> Nirmal >>>>>>>> >>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>> Mobile: +94715779733 >>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> *_________________________________**Madawa Soysa* >>>>>>> >>>>>>> Undergraduate, >>>>>>> >>>>>>> Department of Computer Science and Engineering, >>>>>>> >>>>>>> University of Moratuwa. >>>>>>> >>>>>>> >>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>> madawa...@cse.mrt.ac.lk >>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>> <http://madawas.tumblr.com/> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> *_________________________________**Madawa Soysa* >>>>>> >>>>>> Undergraduate, >>>>>> >>>>>> Department of Computer Science and Engineering, >>>>>> >>>>>> University of Moratuwa. >>>>>> >>>>>> >>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>> madawa...@cse.mrt.ac.lk >>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Thanks & regards, >>>> Nirmal >>>> >>>> Team Lead - WSO2 Machine Learner >>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>> Mobile: +94715779733 >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>>> >>>> >>> >>> >>> -- >>> >>> *_________________________________**Madawa Soysa* >>> >>> Undergraduate, >>> >>> Department of Computer Science and Engineering, >>> >>> University of Moratuwa. >>> >>> >>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>> madawa...@cse.mrt.ac.lk >>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > > *_________________________________**Madawa Soysa* > > Undergraduate, > > Department of Computer Science and Engineering, > > University of Moratuwa. > > > Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: > madawa...@cse.mrt.ac.lk > LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter > <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> > -- Thanks & regards, Nirmal Team Lead - WSO2 Machine Learner Associate Technical Lead - Data Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
