You added them to repository/components/lib right? Can you start the server in OSGi mode (./wso2server.sh -DosgiConsole ) and issue following command in OSGi console;
osgi> p parquet.hadoop.metadata On Wed, Aug 26, 2015 at 8:32 AM, Madawa Soysa <madawa...@cse.mrt.ac.lk> wrote: > Hi Nirmal, > > Not actually. I have added parquet-common, parquet-encoding, > parquet-column, parquet-hadoop as mentioned at > https://github.com/Parquet/parquet-mr > <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2FParquet%2Fparquet-mr&sa=D&sntz=1&usg=AFQjCNEHFd3g0eqXClZCkN35QgFYl0FVww> > . But still get the following exception even though the > class parquet.hadoop.metadata.CompressionCodecName is there in the > parquet-hadoop.jar > > java.lang.ClassNotFoundException: > parquet.hadoop.metadata.CompressionCodecName cannot be found by > spark-sql_2.10_1.4.1.wso2v1 > > I kept trying with different versions, still the issue is there. > > On 26 August 2015 at 08:27, Nirmal Fernando <nir...@wso2.com> wrote: > >> Madawa, hope the issue is resolved now after the instructions given >> offline. >> >> On Mon, Aug 24, 2015 at 8:34 PM, Madawa Soysa <madawa...@cse.mrt.ac.lk> >> wrote: >> >>> I have tested the model.save() with a simple Java program. It works >>> fine. >>> >>> I have noticed that scala-library:2.11.6 is a dependency of >>> spark:spark-core_2.11:1.4.1 [1] >>> <http://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-core_2.11%7C1.4.1%7Cjar> >>> In ML scala version is explicitly specified as 2.10.4. Is there a specific >>> reason to use scala 2.10.4? I guess this version incompatibility could be >>> the reason for this issue. >>> >>> [1] - >>> http://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-core_2.11%7C1.4.1%7Cjar >>> >>> On 24 August 2015 at 10:24, Nirmal Fernando <nir...@wso2.com> wrote: >>> >>>> Madawa, seems this is a Spark issue :-( can you try a simple Java >>>> program and see whether model.save() works? >>>> >>>> On Sat, Aug 22, 2015 at 8:19 AM, Madawa Soysa <madawa...@cse.mrt.ac.lk> >>>> wrote: >>>> >>>>> Hi Nirmal, >>>>> >>>>> outPath is correct and the directory gets created, but the process >>>>> becomes idle after that. Attached the only part that was written to a >>>>> file. >>>>> >>>>> Also the method doesn't throw an exception as well. >>>>> >>>>> On 21 August 2015 at 21:31, Nirmal Fernando <nir...@wso2.com> wrote: >>>>> >>>>>> Hi Madawa, >>>>>> >>>>>> According to Spark API [1], outPath shouldn't be exist. >>>>>> >>>>>> [1] >>>>>> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.scala#L200 >>>>>> >>>>>> On Fri, Aug 21, 2015 at 1:59 PM, Niranda Perera <nira...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> I don't think it's correct. Scala version is 2.10.4 even in the mvn >>>>>>> repo >>>>>>> >>>>>>> On Fri, Aug 21, 2015, 13:46 Madawa Soysa <madawa...@cse.mrt.ac.lk> >>>>>>> wrote: >>>>>>> >>>>>>>> Also I asked this question in StackOverflow[1] >>>>>>>> <http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java> >>>>>>>> and there they have mentioned a version incompatibility between Scala >>>>>>>> and >>>>>>>> Spark versions >>>>>>>> >>>>>>>> [1] - >>>>>>>> http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java >>>>>>>> >>>>>>>> On 21 August 2015 at 13:31, Madawa Soysa <madawa...@cse.mrt.ac.lk> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Yes path is valid, I explicitly set the path here from the >>>>>>>>> MLModelHandler persistModel method. >>>>>>>>> >>>>>>>>> On 21 August 2015 at 10:26, Nirmal Fernando <nir...@wso2.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Aug 20, 2015 at 9:21 PM, Madawa Soysa < >>>>>>>>>> madawa...@cse.mrt.ac.lk> wrote: >>>>>>>>>> >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> There an issue with serializing Spark's MatrixFactorizationModel >>>>>>>>>>> object. The object contains a huge RDD and as I have read in many >>>>>>>>>>> blogs, >>>>>>>>>>> this model cannot be serialized as a java object. Therefore when >>>>>>>>>>> retrieving >>>>>>>>>>> the model I get the same exception as above; >>>>>>>>>>> >>>>>>>>>>> *Caused by: java.lang.ClassNotFoundException: >>>>>>>>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>>>>>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>>>>>>>> >>>>>>>>>>> I have asked this question in Spark mailing lists and they >>>>>>>>>>> recommended me to use the built in save and load functions other >>>>>>>>>>> than using >>>>>>>>>>> Java serializing. So I have used following method to persist the >>>>>>>>>>> model, >>>>>>>>>>> >>>>>>>>>>> model.save(MLCoreServiceValueHolder.*getInstance()*.getSparkContext().sc(), >>>>>>>>>>> outPath);[1] >>>>>>>>>>> <https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9> >>>>>>>>>>> >>>>>>>>>>> Then nothing happens when this line executes. No error is thrown >>>>>>>>>>> as well. Any solution for this? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Can you print outPath and see whether it's a valid file path? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [1] - >>>>>>>>>>> https://github.com/madawas/carbon-ml/commit/3700d3ed5915b0ad3b679bc0d9eb2611608463e9 >>>>>>>>>>> >>>>>>>>>>> On 16 August 2015 at 18:06, Madawa Soysa < >>>>>>>>>>> madawa...@cse.mrt.ac.lk> wrote: >>>>>>>>>>> >>>>>>>>>>>> Yes I was able to resolve the issue by removing RDD fields from >>>>>>>>>>>> the SummaryModel object as @Mano pointed out. Still I have the same >>>>>>>>>>>> exception when retrieving the model. Trying to fix that issue. >>>>>>>>>>>> >>>>>>>>>>>> On 14 August 2015 at 10:43, Nirmal Fernando <nir...@wso2.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Thanks Niranda, this doc is useful. >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Aug 14, 2015 at 10:36 AM, Niranda Perera < >>>>>>>>>>>>> nira...@wso2.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> From what I know, OneToOneDependancy come into play when >>>>>>>>>>>>>> spark tries to create the RDD dependency tree. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Just thought of sharing that. this would be a good resource >>>>>>>>>>>>>> [1] :-) >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] >>>>>>>>>>>>>> https://databricks-training.s3.amazonaws.com/slides/advanced-spark-training.pdf >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Aug 13, 2015 at 12:09 AM, Nirmal Fernando < >>>>>>>>>>>>>> nir...@wso2.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> What is *org.apache.spark.OneToOneDependency ? Is it >>>>>>>>>>>>>>> something you use?* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Aug 12, 2015 at 11:30 PM, Madawa Soysa < >>>>>>>>>>>>>>> madawa...@cse.mrt.ac.lk> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I created a model summary in order to show the model data >>>>>>>>>>>>>>>> in the analysis.jag page. >>>>>>>>>>>>>>>> But when refreshing the page after building the model I get >>>>>>>>>>>>>>>> the following error. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> org.wso2.carbon.ml.core.exceptions.MLAnalysisHandlerException: >>>>>>>>>>>>>>>> An error has occurred while extracting all the models of >>>>>>>>>>>>>>>> analysis id: 13 >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:245) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.wso2.carbon.ml.rest.api.AnalysisApiV10.getAllModelsOfAnalysis(AnalysisApiV10.java:517) >>>>>>>>>>>>>>>> Caused by: >>>>>>>>>>>>>>>> org.wso2.carbon.ml.database.exceptions.DatabaseHandlerException: >>>>>>>>>>>>>>>> An error >>>>>>>>>>>>>>>> has occurred while extracting all the models of analysis id: 13 >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1797) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.wso2.carbon.ml.core.impl.MLAnalysisHandler.getAllModelsOfAnalysis(MLAnalysisHandler.java:243) >>>>>>>>>>>>>>>> ... 52 more >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> *Caused by: java.lang.ClassNotFoundException: >>>>>>>>>>>>>>>> org.apache.spark.OneToOneDependency cannot be found by >>>>>>>>>>>>>>>> org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a3965* >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.wso2.carbon.ml.database.util.MLDBUtil.getModelSummaryFromInputStream(MLDBUtil.java:54) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> org.wso2.carbon.ml.database.internal.MLDatabaseService.getAllModels(MLDatabaseService.java:1790) >>>>>>>>>>>>>>>> ... 53 more >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I guess there is an error in persistence of the model >>>>>>>>>>>>>>>> summary object, what should be the cause for this error? >>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>> <https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22> >>>>>>>>>>>>>>>> contains >>>>>>>>>>>>>>>> the commit which I introduced the model summary. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [1] - >>>>>>>>>>>>>>>> https://github.com/madawas/carbon-ml/commit/987c799231dad2bab6f4046df7acc672d0564f22 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Undergraduate, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> University of Moratuwa. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>>>>>>>>> madawa...@cse.mrt.ac.lk >>>>>>>>>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks & regards, >>>>>>>>>>>>>>> Nirmal >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>>>> Dev@wso2.org >>>>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> *Niranda Perera* >>>>>>>>>>>>>> Software Engineer, WSO2 Inc. >>>>>>>>>>>>>> Mobile: +94-71-554-8430 >>>>>>>>>>>>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>>>>>>>>>>>> https://pythagoreanscript.wordpress.com/ >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks & regards, >>>>>>>>>>>>> Nirmal >>>>>>>>>>>>> >>>>>>>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>>>>> >>>>>>>>>>>> Undergraduate, >>>>>>>>>>>> >>>>>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>>>>> >>>>>>>>>>>> University of Moratuwa. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>>>>> madawa...@cse.mrt.ac.lk >>>>>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>>>> >>>>>>>>>>> Undergraduate, >>>>>>>>>>> >>>>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>>>> >>>>>>>>>>> University of Moratuwa. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>>>> madawa...@cse.mrt.ac.lk >>>>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Thanks & regards, >>>>>>>>>> Nirmal >>>>>>>>>> >>>>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>>> Mobile: +94715779733 >>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>>> >>>>>>>>> Undergraduate, >>>>>>>>> >>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>> >>>>>>>>> University of Moratuwa. >>>>>>>>> >>>>>>>>> >>>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>>> madawa...@cse.mrt.ac.lk >>>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>>> <http://madawas.tumblr.com/> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> *_________________________________**Madawa Soysa* >>>>>>>> >>>>>>>> Undergraduate, >>>>>>>> >>>>>>>> Department of Computer Science and Engineering, >>>>>>>> >>>>>>>> University of Moratuwa. >>>>>>>> >>>>>>>> >>>>>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>>>>> madawa...@cse.mrt.ac.lk >>>>>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>>>>> <https://twitter.com/madawa_rc> | Tumblr >>>>>>>> <http://madawas.tumblr.com/> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Thanks & regards, >>>>>> Nirmal >>>>>> >>>>>> Team Lead - WSO2 Machine Learner >>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>> Mobile: +94715779733 >>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> *_________________________________**Madawa Soysa* >>>>> >>>>> Undergraduate, >>>>> >>>>> Department of Computer Science and Engineering, >>>>> >>>>> University of Moratuwa. >>>>> >>>>> >>>>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>>>> madawa...@cse.mrt.ac.lk >>>>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>>>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Thanks & regards, >>>> Nirmal >>>> >>>> Team Lead - WSO2 Machine Learner >>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>> Mobile: +94715779733 >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>>> >>>> >>> >>> >>> -- >>> >>> *_________________________________**Madawa Soysa* >>> >>> Undergraduate, >>> >>> Department of Computer Science and Engineering, >>> >>> University of Moratuwa. >>> >>> >>> Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: >>> madawa...@cse.mrt.ac.lk >>> LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter >>> <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > > *_________________________________**Madawa Soysa* > > Undergraduate, > > Department of Computer Science and Engineering, > > University of Moratuwa. > > > Mobile: +94 71 461 6050 <%2B94%2075%20812%200726> | Email: > madawa...@cse.mrt.ac.lk > LinkedIn <http://lk.linkedin.com/in/madawasoysa> | Twitter > <https://twitter.com/madawa_rc> | Tumblr <http://madawas.tumblr.com/> > -- Thanks & regards, Nirmal Team Lead - WSO2 Machine Learner Associate Technical Lead - Data Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
