Re: [Dev] [ML] Error when deserializing model summary

I fixed this issue by following commit: https://github.com/wso2/carbon-ml/commit/d94570ed5bab36af60606d06233f1d3eecf58558#diff-7bee5ca92183e001202ee9d8b80918ca On Wed, Aug 26, 2015 at 9:23 AM, Nirmal Fernando wrote: > Shall we try adding an all-in-one parquet jar? Can you

Re: [Dev] [ML] Error when deserializing model summary

Hi Nirmal, Not actually. I have added parquet-common, parquet-encoding, parquet-column, parquet-hadoop as mentioned at https://github.com/Parquet/parquet-mr https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2FParquet%2Fparquet-mrsa=Dsntz=1usg=AFQjCNEHFd3g0eqXClZCkN35QgFYl0FVww . But still get

Re: [Dev] [ML] Error when deserializing model summary

So it's coming from 2 bundles? Can we get rid of 1 and see? On Wed, Aug 26, 2015 at 8:40 AM, Madawa Soysa madawa...@cse.mrt.ac.lk wrote: Gives following output. parquet.hadoop.metadata; version=0.0.0parquet_common_1.6.0_1.0.0 [321] parquet.hadoop.metadata;

Re: [Dev] [ML] Error when deserializing model summary

Madawa, hope the issue is resolved now after the instructions given offline. On Mon, Aug 24, 2015 at 8:34 PM, Madawa Soysa madawa...@cse.mrt.ac.lk wrote: I have tested the model.save() with a simple Java program. It works fine. I have noticed that scala-library:2.11.6 is a dependency of

Re: [Dev] [ML] Error when deserializing model summary

Gives following output. parquet.hadoop.metadata; version=0.0.0parquet_common_1.6.0_1.0.0 [321] parquet.hadoop.metadata; version=0.0.0parquet_hadoop_1.6.0_1.0.0 [323] On 26 August 2015 at 08:35, Nirmal Fernando nir...@wso2.com wrote: You added them to repository/components/lib right? Can you

Re: [Dev] [ML] Error when deserializing model summary

Both bundles are required. I tried removing one at a time and checked. On 26 August 2015 at 08:43, Nirmal Fernando nir...@wso2.com wrote: So it's coming from 2 bundles? Can we get rid of 1 and see? On Wed, Aug 26, 2015 at 8:40 AM, Madawa Soysa madawa...@cse.mrt.ac.lk wrote: Gives following

Re: [Dev] [ML] Error when deserializing model summary

Shall we try adding an all-in-one parquet jar? Can you check whether there's one? http://mvnrepository.com/artifact/com.twitter/parquet/1.6.0 On Wed, Aug 26, 2015 at 8:55 AM, Madawa Soysa madawa...@cse.mrt.ac.lk wrote: Both bundles are required. I tried removing one at a time and checked. On

Re: [Dev] [ML] Error when deserializing model summary

You added them to repository/components/lib right? Can you start the server in OSGi mode (./wso2server.sh -DosgiConsole ) and issue following command in OSGi console; osgi p parquet.hadoop.metadata On Wed, Aug 26, 2015 at 8:32 AM, Madawa Soysa madawa...@cse.mrt.ac.lk wrote: Hi Nirmal, Not

Re: [Dev] [ML] Error when deserializing model summary

I have tested the model.save() with a simple Java program. It works fine. I have noticed that scala-library:2.11.6 is a dependency of spark:spark-core_2.11:1.4.1 [1] http://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-core_2.11%7C1.4.1%7Cjar In ML scala version is explicitly

Re: [Dev] [ML] Error when deserializing model summary

Madawa, seems this is a Spark issue :-( can you try a simple Java program and see whether model.save() works? On Sat, Aug 22, 2015 at 8:19 AM, Madawa Soysa madawa...@cse.mrt.ac.lk wrote: Hi Nirmal, outPath is correct and the directory gets created, but the process becomes idle after that.

Re: [Dev] [ML] Error when deserializing model summary

Hi Nirmal, outPath is correct and the directory gets created, but the process becomes idle after that. Attached the only part that was written to a file. Also the method doesn't throw an exception as well. On 21 August 2015 at 21:31, Nirmal Fernando nir...@wso2.com wrote: Hi Madawa,

Re: [Dev] [ML] Error when deserializing model summary

Hi Madawa, According to Spark API [1], outPath shouldn't be exist. [1] https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.scala#L200 On Fri, Aug 21, 2015 at 1:59 PM, Niranda Perera nira...@wso2.com wrote: I don't

Re: [Dev] [ML] Error when deserializing model summary

I don't think it's correct. Scala version is 2.10.4 even in the mvn repo On Fri, Aug 21, 2015, 13:46 Madawa Soysa madawa...@cse.mrt.ac.lk wrote: Also I asked this question in StackOverflow[1]

Re: [Dev] [ML] Error when deserializing model summary

Also I asked this question in StackOverflow[1] http://stackoverflow.com/questions/32048618/how-to-serialize-apache-sparks-matrixfactorizationmodel-in-java and there they have mentioned a version incompatibility between Scala and Spark versions [1] -

Re: [Dev] [ML] Error when deserializing model summary

Yes path is valid, I explicitly set the path here from the MLModelHandler persistModel method. On 21 August 2015 at 10:26, Nirmal Fernando nir...@wso2.com wrote: On Thu, Aug 20, 2015 at 9:21 PM, Madawa Soysa madawa...@cse.mrt.ac.lk wrote: Hi All, There an issue with serializing Spark's

Re: [Dev] [ML] Error when deserializing model summary

On Thu, Aug 20, 2015 at 9:21 PM, Madawa Soysa madawa...@cse.mrt.ac.lk wrote: Hi All, There an issue with serializing Spark's MatrixFactorizationModel object. The object contains a huge RDD and as I have read in many blogs, this model cannot be serialized as a java object. Therefore when

Re: [Dev] [ML] Error when deserializing model summary

Hi All, There an issue with serializing Spark's MatrixFactorizationModel object. The object contains a huge RDD and as I have read in many blogs, this model cannot be serialized as a java object. Therefore when retrieving the model I get the same exception as above; *Caused by:

Re: [Dev] [ML] Error when deserializing model summary

Yes I was able to resolve the issue by removing RDD fields from the SummaryModel object as @Mano pointed out. Still I have the same exception when retrieving the model. Trying to fix that issue. On 14 August 2015 at 10:43, Nirmal Fernando nir...@wso2.com wrote: Thanks Niranda, this doc is

Re: [Dev] [ML] Error when deserializing model summary

Thanks Niranda, this doc is useful. On Fri, Aug 14, 2015 at 10:36 AM, Niranda Perera nira...@wso2.com wrote: From what I know, OneToOneDependancy come into play when spark tries to create the RDD dependency tree. Just thought of sharing that. this would be a good resource [1] :-) [1]

Re: [Dev] [ML] Error when deserializing model summary

Were you able to resolve the issue? On Thu, Aug 13, 2015 at 8:59 AM, Nirmal Fernando nir...@wso2.com wrote: @Mano can you help Madawa? On Thu, Aug 13, 2015 at 8:56 AM, Madawa Soysa madawa...@cse.mrt.ac.lk wrote: No that's not something I use. On 13 August 2015 at 00:09, Nirmal Fernando

Re: [Dev] [ML] Error when deserializing model summary

From what I know, OneToOneDependancy come into play when spark tries to create the RDD dependency tree. Just thought of sharing that. this would be a good resource [1] :-) [1] https://databricks-training.s3.amazonaws.com/slides/advanced-spark-training.pdf On Thu, Aug 13, 2015 at 12:09 AM,

Re: [Dev] [ML] Error when deserializing model summary

@Mano can you help Madawa? On Thu, Aug 13, 2015 at 8:56 AM, Madawa Soysa madawa...@cse.mrt.ac.lk wrote: No that's not something I use. On 13 August 2015 at 00:09, Nirmal Fernando nir...@wso2.com wrote: What is *org.apache.spark.OneToOneDependency ? Is it something you use?* On Wed, Aug

Re: [Dev] [ML] Error when deserializing model summary

No that's not something I use. On 13 August 2015 at 00:09, Nirmal Fernando nir...@wso2.com wrote: What is *org.apache.spark.OneToOneDependency ? Is it something you use?* On Wed, Aug 12, 2015 at 11:30 PM, Madawa Soysa madawa...@cse.mrt.ac.lk wrote: Hi, I created a model summary in order

Re: [Dev] [ML] Error when deserializing model summary

What is *org.apache.spark.OneToOneDependency ? Is it something you use?* On Wed, Aug 12, 2015 at 11:30 PM, Madawa Soysa madawa...@cse.mrt.ac.lk wrote: Hi, I created a model summary in order to show the model data in the analysis.jag page. But when refreshing the page after building the

[Dev] [ML] Error when deserializing model summary

Hi, I created a model summary in order to show the model data in the analysis.jag page. But when refreshing the page after building the model I get the following error. org.wso2.carbon.ml.core.exceptions.MLAnalysisHandlerException: An error has occurred while extracting all the models of