[ https://issues.apache.org/jira/browse/SPARK-21341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078987#comment-16078987 ]
Yan Facai (颜发才) edited comment on SPARK-21341 at 7/8/17 6:28 AM: ----------------------------------------------------------------- Hi, [~zsellami]. I guess that since the wordVectors is mllib model in fact, which might be removed in the future, so it is marked private and transient. More interestingly, wordVectors are saved in data folder as dataframe, see: {code} 336 override protected def saveImpl(path: String): Unit = { 337 DefaultParamsWriter.saveMetadata(instance, path, sc) 338 339 val wordVectors = instance.wordVectors.getVectors 340 val dataSeq = wordVectors.toSeq.map { case (word, vector) => Data(word, vector) } 341 val dataPath = new Path(path, "data").toString 342 sparkSession.createDataFrame(dataSeq) 343 .repartition(calculateNumberOfPartitions) 344 .write 345 .parquet(dataPath) 346 } {code} In all, developers indeed take a try to save the wordVector, however it seems to break in pipeline as you said. So, could you give an example code to reproduce the bug? I'd like to dig deeper. was (Author: facai): Hi, [~zsellami]. I guess that since the wordVectors is mllib model in fact, which might be removed in the future, so it is marked private and transient. More interestingly, wordVectors are saved in data folder as dataframe, see: {code} 336 override protected def saveImpl(path: String): Unit = { 337 DefaultParamsWriter.saveMetadata(instance, path, sc) 338 339 val wordVectors = instance.wordVectors.getVectors 340 val dataSeq = wordVectors.toSeq.map { case (word, vector) => Data(word, vector) } 341 val dataPath = new Path(path, "data").toString 342 sparkSession.createDataFrame(dataSeq) 343 .repartition(calculateNumberOfPartitions) 344 .write 345 .parquet(dataPath) 346 } {code} In all, developer indeed take a try to save the wordVector information, however it seems to break in pipeline as you said. So, could you give a example code to reproduce the bug? I'd like to dig deeper. > Spark 2.1.1: I want to be able to serialize wordVectors on Word2VecModel > ------------------------------------------------------------------------- > > Key: SPARK-21341 > URL: https://issues.apache.org/jira/browse/SPARK-21341 > Project: Spark > Issue Type: Bug > Components: ML > Affects Versions: 2.1.1 > Reporter: Zied Sellami > > I am using sparContext.saveAsObjectFile to save a complex object containing a > pipelineModel with a Word2Vec ML Transformer. When I load the object and call > myPipelineModel.transform, Word2VecModel raise a null pointer error on line > 292 Word2Vec.scala "wordVectors.getVectors" . I resolve the problem by > removing@transient annotation on val wordVectors and @transient lazy val on > getVectors function. > -Why this 2 val are transient ? > -Any solution to add a boolean function on the Word2Vec Transformer to force > the serialization of wordVectors. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org