[ https://issues.apache.org/jira/browse/SPARK-11994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated SPARK-11994: ------------------------------ Assignee: Antonio Murgia > Word2VecModel load and save cause SparkException when model is bigger than > spark.kryoserializer.buffer.max > ---------------------------------------------------------------------------------------------------------- > > Key: SPARK-11994 > URL: https://issues.apache.org/jira/browse/SPARK-11994 > Project: Spark > Issue Type: Bug > Components: MLlib > Affects Versions: 1.4.1, 1.5.1 > Reporter: Antonio Murgia > Assignee: Antonio Murgia > Labels: kryo, mllib > Fix For: 2.0.0 > > > When loading a Word2VecModel of compressed size 58Mb using the > Word2VecModel.load() method introduced in Spark 1.4.0 I get a > `org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. > Available: 0, required: 2` exception. > This happens because the model is saved as a unique file with no partitioning > and the kryo buffer overflows when tries to serialize it all. > Increasing `spark.kryoserializer.buffer.max` works as a temporary solution > but needs to increased again whenever we increase the model size. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org