[ https://issues.apache.org/jira/browse/SPARK-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiangrui Meng resolved SPARK-2200. ---------------------------------- Resolution: Later Close this JIRA since this is caused by a non-default kryo setting. > breeze DenseVector not serializable with KryoSerializer > ------------------------------------------------------- > > Key: SPARK-2200 > URL: https://issues.apache.org/jira/browse/SPARK-2200 > Project: Spark > Issue Type: Bug > Components: MLlib > Affects Versions: 1.0.0 > Reporter: Neville Li > Priority: Minor > > Spark 1.0.0 depends on breeze 0.7 and for some reason serializing DenseVector > with KryoSerializer throws the following stack trace. Looks like some > recursive field in the object. Upgrading to 0.8.1 solved this. > {code} > java.lang.StackOverflowError > at java.lang.reflect.Field.getDeclaringClass(Field.java:154) > at > sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:54) > at > sun.reflect.UnsafeQualifiedObjectFieldAccessorImpl.get(UnsafeQualifiedObjectFieldAccessorImpl.java:38) > at java.lang.reflect.Field.get(Field.java:379) > at > com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:552) > at > com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213) > at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501) > at > com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564) > at > com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213) > at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501) > ... > {code} > Code to reproduce: > {code} > import breeze.linalg.DenseVector > import org.apache.spark.SparkConf > import org.apache.spark.serializer.KryoSerializer > object SerializerTest { > def main(args: Array[String]) { > val conf = new SparkConf() > .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") > .set("spark.kryo.registrator", classOf[MyRegistrator].getName) > .set("spark.kryo.referenceTracking", "false") > .set("spark.kryoserializer.buffer.mb", "8") > val serializer = new KryoSerializer(conf).newInstance() > serializer.serialize(DenseVector.rand(10)) > } > } > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)