Just posted below for a similar question. Have you seen this thread ?
http://search-hadoop.com/m/JW1q5ezXPH/KryoException%253A+Buffer+overflow&subj=RE+spark+nbsp+kryo+serilizable+nbsp+exception On Tue, Oct 21, 2014 at 2:44 PM, Yang <teddyyyy...@gmail.com> wrote: > this is the stack trace I got with yarn logs -applicationId > > really no idea where to dig further. > > thanks! > yang > > 14/10/21 14:36:43 INFO ConnectionManager: Accepted connection from [ > phxaishdc9dn1262.stratus.phx.ebay.com/10.115.58.21] > 14/10/21 14:36:47 ERROR Executor: Exception in task ID 98 > com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 3, > required: 8 > Serialization trace: > data$mcD$sp (breeze.linalg.DenseVector$mcD$sp) > at com.esotericsoftware.kryo.io.Output.require(Output.java:138) > at com.esotericsoftware.kryo.io.Output.writeLong(Output.java:477) > at com.esotericsoftware.kryo.io.Output.writeDouble(Output.java:596) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$DoubleArraySerializer.write(DefaultArraySerializers.java:212) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$DoubleArraySerializer.write(DefaultArraySerializers.java:200) > at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:549) > at > com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:570) > at > com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213) > at > com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at > com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) > at > com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:34) > at > com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at > com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:38) > at > com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:34) > at > com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:318) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:293) > at > com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at > org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:142) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:193) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > >
