Since the stacktrace shows kryo is being used, maybe, you could also try increasing spark.kryoserializer.buffer.max.mb. Hope this help.
Kelvin On Tue, Feb 10, 2015 at 1:26 AM, Akhil Das <ak...@sigmoidanalytics.com> wrote: > You could try increasing the driver memory. Also, can you be more specific > about the data volume? > > Thanks > Best Regards > > On Mon, Feb 9, 2015 at 3:30 PM, Yifan LI <iamyifa...@gmail.com> wrote: > >> Hi, >> >> I just found the following errors during computation(graphx), anyone has >> ideas on this? thanks so much! >> >> (I think the memory is sufficient, spark.executor.memory 30GB ) >> >> >> 15/02/09 00:37:12 ERROR Executor: Exception in task 162.0 in stage 719.0 >> (TID 7653) >> java.lang.OutOfMemoryError: Java heap space >> at >> com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:410) >> at >> com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntMap.java:113) >> at >> com.esotericsoftware.kryo.util.MapReferenceResolver.addWrittenObject(MapReferenceResolver.java:23) >> at com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Kryo.java:598) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:566) >> at >> com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:29) >> at >> com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:27) >> at >> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) >> at >> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) >> at >> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) >> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) >> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) >> at com.twitter.chill.TraversableSerializer.write(Traversable.scala:27) >> at com.twitter.chill.TraversableSerializer.write(Traversable.scala:21) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:36) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at >> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:318) >> at >> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:293) >> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501) >> at >> com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564) >> at >> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at >> org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:128) >> at >> org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:110) >> 15/02/09 00:37:12 ERROR SparkUncaughtExceptionHandler: Uncaught exception in >> thread Thread[Executor task launch worker-15,5,main] >> java.lang.OutOfMemoryError: Java heap space >> at >> com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:410) >> at >> com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntMap.java:113) >> at >> com.esotericsoftware.kryo.util.MapReferenceResolver.addWrittenObject(MapReferenceResolver.java:23) >> at com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Kryo.java:598) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:566) >> at >> com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:29) >> at >> com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:27) >> at >> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) >> at >> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) >> at >> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) >> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) >> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) >> at com.twitter.chill.TraversableSerializer.write(Traversable.scala:27) >> at com.twitter.chill.TraversableSerializer.write(Traversable.scala:21) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:36) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at >> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:318) >> at >> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:293) >> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501) >> at >> com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564) >> at >> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at >> org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:128) >> at >> org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:110) >> >> >> >> Best, >> Yifan LI >> >> >> >> >> >> >