Did you have a chance to look at this doc http://spark.apache.org/docs/1.2.0/tuning.html
Thanks Best Regards On Tue, Feb 10, 2015 at 4:13 PM, Yifan LI <iamyifa...@gmail.com> wrote: > Hi Akhil, > > Excuse me, I am trying a random-walk algorithm over a not that large > graph(~1GB raw dataset, including ~5million vertices and ~60million edges) > on a cluster which has 20 machines. > > And, the property of each vertex in graph is a hash map, of which size > will increase dramatically during pregel supersteps. so, it seems to suffer > from high GC? > > Best, > Yifan LI > > > > > > On 10 Feb 2015, at 10:26, Akhil Das <ak...@sigmoidanalytics.com> wrote: > > You could try increasing the driver memory. Also, can you be more specific > about the data volume? > > Thanks > Best Regards > > On Mon, Feb 9, 2015 at 3:30 PM, Yifan LI <iamyifa...@gmail.com> wrote: > >> Hi, >> >> I just found the following errors during computation(graphx), anyone has >> ideas on this? thanks so much! >> >> (I think the memory is sufficient, spark.executor.memory 30GB ) >> >> >> 15/02/09 00:37:12 ERROR Executor: Exception in task 162.0 in stage 719.0 >> (TID 7653) >> java.lang.OutOfMemoryError: Java heap space >> at >> com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:410) >> at >> com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntMap.java:113) >> at >> com.esotericsoftware.kryo.util.MapReferenceResolver.addWrittenObject(MapReferenceResolver.java:23) >> at com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Kryo.java:598) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:566) >> at >> com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:29) >> at >> com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:27) >> at >> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) >> at >> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) >> at >> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) >> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) >> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) >> at com.twitter.chill.TraversableSerializer.write(Traversable.scala:27) >> at com.twitter.chill.TraversableSerializer.write(Traversable.scala:21) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:36) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at >> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:318) >> at >> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:293) >> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501) >> at >> com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564) >> at >> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at >> org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:128) >> at >> org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:110) >> 15/02/09 00:37:12 ERROR SparkUncaughtExceptionHandler: Uncaught exception in >> thread Thread[Executor task launch worker-15,5,main] >> java.lang.OutOfMemoryError: Java heap space >> at >> com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:410) >> at >> com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntMap.java:113) >> at >> com.esotericsoftware.kryo.util.MapReferenceResolver.addWrittenObject(MapReferenceResolver.java:23) >> at com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Kryo.java:598) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:566) >> at >> com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:29) >> at >> com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:27) >> at >> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) >> at >> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) >> at >> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) >> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) >> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) >> at com.twitter.chill.TraversableSerializer.write(Traversable.scala:27) >> at com.twitter.chill.TraversableSerializer.write(Traversable.scala:21) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:36) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at >> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:318) >> at >> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:293) >> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501) >> at >> com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564) >> at >> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at >> org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:128) >> at >> org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:110) >> >> >> >> Best, >> Yifan LI >> >> >> >> >> >> > >
