Hi Akhil, Excuse me, I am trying a random-walk algorithm over a not that large graph(~1GB raw dataset, including ~5million vertices and ~60million edges) on a cluster which has 20 machines.
And, the property of each vertex in graph is a hash map, of which size will increase dramatically during pregel supersteps. so, it seems to suffer from high GC? Best, Yifan LI > On 10 Feb 2015, at 10:26, Akhil Das <ak...@sigmoidanalytics.com> wrote: > > You could try increasing the driver memory. Also, can you be more specific > about the data volume? > > Thanks > Best Regards > > On Mon, Feb 9, 2015 at 3:30 PM, Yifan LI <iamyifa...@gmail.com > <mailto:iamyifa...@gmail.com>> wrote: > Hi, > > I just found the following errors during computation(graphx), anyone has > ideas on this? thanks so much! > > (I think the memory is sufficient, spark.executor.memory 30GB ) > > > 15/02/09 00:37:12 ERROR Executor: Exception in task 162.0 in stage 719.0 (TID > 7653) > java.lang.OutOfMemoryError: Java heap space > at > com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:410) > at > com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntMap.java:113) > at > com.esotericsoftware.kryo.util.MapReferenceResolver.addWrittenObject(MapReferenceResolver.java:23) > at com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Kryo.java:598) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:566) > at > com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:29) > at > com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:27) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) > at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) > at com.twitter.chill.TraversableSerializer.write(Traversable.scala:27) > at com.twitter.chill.TraversableSerializer.write(Traversable.scala:21) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:36) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:318) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:293) > at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501) > at > com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564) > at > com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at > org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:128) > at > org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:110) > 15/02/09 00:37:12 ERROR SparkUncaughtExceptionHandler: Uncaught exception in > thread Thread[Executor task launch worker-15,5,main] > java.lang.OutOfMemoryError: Java heap space > at > com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:410) > at > com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntMap.java:113) > at > com.esotericsoftware.kryo.util.MapReferenceResolver.addWrittenObject(MapReferenceResolver.java:23) > at com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Kryo.java:598) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:566) > at > com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:29) > at > com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:27) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) > at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) > at com.twitter.chill.TraversableSerializer.write(Traversable.scala:27) > at com.twitter.chill.TraversableSerializer.write(Traversable.scala:21) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:36) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:318) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:293) > at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501) > at > com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564) > at > com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) > at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) > at > org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:128) > at > org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:110) > > > Best, > Yifan LI > > > > > >
