Yes, I have read it, and am trying to find some way to do that… Thanks :)
Best, Yifan LI > On 10 Feb 2015, at 12:06, Akhil Das <ak...@sigmoidanalytics.com> wrote: > > Did you have a chance to look at this doc > http://spark.apache.org/docs/1.2.0/tuning.html > <http://spark.apache.org/docs/1.2.0/tuning.html> > > Thanks > Best Regards > > On Tue, Feb 10, 2015 at 4:13 PM, Yifan LI <iamyifa...@gmail.com > <mailto:iamyifa...@gmail.com>> wrote: > Hi Akhil, > > Excuse me, I am trying a random-walk algorithm over a not that large > graph(~1GB raw dataset, including ~5million vertices and ~60million edges) on > a cluster which has 20 machines. > > And, the property of each vertex in graph is a hash map, of which size will > increase dramatically during pregel supersteps. so, it seems to suffer from > high GC? > > Best, > Yifan LI > > > > > >> On 10 Feb 2015, at 10:26, Akhil Das <ak...@sigmoidanalytics.com >> <mailto:ak...@sigmoidanalytics.com>> wrote: >> >> You could try increasing the driver memory. Also, can you be more specific >> about the data volume? >> >> Thanks >> Best Regards >> >> On Mon, Feb 9, 2015 at 3:30 PM, Yifan LI <iamyifa...@gmail.com >> <mailto:iamyifa...@gmail.com>> wrote: >> Hi, >> >> I just found the following errors during computation(graphx), anyone has >> ideas on this? thanks so much! >> >> (I think the memory is sufficient, spark.executor.memory 30GB ) >> >> >> 15/02/09 00:37:12 ERROR Executor: Exception in task 162.0 in stage 719.0 >> (TID 7653) >> java.lang.OutOfMemoryError: Java heap space >> at >> com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:410) >> at >> com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntMap.java:113) >> at >> com.esotericsoftware.kryo.util.MapReferenceResolver.addWrittenObject(MapReferenceResolver.java:23) >> at com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Kryo.java:598) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:566) >> at >> com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:29) >> at >> com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:27) >> at >> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) >> at >> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) >> at >> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) >> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) >> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) >> at com.twitter.chill.TraversableSerializer.write(Traversable.scala:27) >> at com.twitter.chill.TraversableSerializer.write(Traversable.scala:21) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:36) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at >> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:318) >> at >> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:293) >> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501) >> at >> com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564) >> at >> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at >> org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:128) >> at >> org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:110) >> 15/02/09 00:37:12 ERROR SparkUncaughtExceptionHandler: Uncaught exception in >> thread Thread[Executor task launch worker-15,5,main] >> java.lang.OutOfMemoryError: Java heap space >> at >> com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:410) >> at >> com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntMap.java:113) >> at >> com.esotericsoftware.kryo.util.MapReferenceResolver.addWrittenObject(MapReferenceResolver.java:23) >> at com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Kryo.java:598) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:566) >> at >> com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:29) >> at >> com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:27) >> at >> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) >> at >> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) >> at >> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) >> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) >> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) >> at com.twitter.chill.TraversableSerializer.write(Traversable.scala:27) >> at com.twitter.chill.TraversableSerializer.write(Traversable.scala:21) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:36) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at >> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:318) >> at >> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:293) >> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501) >> at >> com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564) >> at >> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37) >> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33) >> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568) >> at >> org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:128) >> at >> org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:110) >> >> >> Best, >> Yifan LI >> >> >> >> >> >> > >
