Few things to keep in mind: - I believe Driver memory should not exceed executor memory - Set spark.storage.memoryFraction default is 0.6 - Set spark.rdd.compress default is set to false - Always specify the level of parallelism while doing a groupBy, reduceBy, join, sortBy etc. - If you don't have enough memory and the data is huge, then set the Storage level to DISK_AND_MEMORY_SER
More you can read over here. <http://spark.apache.org/docs/1.0.0/tuning.html> Thanks Best Regards On Sun, Oct 12, 2014 at 10:28 PM, Chengi Liu <chengi.liu...@gmail.com> wrote: > Hi, > I am trying to use spark but I am having hard time configuring the > sparkconf... > My current conf is > conf = > SparkConf().set("spark.executor.memory","10g").set("spark.akka.frameSize", > "100000000").set("spark.driver.memory","16g") > > but I still see the java heap size error > 14/10/12 09:54:50 ERROR Executor: Exception in task 3.0 in stage 0.0 (TID > 3) > java.lang.OutOfMemoryError: Java heap space > at com.esotericsoftware.kryo.io.Input.readBytes(Input.java:296) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.read(DefaultArraySerializers.java:35) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.read(DefaultArraySerializers.java:18) > at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:699) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:332) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:293) > at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729) > at > com.twitter.chill.WrappedArraySerializer.read(WrappedArraySerializer.scala:34) > at > com.twitter.chill.WrappedArraySerializer.read(WrappedArraySerializer.scala:21) > at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729) > at org.apache.spark.serializer.KryoDeserializationStream.readO > > > Whats the right way to turn these knobs and what other knobs I can play > with. > Thanks >
