You could try leaving all the configuration values to default and running your application and see if you are still hitting the heap issue, If so try adding a Swap space to the machines which will definitely help. Another way would be to set the heap space manually (export _JAVA_OPTIONS="-Xmx5g")
Thanks Best Regards On Wed, Apr 8, 2015 at 12:45 AM, Shuai Zheng <szheng.c...@gmail.com> wrote: > Hi All, > > > > I am a bit confused on spark.storage.memoryFraction, this is used to set > the area for RDD usage, will this RDD means only for cached and persisted > RDD? So if my program has no cached RDD at all (means that I have no > .cache() or .persist() call on any RDD), then I can set this > spark.storage.memoryFraction to a very small number or even zero? > > > > I am writing a program which consume a lot of memory (broadcast value, > runtime, etc). But I have no cached RDD, so should I just turn off this > spark.storage.memoryFraction to 0 (which will help me to improve the > performance)? > > > > And I have another issue on the broadcast, when I try to get a broadcast > value, it throws me out of memory error, which part of memory should I > allocate more (if I can’t increase my overall memory size). > > > > java.lang.OutOfMemoryError: Java heap spac > > e > > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$DoubleA > > rraySerializer.read(DefaultArraySerializers.java:218) > > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$DoubleA > > rraySerializer.read(DefaultArraySerializers.java:200) > > at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:699) > > at > com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.rea > > d(FieldSerializer.java:611) > > at > com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSeria > > lizer.java:221) > > at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648) > > at > com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.rea > > d(FieldSerializer.java:605) > > at > com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSeria > > lizer.java:221) > > at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729) > > at > org.apache.spark.serializer.KryoDeserializationStream.readObject(Kryo > > Serializer.scala:138) > > at > org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Ser > > ializer.scala:133) > > at > org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71) > > at > org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:2 > > 48) > > at > org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:13 > > 6) > > at > org.apache.spark.storage.BlockManager.doGetLocal(BlockManager.scala:5 > > 49) > > at > org.apache.spark.storage.BlockManager.getLocal(BlockManager.scala:431 > > ) > > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlo > > ck$1.apply(TorrentBroadcast.scala:167) > > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1152) > > at > org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(Torren > > tBroadcast.scala:164) > > at > org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(Torrent > > Broadcast.scala:64) > > at > org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.s > > cala:64) > > at > org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast > > .scala:87) > > > > > > Regards, > > > > Shuai >
