Hi, I am trying to get the same memory behavior in Spark 1.6 as I had in Spark 1.3 with default settings.
I set --driver-java-options "--Dspark.memory.useLegacyMode=true -Dspark.shuffle.memoryFraction=0.2 -Dspark.storage.memoryFraction=0.6 -Dspark.storage.unrollFraction=0.2" in Spark 1.6. But the numbers don't add up. For instance: --driver-java-options "-Dspark.shuffle.memoryFraction=0.1 -Dspark.storage.memoryFraction=0.1" in Spark 1.3.1 leads to: 16/03/29 14:47:36 INFO MemoryStore: MemoryStore started with capacity 46.1 MB The same in Spark 1.6.0 with -Dspark.memory.useLegacyMode=true -Dspark.shuffle.memoryFraction=0.1 -Dspark.storage.memoryFraction=0.1. 16/03/29 14:50:55 INFO MemoryStore: MemoryStore started with capacity 92.2 MB If I then increase both fractions to 0.2, the numbers of the MemoryStore both double (as one would expect), but that means there is still a 2x difference in allocated memory between Spark 1.3 and Spark 1.6. So my question: What do I need to do to get the default memory behavior of Spark 1.3.1 in Spark 1.6.0? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Using-spark-memory-useLegacyMode-true-does-not-yield-expected-behavior-tp26631.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
