[ https://issues.apache.org/jira/browse/SPARK-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom Hubregtsen closed SPARK-14367. ---------------------------------- Resolution: Not A Bug useLegacyMode works as expected, I just made a wrong assumption > spark.memory.useLegacyMode=true in 1.6 does not yield the same memory > behavior as Spark 1.3 > ------------------------------------------------------------------------------------------- > > Key: SPARK-14367 > URL: https://issues.apache.org/jira/browse/SPARK-14367 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.6.0 > Environment: Ubuntu 15.10 with ibm-java-ppc64le-80. > Reporter: Tom Hubregtsen > Priority: Minor > Labels: backwards-compatibility > > Hi, > I am trying to get the same memory behavior in Spark 1.6 as I had in Spark > 1.3 with default settings. > I set > --driver-java-options "--Dspark.memory.useLegacyMode=true > -Dspark.shuffle.memoryFraction=0.2 -Dspark.storage.memoryFraction=0.6 > -Dspark.storage.unrollFraction=0.2" > in Spark 1.6. > But the numbers don't add up. For instance: > --driver-java-options "-Dspark.shuffle.memoryFraction=0.1 > -Dspark.storage.memoryFraction=0.1" > in Spark 1.3.1 leads to: > 16/03/29 14:47:36 INFO MemoryStore: MemoryStore started with capacity 46.1 MB > The same in Spark 1.6.0 with -Dspark.memory.useLegacyMode=true > -Dspark.shuffle.memoryFraction=0.1 -Dspark.storage.memoryFraction=0.1. > 16/03/29 14:50:55 INFO MemoryStore: MemoryStore started with capacity 92.2 MB > If I then increase both fractions to 0.2, the numbers of the MemoryStore both > double (as one would expect), but that means there is still a 2x difference > in allocated memory between Spark 1.3 and Spark 1.6. So my question: > I believe a parameter that reads > spark.memory.useLegacyMode=true > should yield the *exact* memory behavior as in the Legacy version. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org