[ https://issues.apache.org/jira/browse/SPARK-28021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-28021. ---------------------------------- Resolution: Invalid Let's ask a question to mailing list before filing it as an issue. See https://spark.apache.org/community.html > A unappropriate exception in StaticMemoryManager.getMaxExecutionMemory > ---------------------------------------------------------------------- > > Key: SPARK-28021 > URL: https://issues.apache.org/jira/browse/SPARK-28021 > Project: Spark > Issue Type: Question > Components: Spark Core > Affects Versions: 2.4.3 > Reporter: child2d > Priority: Minor > > When i review StaticMemoryManager.scala, there comes a question to me. > {code:java} > private def getMaxExecutionMemory(conf: SparkConf): Long = { > val systemMaxMemory = conf.getLong("spark.testing.memory", > Runtime.getRuntime.maxMemory) > if (systemMaxMemory < MIN_MEMORY_BYTES) { > throw new IllegalArgumentException(s"System memory $systemMaxMemory must > " + > s"be at least $MIN_MEMORY_BYTES. Please increase heap size using the > --driver-memory " + > s"option or spark.driver.memory in Spark configuration.") > } > if (conf.contains("spark.executor.memory")) { > val executorMemory = conf.getSizeAsBytes("spark.executor.memory") > if (executorMemory < MIN_MEMORY_BYTES) { > throw new IllegalArgumentException(s"Executor memory $executorMemory > must be at least " + > s"$MIN_MEMORY_BYTES. Please increase executor memory using the " + > s"--executor-memory option or spark.executor.memory in Spark > configuration.") > } > } > val memoryFraction = conf.getDouble("spark.shuffle.memoryFraction", 0.2) > val safetyFraction = conf.getDouble("spark.shuffle.safetyFraction", 0.8) > (systemMaxMemory * memoryFraction * safetyFraction).toLong > } > {code} > When a executor tries to getMaxExecutionMemory, it should set systemMaxMemory > by usingĀ Runtime.getRuntime.maxMemory first, then compares the value between > systemMaxMemory and MIN_MEMORY_BYTES. > If the compared value is true, program thows an exception to remind user to > increase heap size by using --driver-memory. > I wonder if it is wrong because the heap size of executors are setted by > --executor-memory? > Although there is another exception about adjusting executor's memory below, > i just think that the first exception may be notĀ appropriate. > Thanks for answering my question! -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org