Yun Zhao created SPARK-10839: -------------------------------- Summary: SPARK_DAEMON_MEMORY has effect on heap size of thriftserver Key: SPARK-10839 URL: https://issues.apache.org/jira/browse/SPARK-10839 Project: Spark Issue Type: Bug Components: Spark Submit Affects Versions: 1.5.0, 1.4.1 Reporter: Yun Zhao
When SPARK_DAEMON_MEMORY in spark-env.sh is setted to modify memory of Master or Worker, there's an effect on heap size of thriftserver, further, this effect cannot be modified by spark.driver.memory or --driver-memory. Version 1.3.1 does not have the same problem. in org.apache.spark.launcher.SparkSubmitCommandBuilder: {quote} String tsMemory = isThriftServer(mainClass) ? System.getenv("SPARK_DAEMON_MEMORY") : null; String memory = firstNonEmpty(tsMemory, firstNonEmptyValue(SparkLauncher.DRIVER_MEMORY, conf, props), System.getenv("SPARK_DRIVER_MEMORY"), System.getenv("SPARK_MEM"), DEFAULT_MEM); cmd.add("-Xms" + memory); cmd.add("-Xmx" + memory); {quote} SPARK_DAEMON_MEMORY has the highest priority. It can be modified like this: {quote} String memory = firstNonEmpty(firstNonEmptyValue(SparkLauncher.DRIVER_MEMORY, conf, props), System.getenv("SPARK_DRIVER_MEMORY"), tsMemory, System.getenv("SPARK_MEM"), DEFAULT_MEM); {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org