John Vines created SPARK-12650: ---------------------------------- Summary: No means to specify Xmx settings for SparkSubmit in yarn-cluster mode Key: SPARK-12650 URL: https://issues.apache.org/jira/browse/SPARK-12650 Project: Spark Issue Type: Bug Affects Versions: 1.5.2 Environment: Hadoop 2.6.0 Reporter: John Vines
Background- I have an app master designed to do some work and then launch a spark job. Issue- If I use yarn-cluster, then the SparkSubmit does not Xmx itself at all, leading to the jvm taking a default heap which is relatively large. This causes a large amount of vmem to be taken, so that it is killed by yarn. This can be worked around by disabling Yarn's vmem check, but that is a hack. If I run it in yarn-client mode, it's fine as long as my container has enough space for the driver, which is manageable. But I feel that the utter lack of Xmx settings for what I believe is a very small jvm is a problem. I believe this was introduced with the fix for SPARK-3884 -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org