John Vines created SPARK-12650:
----------------------------------
Summary: No means to specify Xmx settings for SparkSubmit in
yarn-cluster mode
Key: SPARK-12650
URL: https://issues.apache.org/jira/browse/SPARK-12650
Project: Spark
Issue Type: Bug
Affects Versions: 1.5.2
Environment: Hadoop 2.6.0
Reporter: John Vines
Background-
I have an app master designed to do some work and then launch a spark job.
Issue-
If I use yarn-cluster, then the SparkSubmit does not Xmx itself at all, leading
to the jvm taking a default heap which is relatively large. This causes a large
amount of vmem to be taken, so that it is killed by yarn. This can be worked
around by disabling Yarn's vmem check, but that is a hack.
If I run it in yarn-client mode, it's fine as long as my container has enough
space for the driver, which is manageable. But I feel that the utter lack of
Xmx settings for what I believe is a very small jvm is a problem.
I believe this was introduced with the fix for SPARK-3884
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
