[ https://issues.apache.org/jira/browse/SPARK-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15094189#comment-15094189 ]
John Vines edited comment on SPARK-12650 at 1/12/16 10:52 PM: -------------------------------------------------------------- So I ran it and got this message {code}SPARK_JAVA_OPTS was detected (set to '-Xmx512M'). This is deprecated in Spark 1.0+. Please instead use: - ./spark-submit with conf/spark-defaults.conf to set defaults for an application - ./spark-submit with --driver-java-options to set -X options for a driver - spark.executor.extraJavaOptions to set -X options for executors - SPARK_DAEMON_JAVA_OPTS to set java options for standalone daemons (master or worker) {code} but that was just a warning (so small complaint if this is the proper solution), but it did properly cap the vmem use. EDIT: it appears this value applies to ALL processes though. So when I did 512 it was fine, but when I did 256 exectutors failed since I have their mem set to 512 so I got the containers failing to start due to `Incompatible minimum and maximum heap sizes specified` was (Author: vines): So I ran it and got this message {code}SPARK_JAVA_OPTS was detected (set to '-Xmx512M'). This is deprecated in Spark 1.0+. Please instead use: - ./spark-submit with conf/spark-defaults.conf to set defaults for an application - ./spark-submit with --driver-java-options to set -X options for a driver - spark.executor.extraJavaOptions to set -X options for executors - SPARK_DAEMON_JAVA_OPTS to set java options for standalone daemons (master or worker) {code} but that was just a warning (so small complaint if this is the proper solution), but it did properly cap the vmem use. > No means to specify Xmx settings for SparkSubmit in yarn-cluster mode > --------------------------------------------------------------------- > > Key: SPARK-12650 > URL: https://issues.apache.org/jira/browse/SPARK-12650 > Project: Spark > Issue Type: Bug > Components: Spark Submit > Affects Versions: 1.5.2 > Environment: Hadoop 2.6.0 > Reporter: John Vines > > Background- > I have an app master designed to do some work and then launch a spark job. > Issue- > If I use yarn-cluster, then the SparkSubmit does not Xmx itself at all, > leading to the jvm taking a default heap which is relatively large. This > causes a large amount of vmem to be taken, so that it is killed by yarn. This > can be worked around by disabling Yarn's vmem check, but that is a hack. > If I run it in yarn-client mode, it's fine as long as my container has enough > space for the driver, which is manageable. But I feel that the utter lack of > Xmx settings for what I believe is a very small jvm is a problem. > I believe this was introduced with the fix for SPARK-3884 -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org