[
https://issues.apache.org/jira/browse/SPARK-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15085773#comment-15085773
]
Sean Owen commented on SPARK-12650:
-----------------------------------
This may be an aside but I'm curious -- what is your default max heap size on
your Java 7 JVM? I don't recall it being nearly that big ever, but maybe it is
dependent on the system memory and you have a lot and I haven't been keeping
up. I take your point about the JVM feeling welcome to use its max heap rather
than GC.
Anyway [~vanzin] might know a little more about how the 3 (?) JVMs in play here
have their memory set in the context of the new spark-submit setup.
BTW how are you running these things and what do you set for driver, executor
memory? the full command line plus any config in your conf file might be
helpful.
> No means to specify Xmx settings for SparkSubmit in yarn-cluster mode
> ---------------------------------------------------------------------
>
> Key: SPARK-12650
> URL: https://issues.apache.org/jira/browse/SPARK-12650
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 1.5.2
> Environment: Hadoop 2.6.0
> Reporter: John Vines
>
> Background-
> I have an app master designed to do some work and then launch a spark job.
> Issue-
> If I use yarn-cluster, then the SparkSubmit does not Xmx itself at all,
> leading to the jvm taking a default heap which is relatively large. This
> causes a large amount of vmem to be taken, so that it is killed by yarn. This
> can be worked around by disabling Yarn's vmem check, but that is a hack.
> If I run it in yarn-client mode, it's fine as long as my container has enough
> space for the driver, which is manageable. But I feel that the utter lack of
> Xmx settings for what I believe is a very small jvm is a problem.
> I believe this was introduced with the fix for SPARK-3884
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
