[
https://issues.apache.org/jira/browse/SPARK-15531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15300986#comment-15300986
]
mathieu longtin commented on SPARK-15531:
-----------------------------------------
Correct, on a 128G server, just running {{java}} with no argument will try to
allocate 32G, regardless of ulimit. It's "expected behavior" according to
Oracle.
> spark-class tries to use too much memory when running Launcher
> --------------------------------------------------------------
>
> Key: SPARK-15531
> URL: https://issues.apache.org/jira/browse/SPARK-15531
> Project: Spark
> Issue Type: Bug
> Components: Deploy
> Affects Versions: 1.6.1, 2.0.0
> Environment: Linux running in Univa or Sun Grid Engine
> Reporter: mathieu longtin
> Priority: Minor
> Labels: launcher
>
> When running Java on a server with a lot of memory but a rather small virtual
> memory ulimit, Java will try to allocate a large memory pool and fail:
> {code}
> # System has 128GB of Ram but ulimit set to 7.5G
> $ ulimit -v
> 7812500
> $ java -client
> Error occurred during initialization of VM
> Could not reserve enough space for object heap
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> {code}
> This is a known issue with Java, but unlikely to get fixed.
> As a result, when starting various Spark process (spark-submit, master or
> workers), they fail when {{spark-class}} tries to run
> {{org.apache.spark.launcher.Main}}.
> To fix this, add {{-Xmx128m}} to this line
> {code}
> "$RUNNER" -Xmx128m -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main
> "$\@"
> {code}
> (https://github.com/apache/spark/blob/master/bin/spark-class#L71)
> We've been using 128m and that works in our setup. Considering all the
> launcher does is analyze the arguments and env var and spit out some command,
> it should be plenty. All other calls to Java seem to include some value for
> -Xmx, so it is not an issue elsewhere.
> I don't mind submitting a PR, but I'm sure somebody has opinions on the 128m
> (bigger, smaller, configurable, ...), so I'd rather it would be discussed
> first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
