[
https://issues.apache.org/jira/browse/SPARK-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000044#comment-15000044
]
Chris Howard commented on SPARK-11154:
--------------------------------------
On review it looks like the following conf values need changes
- spark.yarn.driver.memoryOverhead
- spark.yarn.am.memoryOverhead
- spark.yarn.am.memory
- spark.yarn.executor.memoryOverhead
Changes required to
- yarn/ClientArguments.scala
- yarn/Client.scala
- yarn/YarnAllocator.scala
org.apache.spark.deploy.ClientArguments.scala uses
org.apache.spark.util.MemoryParam to convert to MB so propose same.
If we were to replicate the functionality of deploy.ClientArguments.scala we
should also consider allowing setting via command line args.
Propose we add new config values to ensure backward compatibility.
Anyone issues with naing as follows?
- spark.yarn.driver.memoryOverhead -> spark.yarn.driver.memOverhead
- spark.yarn.am.memoryOverhead -> spark.yarn.am.memOverhead
- spark.yarn.am.memory -> spark.yarn.am.mem
- spark.yarn.executor.memoryOverhead -> spark.yarn.executor.memOverhead
Docs will also need updating to reflect fact that original option is MB only
and new options support k | m | g variants.
Happy to run with this if no objections.
> make specificaition spark.yarn.executor.memoryOverhead consistent with
> typical JVM options
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-11154
> URL: https://issues.apache.org/jira/browse/SPARK-11154
> Project: Spark
> Issue Type: Improvement
> Components: Documentation, Spark Submit
> Reporter: Dustin Cote
> Priority: Minor
>
> spark.yarn.executor.memoryOverhead is currently specified in megabytes by
> default, but it would be nice to allow users to specify the size as though it
> were a typical -Xmx option to a JVM where you can have 'm' and 'g' appended
> to the end to explicitly specify megabytes or gigabytes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
