GitHub user ryan-williams opened a pull request:
https://github.com/apache/spark/pull/3525
Improve YarnAllocator's parsing of "memory overhead" param
* let it be specified as a fraction of the executor memory
* add/generalize some utilities for parsing "memory strings" that are found
around the Spark codebase.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ryan-williams/spark mem-overhead
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3525.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3525
----
commit 849c8bc129a8f12290b896085bce3f9feb8b041b
Author: Ryan Williams <ryan.blake.willi...@gmail.com>
Date: 2014-11-13T02:32:34Z
let yarn mem overhead be specified as a fraction
.. of total executor memory. Several people have alluded to the ratio
of the executor.memoryOverhead to executor.memory as being more
important than the absolute amount of executor.memoryOverhead, so this
commit allows for specifying the former, and takes the maximum of the
computed overheads implied by the fractional and absolute (# of mb)
conf variables.
commit 5d2327a5dd6b479149b01b8a7eddbbbf94d62aa2
Author: Ryan Williams <ryan.blake.willi...@gmail.com>
Date: 2014-11-13T02:52:50Z
memoryStringToMb can have default scale specified
Previously it assumed a unitless number represented raw bytes, but I
want to use it for a config variable that previously defaulted to # of
megabytes and not break backwards-compatibility.
commit 928831e588bf0a1073ea73a34d47a8ecd4b931f8
Author: Ryan Williams <ryan.blake.willi...@gmail.com>
Date: 2014-11-13T02:53:01Z
remove unused import
commit fe8a348cb5e7a50253351193470042694309f4c1
Author: Ryan Williams <ryan.blake.willi...@gmail.com>
Date: 2014-11-13T04:00:19Z
generalize Utils.memoryStringToMb to b/k/m/g scale
- one existing usage in Utils converts a memory string to bytes, so I
folded that into the generalized `parseMemoryString` function that
`memoryStringToMb` is also now a wrapper for.
- a usage in YarnAllocator (see following commit) needs some special
handling around the unit-less case, further motivating this
abstraction
commit 5b361397dc0bcdd478a7328029bc51d22aba3fd0
Author: Ryan Williams <ryan.blake.willi...@gmail.com>
Date: 2014-11-13T04:00:57Z
executor.memoryOverhead takes a âmemory stringâ
previously just assumed megabytes
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
