[
https://issues.apache.org/jira/browse/TEZ-699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14046366#comment-14046366
]
Rohini Palaniswamy commented on TEZ-699:
----------------------------------------
- I don't think it is advisable to set Xms by default. We have run into one
issue of lot of lost nodes because people set Xms and tasks were consuming more
memory than it should. From [~nroberts] 's comment on the internal issue, that
nodes can overcommit the resources (For eg: 24G node advertises 25G) in yarn
and with swapiness at 0, the nodemanagers were hitting OOM and killed by
monitoring process. But in that specific case user had set Xms and Xmx same and
that did not work well with overcommitting. Xms at 40% may not cause the
problem of lost nodes but still might cause more usage of resources than
normal. If you consider pig or hive, some of the vertices might use way less
physical memory than others even though whole DAG runs with same heap size. For
eg: If one portion of the script requires 4G memory, then user runs whole
script with 4G memory (We are yet to have options to have different memory for
different portions of the script). In this case all vertices will have 1.6G of
Xms which will be wastage of resources.
- Minor. javaOpts.contains("-Xmx") could be in the same if condition
instead of nested.
> Have sensible defaults for java opts
> ------------------------------------
>
> Key: TEZ-699
> URL: https://issues.apache.org/jira/browse/TEZ-699
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Bikas Saha
> Assignee: Hitesh Shah
> Attachments: TEZ-699.1.patch
>
>
> Its a burden to have to specify them all the time. API's already exist if
> users want to set special values. Also, for vertex memory would be good to
> have a TezUtils.getDefaultVertexResource() with a sensible resource size.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
