[
https://issues.apache.org/jira/browse/HADOOP-4439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640465#action_12640465
]
Hemanth Yamijala commented on HADOOP-4439:
------------------------------------------
The following changes are identified:
- We remove the concept of having a default memory per task on the TT,
introduced in HADOOP-3759 as the max memory per TT / number of slots. The
problem with this model is that in a heterogenous cluster, different TTs could
give different default memory per task values for the same job, which is
confusing.
- Instead, we introduce a default memory per task configuration variable that
is expected to be controlled by the cluster admin. This is the value that will
be used for a job which does not specify any memory requirements. The advantage
with this model is that it eases configuration, and makes the default value
consistent for the users.
- If a job has not specified any memory requirements, this variable would be
set to the job's configuration, maybe via the {{Task}} object.
- We modify the algorithm of protecting RAM limits introduced in HADOOP-3581 to
use the configured memory per task instead of the default memory per task.
- We remove the reporting of the default memory per task, introduced in
HADOOP-3759, done via the {{TaskTrackerStatus.ResourceStatus}}. Instead we
report the total memory available on the TT instead of the default memory per
task.
- When HADOOP-4053 is fixed, the above values would be used to schedule tasks.
- However until HADOOP-4053 is fixed, these configuration parameters and the
corresponding {{JobConf}} variables should not be exposed in any public API or
documentation, as they could confuse users. This can be turned on after Hadoop
0.19
> Cleanup memory related resource management
> ------------------------------------------
>
> Key: HADOOP-4439
> URL: https://issues.apache.org/jira/browse/HADOOP-4439
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.19.0
>
>
> HADOOP-3759 and HADOOP-3581 introduced memory based resource management. This
> JIRA is to cleanup certain aspects of the two issues that came up while doing
> HADOOP-4035, which is filed to support memory based scheduling
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
