[
https://issues.apache.org/jira/browse/MAPREDUCE-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798658#action_12798658
]
Vinod K V commented on MAPREDUCE-1218:
--------------------------------------
Sorry for the delay, got tied up with other things. Gone through your latest
patch. Looks good overall, only two points:
Regarding MINIMUM_UPDATE_INTERVAL
bq. Then the second update will have a very short update window. This may
result inaccurate values.
I see. In that case, should we instead set it to the {{jiffyLengthInMillis}} as
that is the smallest unit of measurement for cpu-time?
bq. To handle the deprecation of mapreduce.tasktracker.memorycalculatorplugin
in TaskTracker, for memory calculations we should first try to use the class
denoted by this configuration if present, otherwise only we should fall back to
the new resource-calculator. To facilitate this we will also retain a
deprecated TTConfig.TT_MEMORY_CALCULATOR_PLUGIN constant.
By this I actually meant the behaviour of the TT with old configuration. Even
with your latest patch, if someone has the old configuration
{{mapreduce.tasktracker.memorycalculatorplugin}}, it won't be of any use at all
to the TT. Instead in {{TaskTracker.inintializeMemoryManagement()}}, I think we
should do something like the following:
{code}
calculatorPlugin = null;
if (conf.get(TTConfig.TT_MEMORY_CALCULATOR_PLUGIN)!=null) {
calculatorPlugin = MemoryCalculatorPlugin
.getMemoryCalculatorPlugin(clazz, fConf);
} else {
calculatorPlugin = resourceCalculatorPlugin;
}
if (calculatorPlugin != null) {
totalVirtualMemoryOnTT = calculatorPlugin.getVirtualMemorySize();
.....
}
{code}
Also, then, we will need to deprecate {{TTConfig.TT_MEMORY_CALCULATOR_PLUGIN}}
constant.
> Collecting cpu and memory usage for TaskTrackers
> ------------------------------------------------
>
> Key: MAPREDUCE-1218
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1218
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Affects Versions: 0.22.0
> Environment: linux
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1218-rename.sh, MAPREDUCE-1218-v2.patch,
> MAPREDUCE-1218-v3.patch, MAPREDUCE-1218-v4.patch, MAPREDUCE-1218-v5.patch,
> MAPREDUCE-1218.patch
>
>
> The information can be used for resource aware scheduling.
> Note that this is related to MAPREDUCE-220. There the per task resource
> information is collected.
> This one collects the per machine information.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
