[
https://issues.apache.org/jira/browse/MAPREDUCE-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846613#action_12846613
]
Scott Chen commented on MAPREDUCE-1221:
---------------------------------------
The definitions of the parameters in the previous comment are not very clear.
Here are the definitions.
*mapreduce.tasktracker.reserved.physicalmemory.mb*
Specify how much memory on the TT that will not be used by tasks.
For example if we configure this to be 2G and we have 16G of memory.
The tasks only allow to use 14G of memory. If the limit is exceeded, TT will
kill the task consumes the most amount of memory.
We use reserved memory instead of total memory because there may be cases that
we have nonuniform cluster.
*mapreduce.map.memory.physical.mb*
Memory limit for one mapper. If the mapper uses more memory that this, it will
fail.
*mapreduce.reduce.memory.physical.mb*
Memory limit for one reducer. If the reducer uses more memory that this, it
will fail.
> Kill tasks on a node if the free physical memory on that machine falls below
> a configured threshold
> ---------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1221
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1221
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1221-v1.patch, MAPREDUCE-1221-v2.patch,
> MAPREDUCE-1221-v3.patch, MAPREDUCE-1221-v4.patch
>
>
> The TaskTracker currently supports killing tasks if the virtual memory of a
> task exceeds a set of configured thresholds. I would like to extend this
> feature to enable killing tasks if the physical memory used by that task
> exceeds a certain threshold.
> On a certain operating system (guess?), if user space processes start using
> lots of memory, the machine hangs and dies quickly. This means that we would
> like to prevent map-reduce jobs from triggering this condition. From my
> understanding, the killing-based-on-virtual-memory-limits (HADOOP-5883) were
> designed to address this problem. This works well when most map-reduce jobs
> are Java jobs and have well-defined -Xmx parameters that specify the max
> virtual memory for each task. On the other hand, if each task forks off
> mappers/reducers written in other languages (python/php, etc), the total
> virtual memory usage of the process-subtree varies greatly. In these cases,
> it is better to use kill-tasks-using-physical-memory-limits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
