[
https://issues.apache.org/jira/browse/MAPREDUCE-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838066#action_12838066
]
Scott Chen commented on MAPREDUCE-1221:
---------------------------------------
@Arun: Thanks again for the comment. I appreciate the help.
I agree that virtual-memory-limit has lots of benefit.
It kills the tasks quickly before the task starts doing bad things and it is
more predictable.
But virtual-memory-limit also wastes some memory because the processes does not
actually use that much of physical memory.
For example, there are tasks acquire 1.6GB of virtual memory but use only 600mb
all the time.
If you cumulate the total, the memory that is not being used is a lot.
The other main reason that we want to do this is that per task
virtual-memory-limit is an API change for our users.
The jobs they used to submit everyday may failed and they have to learn to set
the per-task limit.
The good thing about this patch is that it is backward compatible.
I think it is may not be that bad that we kill the task.
It will be scheduled again on other machine and the job can still be finished.
It will only slow down the job.
But if we don't kill it, it is very likely that it will caused the machine to
fail and we fail the task anyway.
Also all the tasks on the machine fail that is even worse.
> Kill tasks on a node if the free physical memory on that machine falls below
> a configured threshold
> ---------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1221
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1221
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1221-v1.patch, MAPREDUCE-1221-v2.patch,
> MAPREDUCE-1221-v3.patch
>
>
> The TaskTracker currently supports killing tasks if the virtual memory of a
> task exceeds a set of configured thresholds. I would like to extend this
> feature to enable killing tasks if the physical memory used by that task
> exceeds a certain threshold.
> On a certain operating system (guess?), if user space processes start using
> lots of memory, the machine hangs and dies quickly. This means that we would
> like to prevent map-reduce jobs from triggering this condition. From my
> understanding, the killing-based-on-virtual-memory-limits (HADOOP-5883) were
> designed to address this problem. This works well when most map-reduce jobs
> are Java jobs and have well-defined -Xmx parameters that specify the max
> virtual memory for each task. On the other hand, if each task forks off
> mappers/reducers written in other languages (python/php, etc), the total
> virtual memory usage of the process-subtree varies greatly. In these cases,
> it is better to use kill-tasks-using-physical-memory-limits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
