[
https://issues.apache.org/jira/browse/HADOOP-2227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550044
]
Amareshwari Sri Ramadasu commented on HADOOP-2227:
--------------------------------------------------
We have two options here.
1. The simple approach of just checking whether at least one of the disks
has enough space. This will cover all the intermediate outputs generation/copy
cases (for maps and reduces respectively). The bulk of the other operations
that the tasktracker does to do with disks is copying the job jar file, and,
localizing the tasks' cache. Both these are done before the task launch
(process spawn) and the tasktracker won't even launch the process if it
encounters a problem in this part (it will throw an exception). The tasktracker
then marks that task as having "failed". This behavior should be okay since we
haven't spent much time executing the task yet.
2. The other option is to change things like localizeJob, localizeTask and
localizeCache, to use the LocalDirAllocator but in order to do that, we have to
know the sizes of the cache files. This is doable, but involves an RPC.
Thoughts?
> wrong usage of mapred.local.dir.minspacestart
> ----------------------------------------------
>
> Key: HADOOP-2227
> URL: https://issues.apache.org/jira/browse/HADOOP-2227
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.15.1
> Reporter: Christian Kunz
> Assignee: Amareshwari Sri Ramadasu
> Priority: Critical
>
> As I understand it, mapred.local.dir.minspacestart should be used to specify
> that a map or reduce task requires a minimum disk space to be executed. When
> several disks are available, and mapred.local.dir lists all of them, then the
> free space should be added up. But TaskTracker.enoughFreeSpace returns false
> whenever one of the disks listed in mapred.local.dir has less than the space
> specified by mapred.local.dir.minspacestart.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
