[
https://issues.apache.org/jira/browse/HADOOP-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557918#action_12557918
]
owen.omalley edited comment on HADOOP-2116 at 1/10/08 10:42 PM:
-----------------------------------------------------------------
*Ugh* is right.
I'd propose some better names:
{code}
$local/work/$jobid/
cache/ -- file cache
jars/ -- expanded jar
job.xml -- the generic job conf
$taskid/
job.xml -- task localized job conf
output/ -- map outputs
work/ -- cwd for task
{code}
with each of the leaf directories being placed independently on the partitions.
We should define localized attributes to point to where each of the leaf
directories is.
I agree with Arun that we should re-work the -file option to use the file cache
with symlinks.
was (Author: owen.omalley):
*Ugh* is right.
I'd propose some better names:
$local/work/$jobid/
cache/ -- file cache
jars/ -- expanded jar
job.xml -- the generic job conf
$taskid/
job.xml -- task localized job conf
output/ -- map outputs
work/ -- cwd for task
with each of the leaf directories being placed independently on the partitions.
We should define localized attributes to point to where each of the leaf
directories is.
I agree with Arun that we should re-work the -file option to use the file cache
with symlinks.
> Job.local.dir to be exposed to tasks
> ------------------------------------
>
> Key: HADOOP-2116
> URL: https://issues.apache.org/jira/browse/HADOOP-2116
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.14.3
> Environment: All
> Reporter: Milind Bhandarkar
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.16.0
>
> Attachments: patch-2116.txt, patch-2116.txt
>
>
> Currently, since all task cwds are created under a jobcache directory, users
> that need a job-specific shared directory for use as scratch space, create
> ../work. This is hacky, and will break when HADOOP-2115 is addressed. For
> such jobs, hadoop mapred should expose job.local.dir via localized
> configuration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
