[
https://issues.apache.org/jira/browse/HADOOP-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12554603
]
Milind Bhandarkar commented on HADOOP-2116:
-------------------------------------------
I would prefer separating the two. I.e. where job.jar goes, versus where the
job.local.dir goes. Especially for streaming, where side-effect tasks are
common, the mapper and reducer commands would need to have a clean directory
(empty) where they can cache job-specific data (dictionaries downloaded off the
network etc, that cannot be packaged as distributed archives). If job.jar also
lives there, it might someday clash with the files downloaded, and cause issues.
So, mapred.jar, jobCacheDir, and job.local.dir all need to be different
locations.
Is jobCacheDir available via a config variable ?
> Job.local.dir to be exposed to tasks
> ------------------------------------
>
> Key: HADOOP-2116
> URL: https://issues.apache.org/jira/browse/HADOOP-2116
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.14.3
> Environment: All
> Reporter: Milind Bhandarkar
> Fix For: 0.16.0
>
>
> Currently, since all task cwds are created under a jobcache directory, users
> that need a job-specific shared directory for use as scratch space, create
> ../work. This is hacky, and will break when HADOOP-2115 is addressed. For
> such jobs, hadoop mapred should expose job.local.dir via localized
> configuration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
