[ https://issues.apache.org/jira/browse/MAPREDUCE-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13686233#comment-13686233 ]
Xi Fang commented on MAPREDUCE-5278: ------------------------------------ Thanks Bikas. A config name was added in JobClient.java {code} private static final String CLIENT_ACCESSIBLE_REMOTE_SCHEMES_KEY = "mapreduce.client.accessible.remote.schemes"; {code} And in copyRemoteFiles(), I changed to {code} String [] accessibleSchemes = job.getStrings( CLIENT_ACCESSIBLE_REMOTE_SCHEMES_KEY, null); {code} > Distributed cache is broken when JT staging dir is not on the default FS > ------------------------------------------------------------------------ > > Key: MAPREDUCE-5278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5278 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distributed-cache > Affects Versions: 1-win > Environment: Windows > Reporter: Xi Fang > Assignee: Xi Fang > Fix For: 1-win > > Attachments: MAPREDUCE-5278.2.patch, MAPREDUCE-5278.3.patch, > MAPREDUCE-5278.patch > > > Today, the JobTracker staging dir ("mapreduce.jobtracker.staging.root.dir) is > set to point to HDFS, even though other file systems (e.g. Amazon S3 file > system and Windows ASV file system) are the default file systems. > For ASV, this config was chosen and there are a few reasons why: > 1. To prevent leak of the storage account credentials to the user's storage > account; > 2. It uses HDFS for the transient job files what is good for two reasons – a) > it does not flood the user's storage account with irrelevant data/files b) it > leverages HDFS locality for small files > However, this approach conflicts with how distributed cache caching works, > completely negating the feature's functionality. > When files are added to the distributed cache (thru files/achieves/libjars > hadoop generic options), they are copied to the job tracker staging dir only > if they reside on a file system different that the jobtracker's. Later on, > this path is used as a "key" to cache the files locally on the tasktracker's > machine, and avoid localization (download/unzip) of the distributed cache > files if they are already localized. > In this configuration the caching is completely disabled and we always end up > copying dist cache files to the job tracker's staging dir first and > localizing them on the task tracker machine second. > This is especially not good for Oozie scenarios as Oozie uses dist cache to > populate Hive/Pig jars throughout the cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira