[
https://issues.apache.org/jira/browse/MAPREDUCE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13653473#comment-13653473
]
Xi Fang commented on MAPREDUCE-5224:
------------------------------------
The original motivation of this JIRA is trying to fix the following scenario.
In Azure, the default file system is set as ASV (Windows Azure Blob Storage),
but we would still like the system directory to be in DFS, because we don't
want to put such files in ASV that charge Azure customers fee. Thus, we want to
change JobTracker.java to allow that.
The problem in the current JobTracker.java is that we want to use
makeQualified() to assemble a path. But getSystemDir() uses the wrong fs object
to call fs.makeQualified(), if default (e.g. Azure in our scanerio) and
"mapred.system.dir" are using different file systems. In the proposed fix, we
rely on FileSystem.get() to choose the appropriate file system according to
"mapred.system.dir". It falls back on the default file system if the scheme is
not there.
Although the original motivation is trying to fix the problem for Azure, this
fix also applies to other scenarios where the default file system and
"mapred.system.dir" are supposed to use different file systems.
A unit test will follow.
> JobTracker should allow the system directory to be in non-default FS
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-5224
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5224
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Xi Fang
> Assignee: Xi Fang
> Priority: Minor
> Fix For: 1-win
>
> Attachments: MAPREDUCE-5224.patch
>
>
> JobTracker today expects the system directory to be in the default file
> system
> if (fs == null) {
> fs = mrOwner.doAs(new PrivilegedExceptionAction<FileSystem>() {
> public FileSystem run() throws IOException {
> return FileSystem.get(conf);
> }});
> }
> ...
> public String getSystemDir() {
> Path sysDir = new Path(conf.get("mapred.system.dir",
> "/tmp/hadoop/mapred/system"));
> return fs.makeQualified(sysDir).toString();
> }
> In Cloud like Azure the default file system is set as ASV (Windows Azure Blob
> Storage), but we would still like the system directory to be in DFS. We
> should change JobTracker to allow that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
