[
https://issues.apache.org/jira/browse/MAPREDUCE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663457#comment-13663457
]
Xi Fang commented on MAPREDUCE-5224:
------------------------------------
Thanks Chuan and Ivan.
1. For Chuan's comment: I added an assert to check this dir exists indeed. This
also addresses Ivan's 10th comment.
2. For Ivan's comment:
a. For comments #1, 3, 5, 7, 9, 10, I just followed the comments.
b. For comment #2: I personally think it would be better if we can throw out an
exception rather than swallowing it and setting it back to default file system.
As mentioned by Mostafa (offline), for example, if someone configured the
system dir as http://www.awesome.com/system, then with the fallback solution
the exception saying "HTTP is not supported" will be swallowed and we'll set
the system directory as just /system in the default file system, which doesn't
seem like good behavior. We may want someone explicitly know/handle this at the
moment this happens.
c. For comment #4: I renamed the JobTracker#fs to defaultFs and still keep it
just for possible future use/reference of this variable.
d. For comment #6, 8: I put the initialization of MiniDFSCluster, MiniMRCluster
in the test case and let setUp() just construct a configuration. In this way,
we don't have to throw IOException in setUp() and test case would fail if my
code changes are not applied.
> JobTracker should allow the system directory to be in non-default FS
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-5224
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5224
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Xi Fang
> Assignee: Xi Fang
> Priority: Minor
> Fix For: 1-win
>
> Attachments: MAPREDUCE-5224.2.patch, MAPREDUCE-5224.3.patch,
> MAPREDUCE-5224.patch
>
>
> JobTracker today expects the system directory to be in the default file
> system
> if (fs == null) {
> fs = mrOwner.doAs(new PrivilegedExceptionAction<FileSystem>() {
> public FileSystem run() throws IOException {
> return FileSystem.get(conf);
> }});
> }
> ...
> public String getSystemDir() {
> Path sysDir = new Path(conf.get("mapred.system.dir",
> "/tmp/hadoop/mapred/system"));
> return fs.makeQualified(sysDir).toString();
> }
> In Cloud like Azure the default file system is set as ASV (Windows Azure Blob
> Storage), but we would still like the system directory to be in DFS. We
> should change JobTracker to allow that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
