[
https://issues.apache.org/jira/browse/MAPREDUCE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13656644#comment-13656644
]
Xi Fang commented on MAPREDUCE-5224:
------------------------------------
I updated the patch. It includes the unit test. I also made some change to
MAPREDUCE-5224.patch because the previous one is not complete. In many places,
the default file system is used to access the system directory rather than only
in getSystemDir(). Thus, this requires much more changes to the original code.
I still have some questions I am not quite sure.
1. In the constructor of JobTracker:
try {
FileStatus systemDirStatus = systemDirFs.getFileStatus(systemDir);
if (!systemDirStatus.isOwnedByUser(
mrOwner.getShortUserName(), mrOwner.getGroupNames()))
{ throw new AccessControlException("The systemdir " + systemDir + " is not
owned by " + mrOwner.getShortUserName()); }
if (!systemDirStatus.getPermission().equals(SYSTEM_DIR_PERMISSION))
{ LOG.warn("Incorrect permissions on " + systemDir + ". Setting it to " +
SYSTEM_DIR_PERMISSION); systemDirFs.setPermission(systemDir,new
FsPermission(SYSTEM_DIR_PERMISSION)); }
}
Basically, I have changed the file system used to access the system dir. But I
am not quite sure if I should change the two IF statements, because the file
permission might be a problem.
2. LocalJobRunner has a method getSystemDir() as well. It uses the default file
system to access the system directory.
public String getSystemDir()
{ Path sysDir = new Path(conf.get("mapred.system.dir",
"/tmp/hadoop/mapred/system")); return fs.makeQualified(sysDir).toString(); }
I am not quite sure if I need to change this as well.
Thanks!
> JobTracker should allow the system directory to be in non-default FS
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-5224
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5224
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Reporter: Xi Fang
> Assignee: Xi Fang
> Priority: Minor
> Fix For: 1-win
>
> Attachments: MAPREDUCE-5224.2.patch, MAPREDUCE-5224.patch
>
>
> JobTracker today expects the system directory to be in the default file
> system
> if (fs == null) {
> fs = mrOwner.doAs(new PrivilegedExceptionAction<FileSystem>() {
> public FileSystem run() throws IOException {
> return FileSystem.get(conf);
> }});
> }
> ...
> public String getSystemDir() {
> Path sysDir = new Path(conf.get("mapred.system.dir",
> "/tmp/hadoop/mapred/system"));
> return fs.makeQualified(sysDir).toString();
> }
> In Cloud like Azure the default file system is set as ASV (Windows Azure Blob
> Storage), but we would still like the system directory to be in DFS. We
> should change JobTracker to allow that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
