[ https://issues.apache.org/jira/browse/MAPREDUCE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092840#comment-14092840 ]
Daryn Sharp commented on MAPREDUCE-6032: ---------------------------------------- {{Filesystem#makeQualified(Path)}} is intended to fail if the fs is already qualified for another fs. {{Path#makeQualified(FileSystem)}}, which is oddly deprecated, is intended to qualify against the given fs only if the path isn't already qualified. What you found with it mangling a localfs authority is a nasty bug! All said, why even bother explicitly qualifying? This should work: {code} stagingDirPath = new Path(stagingDirStr); statingDirFS = stagingDirPath.getFileSystem(conf); {code} > Unable to check mapreduce job status if submitted using a non-default namenode > ------------------------------------------------------------------------------ > > Key: MAPREDUCE-6032 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6032 > Project: Hadoop Map/Reduce > Issue Type: Bug > Affects Versions: 2.0.5-alpha, 2.1.1-beta, 2.0.6-alpha, 2.2.0, 2.3.0, > 2.2.1, 2.4.1 > Environment: Any > Reporter: Benjamin Zhitomirsky > Assignee: Benjamin Zhitomirsky > Fix For: trunk > > Attachments: MAPREDUCE-6032.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > When MRv2 job container runs in a context of non-default file system > JobHistoryUtils.java obtains mapreduce.jobhistory.done-dir and > mapreduce.jobhistory.intermediate-done-dir as a non-qualified paths (e.g. > /mapred/history). This path is considered to belong to the current > container's context. As result the application history is being written to > another file system and job history server is unable to pick it up, because > it expects it to be found on the default file system. Currently providing > fully qualified path to those parameters is not supported as well, because of > a bug in JobHistoryEventHandler. > After this fix two scenarios will be supported: > - mapreduce.jobhistory.done-dir and > mapreduce.jobhistory.intermediate-done-dir (and the staging directory BTW) > will support a fully qualified path > - If a non-qualified path is configured then it will always be defaulted to > the default file system (core-site.xml). That's how consistency of history > location will be archived > Implementation notes: > - FileSystem#makeQualified throws exception if specified path belongs to > another file system. However FileContext#makeQualified work properly in this > case, and this is the meaning of the fix in JobHistoryEventHandler. I was not > ready to change behavior FileSystem#makeQualified because much more thought > is required. I afraid that many users expect such behavior, and fixing it > would break their code. > - The fix in JobHistoryUtils detects non-default namenode configuration only > if it comes from some "real" configuration: core-default.xml is ignored. This > is done primary as a kind of test hook, because otherwise setting > fs.defaultFS value during test executions would be always recognized by > JobHistoryUtils as a non-default namenode against 'file:///' specified in > core-default.xml. > (Remark. Note that makeQualified doesn't behave properly with file:/// > filesystem, for example: > new Path("file:///dir/subdir").makeQualified(new URI("hdfs://server:8020"), > new Path("/dir")) > Returns: "file://server:8020/dir/subdir" which doesn't make sense. > However I don't believe it worth fixing, since nobody really case about local > file system besides tests. My fix just ensures that all tests run smoothly by > ignoring core-default.xml file system in the logic.) -- This message was sent by Atlassian JIRA (v6.2#6252)