[
https://issues.apache.org/jira/browse/MAPREDUCE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Zhitomirsky updated MAPREDUCE-6032:
--------------------------------------------
Attachment: MAPREDUCE-6032.patch
Patch uploaded
> Unable to check mapreduce job status if submitted using a non-default namenode
> ------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6032
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6032
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.0.5-alpha, 2.1.1-beta, 2.0.6-alpha, 2.2.0, 2.3.0,
> 2.2.1, 2.4.1
> Environment: Any
> Reporter: Benjamin Zhitomirsky
> Assignee: Benjamin Zhitomirsky
> Fix For: trunk
>
> Attachments: MAPREDUCE-6032.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> When MRv2 job container runs in a context of non-default file system
> JobHistoryUtils.java obtains mapreduce.jobhistory.done-dir and
> mapreduce.jobhistory.intermediate-done-dir as a non-qualified paths (e.g.
> /mapred/history). This path is considered to belong to the current
> container's context. As result the application history is being written to
> another file system and job history server is unable to pick it up, because
> it expects it to be found on the default file system. Currently providing
> fully qualified path to those parameters is not supported as well, because of
> a bug in JobHistoryEventHandler.
> After this fix two scenarios will be supported:
> - mapreduce.jobhistory.done-dir and
> mapreduce.jobhistory.intermediate-done-dir (and the staging directory BTW)
> will support a fully qualified path
> - If a non-qualified path is configured then it will always be defaulted to
> the default file system (core-site.xml). That's how consistency of history
> location will be archived
> Implementation notes:
> - FileSystem#makeQualified throws exception if specified path belongs to
> another file system. However FileContext#makeQualified work properly in this
> case, and this is the meaning of the fix in JobHistoryEventHandler. I was not
> ready to change behavior FileSystem#makeQualified because much more thought
> is required. I afraid that many users expect such behavior, and fixing it
> would break their code.
> - The fix in JobHistoryUtils detects non-default namenode configuration only
> if it comes from some "real" configuration: core-default.xml is ignored. This
> is done primary as a kind of test hook, because otherwise setting
> fs.defaultFS value during test executions would be always recognized by
> JobHistoryUtils as a non-default namenode against 'file:///' specified in
> core-default.xml.
> (Remark. Note that makeQualified doesn't behave properly with file:///
> filesystem, for example:
> new Path("file:///dir/subdir").makeQualified(new URI("hdfs://server:8020"),
> new Path("/dir"))
> Returns: "file://server:8020/dir/subdir" which doesn't make sense.
> However I don't believe it worth fixing, since nobody really case about local
> file system besides tests. My fix just ensures that all tests run smoothly by
> ignoring core-default.xml file system in the logic.)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
