[
https://issues.apache.org/jira/browse/MAPREDUCE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14099662#comment-14099662
]
Hudson commented on MAPREDUCE-6032:
-----------------------------------
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1865 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1865/])
MAPREDUCE-6032. Fix the non-unicode character in the change log in CHANGES.txt.
(zjshen:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1618270)
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
MAPREDUCE-6032. Made MR jobs write job history files on the default FS when the
current context’s FS is different. Contributed by Benjamin Zhitomirsky.
(zjshen:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1618269)
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestHistoryFileManager.java
> Unable to check mapreduce job status if submitted using a non-default namenode
> ------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6032
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6032
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 2.0.5-alpha, 2.1.1-beta, 2.0.6-alpha, 2.2.0, 2.3.0,
> 2.2.1, 2.4.1
> Environment: Any
> Reporter: Benjamin Zhitomirsky
> Assignee: Benjamin Zhitomirsky
> Fix For: 2.6.0
>
> Attachments: MAPREDUCE-6032.1.patch, MAPREDUCE-6032.patch
>
> Original Estimate: 24h
> Time Spent: 24h
> Remaining Estimate: 0h
>
> When MRv2 job container runs in a context of non-default file system
> JobHistoryUtils.java obtains mapreduce.jobhistory.done-dir and
> mapreduce.jobhistory.intermediate-done-dir as a non-qualified paths (e.g.
> /mapred/history). This path is considered to belong to the current
> container's context. As result the application history is being written to
> another file system and job history server is unable to pick it up, because
> it expects it to be found on the default file system. Currently providing
> fully qualified path to those parameters is not supported as well, because of
> a bug in JobHistoryEventHandler.
> After this fix two scenarios will be supported:
> - mapreduce.jobhistory.done-dir and
> mapreduce.jobhistory.intermediate-done-dir (and the staging directory BTW)
> will support a fully qualified path
> - If a non-qualified path is configured then it will always be defaulted to
> the default file system (core-site.xml). That's how consistency of history
> location will be archived
> Implementation notes:
> - FileSystem#makeQualified throws exception if specified path belongs to
> another file system. However FileContext#makeQualified work properly in this
> case, and this is the meaning of the fix in JobHistoryEventHandler. I was not
> ready to change behavior FileSystem#makeQualified because much more thought
> is required. I afraid that many users expect such behavior, and fixing it
> would break their code.
> - The fix in JobHistoryUtils detects non-default namenode configuration only
> if it comes from some "real" configuration: core-default.xml is ignored. This
> is done primary as a kind of test hook, because otherwise setting
> fs.defaultFS value during test executions would be always recognized by
> JobHistoryUtils as a non-default namenode against 'file:///' specified in
> core-default.xml.
> (Remark. Note that makeQualified doesn't behave properly with file:///
> filesystem, for example:
> new Path("file:///dir/subdir").makeQualified(new URI("hdfs://server:8020"),
> new Path("/dir"))
> Returns: "file://server:8020/dir/subdir" which doesn't make sense.
> However I don't believe it worth fixing, since nobody really case about local
> file system besides tests. My fix just ensures that all tests run smoothly by
> ignoring core-default.xml file system in the logic.)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
