[ https://issues.apache.org/jira/browse/MAPREDUCE-6044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Nauroth updated MAPREDUCE-6044: ------------------------------------- Description: After MAPREDUCE-6032, the string of the intermediate done dir will be a fully qualified path. The following code in JobHistoryUtils tries to concat this path and user name to create a per-user dir path, using File.separator as the seperator (on Windows, it is "\"). {code} public static String getHistoryIntermediateDoneDirForUser(Configuration conf) throws IOException { return getConfiguredHistoryIntermediateDoneDirPrefix(conf) + File.separator + UserGroupInformation.getCurrentUser().getShortUserName(); } {code} Therefore, an intermediate done dir for user will become "hdfs://localhost:9201/mapred/history/done_intermediate\user". With the scheme available in the path, Path class will not replace "\" with "/", and finally FS cannot handle this path correctly: it will take "done_intermediateuser" as a single directory name. was: After MAPREDUCE-6032, the string of the intermediate done dir will be a fully qualified path. The following code in JobHistroyUtils tries to concat this path and user name to create a per-user dir path, using File.separator as the seperator (on Windows, it is "\"). {code} public static String getHistoryIntermediateDoneDirForUser(Configuration conf) throws IOException { return getConfiguredHistoryIntermediateDoneDirPrefix(conf) + File.separator + UserGroupInformation.getCurrentUser().getShortUserName(); } {code} Therefore, an intermediate done dir for user will become "hdfs://localhost:9201/mapred/history/done_intermediate\user". With the scheme available in the path, Path class will not replace "\" with "/", and finally FS cannot handle this path correctly: it will take "done_intermediateuser" as a single directory name. > Fully qualified intermediate done directory will break per-user dir creation > on Windows > --------------------------------------------------------------------------------------- > > Key: MAPREDUCE-6044 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6044 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver > Affects Versions: 3.0.0, 2.6.0 > Environment: Windows > Reporter: Zhijie Shen > Assignee: Zhijie Shen > Attachments: MAPREDUCE-6044.1.patch > > > After MAPREDUCE-6032, the string of the intermediate done dir will be a fully > qualified path. > The following code in JobHistoryUtils tries to concat this path and user name > to create a per-user dir path, using File.separator as the seperator (on > Windows, it is "\"). > {code} > public static String getHistoryIntermediateDoneDirForUser(Configuration > conf) throws IOException { > return getConfiguredHistoryIntermediateDoneDirPrefix(conf) + > File.separator > + UserGroupInformation.getCurrentUser().getShortUserName(); > } > {code} > Therefore, an intermediate done dir for user will become > "hdfs://localhost:9201/mapred/history/done_intermediate\user". With the > scheme available in the path, Path class will not replace "\" with "/", and > finally FS cannot handle this path correctly: it will take > "done_intermediateuser" as a single directory name. -- This message was sent by Atlassian JIRA (v6.2#6252)