jay vyas created MAPREDUCE-5902:
-----------------------------------
Summary: JobHistoryServer needs more debug logs.
Key: MAPREDUCE-5902
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5902
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Reporter: jay vyas
With the JobHistory Server , it appears that its possible sometimes to skip
over certain history files. I havent been able to determine why yet, but I've
found that some long named .jhist files aren't getting collected into the done/
directory.
After tracing some in the actual source, and turning on DEBUG level logging, it
became clear that this snippet is an important workhorse
(scanDirectoryForIntermediateFiles, and scanDirectoryForHistoryFiles ultimately
boil down to scanDirectory()).
It would be extremely useful , then, to have a couple of gaurded logs at this
level of the code, so that we can see, in the log folders, why files are being
filtered out , i.e. it is due to filterint or visibility.
{noformat}
private static List<FileStatus> scanDirectory(Path path, FileContext fc,
PathFilter pathFilter) throws IOException {
path = fc.makeQualified(path);
List<FileStatus> jhStatusList = new ArrayList<FileStatus>();
RemoteIterator<FileStatus> fileStatusIter = fc.listStatus(path);
while (fileStatusIter.hasNext()) {
FileStatus fileStatus = fileStatusIter.next();
Path filePath = fileStatus.getPath();
if (fileStatus.isFile() && pathFilter.accept(filePath)) {
jhStatusList.add(fileStatus);
}
}
return jhStatusList;
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
