[jira] [Updated] (MAPREDUCE-5902) JobHistoryServer (HistoryFileManager) needs more debug logs, fails to pick up jobs with % characters in the name.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jay vyas updated MAPREDUCE-5902:
Description:
1) JobHistoryServer sometimes skips over certain history files, and ignores
serving them as completed.
2) In addition to skipping these files, the JobHistoryServer doesnt effectively
log which files are being skipped , and why.
So In addition to determining why certain types of files are skipped (file name
length doesnt appear to be the reason, rather, it appears to be that %
characters throw the JobHistoryServer filter off), we should log completed
.jhist files which are available in the mr-history/tmp directory, yet they
are skipped for some reason.
*Regarding the actual bug : Skipping completed jhist files*
We will need an author of the JobHistoryServer, I think, to chime in on what
types of paths for jobs are actually valid. It appears that at least some
characters, if in a job name, will make the jobhistoryserver skip recognition
of a completed jhist file.
*Regarding logging*
It would be extremely useful , then, to have a couple of gaurded logs at this
level of the code, so that we can see, in the log folders, why files are being
filtered out , i.e. it is due to filterint or visibility.
{noformat}
private static List scanDirectory(Path path, FileContext fc,
PathFilter pathFilter) throws IOException {
path = fc.makeQualified(path);
List jhStatusList = new ArrayList();
RemoteIterator fileStatusIter = fc.listStatus(path);
while (fileStatusIter.hasNext()) {
FileStatus fileStatus = fileStatusIter.next();
Path filePath = fileStatus.getPath();
if (fileStatus.isFile() && pathFilter.accept(filePath)) {
jhStatusList.add(fileStatus);
}
}
return jhStatusList;
}
{noformat}
*Reproducing*
I was able to reproduce this bug by writing a custom mapreduce job with a job
name, which contained % characters. I have also seen this with a version of
the Mahout ParallelALSFactorizationJob, which includes "-" characters in its
name, which wind up getting replaced by "%2D" later on at some stage in the job
pipeline.
was:
1) JobHistoryServer sometimes skips over certain history files, and ignores
serving them as completed.
2) In addition to skipping these files, the JobHistoryServer doesnt effectively
log which files are being skipped , and why.
So In addition to determining why certain types of files are skipped (file name
length doesnt appear to be the reason, rather, it appears to be that %
characters throw the JobHistoryServer filter off), we should log completed
.jhist files which are available in the mr-history/tmp directory, yet they
are skipped for some reason.
** Regarding the actual bug : Skipping completed jhist files **
We will need an author of the JobHistoryServer, I think, to chime in on what
types of paths for jobs are actually valid. It appears that at least some
characters, if in a job name, will make the jobhistoryserver skip recognition
of a completed jhist file.
** Regarding logging **
It would be extremely useful , then, to have a couple of gaurded logs at this
level of the code, so that we can see, in the log folders, why files are being
filtered out , i.e. it is due to filterint or visibility.
{noformat}
private static List scanDirectory(Path path, FileContext fc,
PathFilter pathFilter) throws IOException {
path = fc.makeQualified(path);
List jhStatusList = new ArrayList();
RemoteIterator fileStatusIter = fc.listStatus(path);
while (fileStatusIter.hasNext()) {
FileStatus fileStatus = fileStatusIter.next();
Path filePath = fileStatus.getPath();
if (fileStatus.isFile() && pathFilter.accept(filePath)) {
jhStatusList.add(fileStatus);
}
}
return jhStatusList;
}
{noformat}
** Reproducing **
I was able to reproduce this bug by writing a custom mapreduce job with a job
name, which contained % characters. I have also seen this with a version of
the Mahout ParallelALSFactorizationJob, which includes "-" characters in its
name, which wind up getting replaced by "%2D" later on at some stage in the job
pipeline.
> JobHistoryServer (HistoryFileManager) needs more debug logs, fails to pick up
> jobs with % characters in the name.
> -
>
> Key: MAPREDUCE-5902
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5902
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: jay vyas
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> 1) JobHistoryServer sometimes skips over certain history files, and ignores
> serving them as completed.
> 2) In addition to skipping these files, the Job
[jira] [Updated] (MAPREDUCE-5902) JobHistoryServer (HistoryFileManager) needs more debug logs, fails to pick up jobs with % characters in the name.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jay vyas updated MAPREDUCE-5902:
Description:
1) JobHistoryServer sometimes skips over certain history files, and ignores
serving them as completed.
2) In addition to skipping these files, the JobHistoryServer doesnt effectively
log which files are being skipped , and why.
So In addition to determining why certain types of files are skipped (file name
length doesnt appear to be the reason, rather, it appears to be that %
characters throw the JobHistoryServer filter off), we should log completed
.jhist files which are available in the mr-history/tmp directory, yet they
are skipped for some reason.
** Regarding the actual bug : Skipping completed jhist files **
We will need an author of the JobHistoryServer, I think, to chime in on what
types of paths for jobs are actually valid. It appears that at least some
characters, if in a job name, will make the jobhistoryserver skip recognition
of a completed jhist file.
** Regarding logging **
It would be extremely useful , then, to have a couple of gaurded logs at this
level of the code, so that we can see, in the log folders, why files are being
filtered out , i.e. it is due to filterint or visibility.
{noformat}
private static List scanDirectory(Path path, FileContext fc,
PathFilter pathFilter) throws IOException {
path = fc.makeQualified(path);
List jhStatusList = new ArrayList();
RemoteIterator fileStatusIter = fc.listStatus(path);
while (fileStatusIter.hasNext()) {
FileStatus fileStatus = fileStatusIter.next();
Path filePath = fileStatus.getPath();
if (fileStatus.isFile() && pathFilter.accept(filePath)) {
jhStatusList.add(fileStatus);
}
}
return jhStatusList;
}
{noformat}
** Reproducing **
I was able to reproduce this bug by writing a custom mapreduce job with a job
name, which contained % characters. I have also seen this with a version of
the Mahout ParallelALSFactorizationJob, which includes "-" characters in its
name, which wind up getting replaced by "%2D" later on at some stage in the job
pipeline.
was:
With the JobHistory Server , it appears that its possible sometimes to skip
over certain history files. I havent been able to determine why yet, but I've
found that some long named .jhist files aren't getting collected into the done/
directory.
After tracing some in the actual source, and turning on DEBUG level logging, it
became clear that this snippet is an important workhorse
(scanDirectoryForIntermediateFiles, and scanDirectoryForHistoryFiles ultimately
boil down to scanDirectory()).
It would be extremely useful , then, to have a couple of gaurded logs at this
level of the code, so that we can see, in the log folders, why files are being
filtered out , i.e. it is due to filterint or visibility.
{noformat}
private static List scanDirectory(Path path, FileContext fc,
PathFilter pathFilter) throws IOException {
path = fc.makeQualified(path);
List jhStatusList = new ArrayList();
RemoteIterator fileStatusIter = fc.listStatus(path);
while (fileStatusIter.hasNext()) {
FileStatus fileStatus = fileStatusIter.next();
Path filePath = fileStatus.getPath();
if (fileStatus.isFile() && pathFilter.accept(filePath)) {
jhStatusList.add(fileStatus);
}
}
return jhStatusList;
}
{noformat}
> JobHistoryServer (HistoryFileManager) needs more debug logs, fails to pick up
> jobs with % characters in the name.
> -
>
> Key: MAPREDUCE-5902
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5902
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: jay vyas
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> 1) JobHistoryServer sometimes skips over certain history files, and ignores
> serving them as completed.
> 2) In addition to skipping these files, the JobHistoryServer doesnt
> effectively log which files are being skipped , and why.
> So In addition to determining why certain types of files are skipped (file
> name length doesnt appear to be the reason, rather, it appears to be that %
> characters throw the JobHistoryServer filter off), we should log completed
> .jhist files which are available in the mr-history/tmp directory, yet they
> are skipped for some reason.
> ** Regarding the actual bug : Skipping completed jhist files **
> We will need an author of the JobHistoryServer, I think, to chime in on what
> types of paths for jobs are actually valid. It appears that at least some
> characters, if in a job name, will make the jobhistoryserver skip recognition
> of a completed
[jira] [Updated] (MAPREDUCE-5902) JobHistoryServer (HistoryFileManager) needs more debug logs, fails to pick up jobs with % characters in the name.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jay vyas updated MAPREDUCE-5902:
Summary: JobHistoryServer (HistoryFileManager) needs more debug logs, fails
to pick up jobs with % characters in the name. (was: JobHistoryServer
(HistoryFileManager) needs more debug logs.)
> JobHistoryServer (HistoryFileManager) needs more debug logs, fails to pick up
> jobs with % characters in the name.
> -
>
> Key: MAPREDUCE-5902
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5902
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: jay vyas
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> With the JobHistory Server , it appears that its possible sometimes to skip
> over certain history files. I havent been able to determine why yet, but
> I've found that some long named .jhist files aren't getting collected into
> the done/ directory.
> After tracing some in the actual source, and turning on DEBUG level logging,
> it became clear that this snippet is an important workhorse
> (scanDirectoryForIntermediateFiles, and scanDirectoryForHistoryFiles
> ultimately boil down to scanDirectory()).
> It would be extremely useful , then, to have a couple of gaurded logs at this
> level of the code, so that we can see, in the log folders, why files are
> being filtered out , i.e. it is due to filterint or visibility.
> {noformat}
> private static List scanDirectory(Path path, FileContext fc,
> PathFilter pathFilter) throws IOException {
> path = fc.makeQualified(path);
> List jhStatusList = new ArrayList();
> RemoteIterator fileStatusIter = fc.listStatus(path);
> while (fileStatusIter.hasNext()) {
> FileStatus fileStatus = fileStatusIter.next();
> Path filePath = fileStatus.getPath();
> if (fileStatus.isFile() && pathFilter.accept(filePath)) {
> jhStatusList.add(fileStatus);
> }
> }
> return jhStatusList;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
