Morty Zhong created MAPREDUCE-7229:
--------------------------------------
Summary: Job History Server should ignore continuing move
done_intermediate files when rename failed
Key: MAPREDUCE-7229
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7229
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.1.2
Reporter: Morty Zhong
{code:java}
private void moveToDoneNow(final Path src, final Path target)
throws IOException {
LOG.info("Moving " + src.toString() + " to " + target.toString());
try {
intermediateDoneDirFc.rename(src, target, Options.Rename.NONE);
} catch (FileNotFoundException e) {
if (doneDirFc.util().exists(target)) {
LOG.info("Source file " + src.toString() + " not found, but target "
+ "file " + target.toString() + " already exists. Move already "
+ "happened.");
} else {
throw e;
}
}
}
{code}
if files in done_intermediate dir were already deleted(deleted manually), not
move to done dir. Job history will infinitely rename these files later, and
throw exp always.This will cause newly created files in done_intermediate dir
can not be moved. Later, when scan done_intermediate dir,
FileContext.listStatus(done_intermediate) will so slow, and gc problem
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
