[ https://issues.apache.org/jira/browse/MAPREDUCE-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236859#comment-15236859 ]
Akira AJISAKA commented on MAPREDUCE-6607: ------------------------------------------ Additional comment: Would you add a test case to verify that the staging files that are not matched to file pattern are not preserved? > .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or > mapreduce.task.files.preserve.filepattern are set > ------------------------------------------------------------------------------------------------------------------------------- > > Key: MAPREDUCE-6607 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6607 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster > Affects Versions: 2.7.1 > Reporter: Maysam Yabandeh > Assignee: Kai Sasaki > Priority: Minor > Attachments: MAPREDUCE-6607.01.patch, MAPREDUCE-6607.02.patch, > MAPREDUCE-6607.03.patch, MAPREDUCE-6607.04.patch > > > if either of the following configs are set, then .staging dir is not cleaned > up: > * mapreduce.task.files.preserve.failedtask > * mapreduce.task.files.preserve.filepattern > The former was supposed to keep only .staging of failed tasks and the latter > was supposed to be used only if that task name matches against the specified > regular expression. > {code} > protected boolean keepJobFiles(JobConf conf) { > return (conf.getKeepTaskFilesPattern() != null || conf > .getKeepFailedTaskFiles()); > } > {code} > {code} > public void cleanupStagingDir() throws IOException { > /* make sure we clean the staging files */ > String jobTempDir = null; > FileSystem fs = getFileSystem(getConfig()); > try { > if (!keepJobFiles(new JobConf(getConfig()))) { > jobTempDir = getConfig().get(MRJobConfig.MAPREDUCE_JOB_DIR); > if (jobTempDir == null) { > LOG.warn("Job Staging directory is null"); > return; > } > Path jobTempDirPath = new Path(jobTempDir); > LOG.info("Deleting staging directory " + > FileSystem.getDefaultUri(getConfig()) + > " " + jobTempDir); > fs.delete(jobTempDirPath, true); > } > } catch(IOException io) { > LOG.error("Failed to cleanup staging dir " + jobTempDir, io); > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)