[ https://issues.apache.org/jira/browse/MAPREDUCE-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated MAPREDUCE-1422: ---------------------------------- Attachment: mapreduce-1422-test-v1.0.patch Attaching an example test-job that can be run on a cluster to verify the fix. Here are the steps to verify : 1) _Reproduce the bug_ {code} 1) Apply this patch. 2) Deploy the cluster 3) run './bin/hadoop jar build/hadoop-mapred-test-*.jar badjobexample -m 1 -mt 10000 -recordt 10000' 4) While the job is running, check ${job-work-dir}/subDir/ on the tasktracker 5) Even after the job completes, some files are left behind in the job-work-dir. {code} 2) _Verify the fix_ Apply the [patch|https://issues.apache.org/jira/secure/attachment/12436775/mapreduce-1422-v1.4.2.patch] along with the attached patch and repeat the steps mentioned above. With this patch, the job-work-dir is properly cleanup as expected. Note : job-work-dir for the default case would look something like {mapred.local.dir}/taskTracker/{user.name}/jobcache/{job.id}/work/ > Changing permissions of files/dirs under job-work-dir may be needed sothat > cleaning up of job-dir in all mapred-local-directories succeeds always > ------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: MAPREDUCE-1422 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1422 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker > Reporter: Ravi Gummadi > Assignee: Amar Kamat > Attachments: mapreduce-1422-test-v1.0.patch, > mapreduce-1422-v1.4.2.patch > > > After MAPREDUCE-896, if LinuxTaskController is set in config, task-controller > binary is launched for changing permissions of taskAttemptDir and taskWorkDir > before cleaning up of these directories sothat cleanup will be succeeded even > if user had created files/dirs under taskAttemptDir or taskWorkDir with > non-writable permissions. Users can't create files/dirs under job-dir > directly as we set 2570 for job-dir. But as job-work-dir has 2770 permissions > and user can create files/dirs under job-work-dir with non-writable > permissions, Changing permissions of files/dirs under job-work-dir may be > needed sothat cleaning up of job-dir in all mapred-local-directories succeeds > always. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.