[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-1186: Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I committed this to trunk. Thanks, Amareshwari ! While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1186-1.txt, patch-1186-2.txt, patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-3.txt, patch-1186-4.txt, patch-1186-5.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1186: --- Status: Open (was: Patch Available) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1186-1.txt, patch-1186-2.txt, patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-3.txt, patch-1186-4.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1186: --- Attachment: patch-1186-5.txt Patch with comments incorporated. While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1186-1.txt, patch-1186-2.txt, patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-3.txt, patch-1186-4.txt, patch-1186-5.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1186: --- Status: Patch Available (was: Open) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1186-1.txt, patch-1186-2.txt, patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-3.txt, patch-1186-4.txt, patch-1186-5.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1186: --- Attachment: patch-1186-4.txt Patch with suggested proposal and incorporating review comments. While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1186-1.txt, patch-1186-2.txt, patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-3.txt, patch-1186-4.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1186: --- Status: Patch Available (was: Open) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1186-1.txt, patch-1186-2.txt, patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-3.txt, patch-1186-4.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1186: --- Status: Open (was: Patch Available) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1186-1.txt, patch-1186-2.txt, patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-3.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1186: --- Status: Open (was: Patch Available) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1186-1.txt, patch-1186-2.txt, patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-3.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1186: --- Status: Patch Available (was: Open) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1186-1.txt, patch-1186-2.txt, patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-3.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1186: --- Attachment: patch-1186-2.txt Patch for review, implementing the proposal. While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: patch-1186-1.txt, patch-1186-2.txt, patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1186: --- Fix Version/s: (was: 0.21.0) 0.22.0 Status: Patch Available (was: Open) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: patch-1186-1.txt, patch-1186-2.txt, patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1186: --- Attachment: patch-1186-1.txt Patch for trunk, with changes done in task-controller.c While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: patch-1186-1.txt, patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1186: --- Attachment: patch-1186-3-ydist.txt Patch renaming uniqueSubDir. I think the log will be useful in debugging, so leaving it as is. Verified that chmod is not done on baseDir, by manually verifying logs. Reproduced the issue by putting about 1Lakh files in base dir, and running a job using 5 files on distributed cache. Verified that the same job, under same environment, finishes faster with the patch. Verified that streaming and pipes jobs, which use executables from distributed cache, work fine (with both DefaultTaskController and LinuxTaskController). Thanks Karthikeyan for helping in testing. test-patch and ant test passed on local machine, except TestHdfsProxy. While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: patch-1186-3-ydist.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1186: - Attachment: patch-1186-3-ydist.txt I dither, I preferred both 'unique' and 'parent' prefixes for CacheStatus.parentDir, so I renamed it to be CacheStatus.uniqueParentDir... no other changes. *grazing at my navel* While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1186: - Attachment: (was: patch-1186-3-ydist.txt) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: patch-1186-3-ydist.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1186: - Attachment: patch-1186-3-ydist.txt Forgot --no-prefix While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1186: - Status: Open (was: Patch Available) The trunk patch needs more work, no? Nits on ydist patch: # CacheStatus.uniqueSubDir should probably be called CacheStatus.parentDir # There are some extra LOG.info statements introduced here, are they just for debugging or do you intend to keep them in the source? While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1186: --- Attachment: patch-1186-ydist.txt Patch incorporating Arun's offline review comments on Y!20 patch. While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir --- Key: MAPREDUCE-1186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.21.0 Reporter: Vinod K V Assignee: Amareshwari Sriramadasu Fix For: 0.21.0 Attachments: patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
