[jira] Commented: (MAPREDUCE-1435) symlinks in cwd of the task are not handled properly after MAPREDUCE-896
[ https://issues.apache.org/jira/browse/MAPREDUCE-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842276#action_12842276 ] Hudson commented on MAPREDUCE-1435: --- Integrated in Hadoop-Mapreduce-trunk #250 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/250/]) > symlinks in cwd of the task are not handled properly after MAPREDUCE-896 > > > Key: MAPREDUCE-1435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1435 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1435.patch, 1435.v1.patch, 1435.v2.patch, 1435.v3.patch, > 1435.v4.patch, MR-1435-y20s.patch > > > With JVM reuse, TaskRunner.setupWorkDir() lists the contents of workDir and > does a fs.delete on each path listed. If the listed file is a symlink to > directory, it will delete the contents of those linked directories. This > would delete files from distributed cache and jars directory,if > mapred.create.symlink is true. > Changing ownership/permissions of symlinks through ENABLE_TASK_FOR_CLEANUP > would change ownership/permissions of underlying files. > This is observed by Karam while running streaming jobs with DistributedCache > and jvm reuse. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1435) symlinks in cwd of the task are not handled properly after MAPREDUCE-896
[ https://issues.apache.org/jira/browse/MAPREDUCE-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841667#action_12841667 ] Hudson commented on MAPREDUCE-1435: --- Integrated in Hadoop-Mapreduce-trunk-Commit #261 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/261/]) . Fix symlink handling in task work directory when cleaning up, essentially to avoid following links. Contributed by Ravi Gummadi. > symlinks in cwd of the task are not handled properly after MAPREDUCE-896 > > > Key: MAPREDUCE-1435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1435 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1435.patch, 1435.v1.patch, 1435.v2.patch, 1435.v3.patch, > 1435.v4.patch, MR-1435-y20s.patch > > > With JVM reuse, TaskRunner.setupWorkDir() lists the contents of workDir and > does a fs.delete on each path listed. If the listed file is a symlink to > directory, it will delete the contents of those linked directories. This > would delete files from distributed cache and jars directory,if > mapred.create.symlink is true. > Changing ownership/permissions of symlinks through ENABLE_TASK_FOR_CLEANUP > would change ownership/permissions of underlying files. > This is observed by Karam while running streaming jobs with DistributedCache > and jvm reuse. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1435) symlinks in cwd of the task are not handled properly after MAPREDUCE-896
[ https://issues.apache.org/jira/browse/MAPREDUCE-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841613#action_12841613 ] Hadoop QA commented on MAPREDUCE-1435: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437873/1435.v4.patch against trunk revision 918864. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/500/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/500/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/500/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/500/console This message is automatically generated. > symlinks in cwd of the task are not handled properly after MAPREDUCE-896 > > > Key: MAPREDUCE-1435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1435 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1435.patch, 1435.v1.patch, 1435.v2.patch, 1435.v3.patch, > 1435.v4.patch, MR-1435-y20s.patch > > > With JVM reuse, TaskRunner.setupWorkDir() lists the contents of workDir and > does a fs.delete on each path listed. If the listed file is a symlink to > directory, it will delete the contents of those linked directories. This > would delete files from distributed cache and jars directory,if > mapred.create.symlink is true. > Changing ownership/permissions of symlinks through ENABLE_TASK_FOR_CLEANUP > would change ownership/permissions of underlying files. > This is observed by Karam while running streaming jobs with DistributedCache > and jvm reuse. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1435) symlinks in cwd of the task are not handled properly after MAPREDUCE-896
[ https://issues.apache.org/jira/browse/MAPREDUCE-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841341#action_12841341 ] Ravi Gummadi commented on MAPREDUCE-1435: - Patch looks good. +1 > symlinks in cwd of the task are not handled properly after MAPREDUCE-896 > > > Key: MAPREDUCE-1435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1435 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1435.patch, 1435.v1.patch, 1435.v2.patch, 1435.v3.patch, > 1435.v4.patch, MR-1435-y20s.patch > > > With JVM reuse, TaskRunner.setupWorkDir() lists the contents of workDir and > does a fs.delete on each path listed. If the listed file is a symlink to > directory, it will delete the contents of those linked directories. This > would delete files from distributed cache and jars directory,if > mapred.create.symlink is true. > Changing ownership/permissions of symlinks through ENABLE_TASK_FOR_CLEANUP > would change ownership/permissions of underlying files. > This is observed by Karam while running streaming jobs with DistributedCache > and jvm reuse. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1435) symlinks in cwd of the task are not handled properly after MAPREDUCE-896
[ https://issues.apache.org/jira/browse/MAPREDUCE-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841167#action_12841167 ] Hemanth Yamijala commented on MAPREDUCE-1435: - Output of test-patch: {noformat} [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 18 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. {noformat} > symlinks in cwd of the task are not handled properly after MAPREDUCE-896 > > > Key: MAPREDUCE-1435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1435 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1435.patch, 1435.v1.patch, 1435.v2.patch, 1435.v3.patch, > 1435.v4.patch, MR-1435-y20s.patch > > > With JVM reuse, TaskRunner.setupWorkDir() lists the contents of workDir and > does a fs.delete on each path listed. If the listed file is a symlink to > directory, it will delete the contents of those linked directories. This > would delete files from distributed cache and jars directory,if > mapred.create.symlink is true. > Changing ownership/permissions of symlinks through ENABLE_TASK_FOR_CLEANUP > would change ownership/permissions of underlying files. > This is observed by Karam while running streaming jobs with DistributedCache > and jvm reuse. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1435) symlinks in cwd of the task are not handled properly after MAPREDUCE-896
[ https://issues.apache.org/jira/browse/MAPREDUCE-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840524#action_12840524 ] Ravi Gummadi commented on MAPREDUCE-1435: - Would it be better if we check if the owner and group of public distributed cache files are correct also ? Once we add that also to the checks of public distributed cache files, then ClusterWithLinuxTaskController.checkPermissionsOnDir() can be reused for these checks also and can avoid TestTrackerDistributedCacheManager.checkPublicFilePermissions() possibly. > symlinks in cwd of the task are not handled properly after MAPREDUCE-896 > > > Key: MAPREDUCE-1435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1435 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1435.patch, 1435.v1.patch, 1435.v2.patch, 1435.v3.patch, > MR-1435-y20s.patch > > > With JVM reuse, TaskRunner.setupWorkDir() lists the contents of workDir and > does a fs.delete on each path listed. If the listed file is a symlink to > directory, it will delete the contents of those linked directories. This > would delete files from distributed cache and jars directory,if > mapred.create.symlink is true. > Changing ownership/permissions of symlinks through ENABLE_TASK_FOR_CLEANUP > would change ownership/permissions of underlying files. > This is observed by Karam while running streaming jobs with DistributedCache > and jvm reuse. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1435) symlinks in cwd of the task are not handled properly after MAPREDUCE-896
[ https://issues.apache.org/jira/browse/MAPREDUCE-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840518#action_12840518 ] Amareshwari Sriramadasu commented on MAPREDUCE-1435: In TestStreamingAsDifferentUser, group ownership is compared with primary group of the launcher. It would fail if task-controller binary is owned by secondary group of the launcher. It can be compared with ClusterWithLinuxTaskController.taskTrackerSpecialGroup directly, which introduced in MAPREDUCE-1421. > symlinks in cwd of the task are not handled properly after MAPREDUCE-896 > > > Key: MAPREDUCE-1435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1435 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1435.patch, 1435.v1.patch, 1435.v2.patch, 1435.v3.patch, > MR-1435-y20s.patch > > > With JVM reuse, TaskRunner.setupWorkDir() lists the contents of workDir and > does a fs.delete on each path listed. If the listed file is a symlink to > directory, it will delete the contents of those linked directories. This > would delete files from distributed cache and jars directory,if > mapred.create.symlink is true. > Changing ownership/permissions of symlinks through ENABLE_TASK_FOR_CLEANUP > would change ownership/permissions of underlying files. > This is observed by Karam while running streaming jobs with DistributedCache > and jvm reuse. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1435) symlinks in cwd of the task are not handled properly after MAPREDUCE-896
[ https://issues.apache.org/jira/browse/MAPREDUCE-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840508#action_12840508 ] Hemanth Yamijala commented on MAPREDUCE-1435: - bq. -1 core tests. The patch failed core unit tests. The core test failure is being tracked in MAPREDUCE-1520. > symlinks in cwd of the task are not handled properly after MAPREDUCE-896 > > > Key: MAPREDUCE-1435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1435 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1435.patch, 1435.v1.patch, 1435.v2.patch, 1435.v3.patch, > MR-1435-y20s.patch > > > With JVM reuse, TaskRunner.setupWorkDir() lists the contents of workDir and > does a fs.delete on each path listed. If the listed file is a symlink to > directory, it will delete the contents of those linked directories. This > would delete files from distributed cache and jars directory,if > mapred.create.symlink is true. > Changing ownership/permissions of symlinks through ENABLE_TASK_FOR_CLEANUP > would change ownership/permissions of underlying files. > This is observed by Karam while running streaming jobs with DistributedCache > and jvm reuse. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1435) symlinks in cwd of the task are not handled properly after MAPREDUCE-896
[ https://issues.apache.org/jira/browse/MAPREDUCE-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840506#action_12840506 ] Hadoop QA commented on MAPREDUCE-1435: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437630/1435.v3.patch against trunk revision 918206. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/15/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/15/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/15/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/15/console This message is automatically generated. > symlinks in cwd of the task are not handled properly after MAPREDUCE-896 > > > Key: MAPREDUCE-1435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1435 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1435.patch, 1435.v1.patch, 1435.v2.patch, 1435.v3.patch, > MR-1435-y20s.patch > > > With JVM reuse, TaskRunner.setupWorkDir() lists the contents of workDir and > does a fs.delete on each path listed. If the listed file is a symlink to > directory, it will delete the contents of those linked directories. This > would delete files from distributed cache and jars directory,if > mapred.create.symlink is true. > Changing ownership/permissions of symlinks through ENABLE_TASK_FOR_CLEANUP > would change ownership/permissions of underlying files. > This is observed by Karam while running streaming jobs with DistributedCache > and jvm reuse. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1435) symlinks in cwd of the task are not handled properly after MAPREDUCE-896
[ https://issues.apache.org/jira/browse/MAPREDUCE-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830500#action_12830500 ] Hemanth Yamijala commented on MAPREDUCE-1435: - A few minor nits: - Typo: contets in comments of ClusterWithLinuxTaskController. - checkPermissionsOnPrivateDistCache: Do we expect at least one of the directories to have the file. If yes, we should check for that; otherwise we will mask errors. Similarly for checkPermissionsOnPublicDistCache - checkPublicFilePermissions seems to be verifying permissions only for others. Should we also check permissions for owner and group ? - Comments in createSubDirsAndSymLinks should be dir1/subDir, dir1/file - Do we need DefaultTaskController.enableTaskForCleanup to give rwx access to group ? Setting the permissions for user should be sufficient, I think. > symlinks in cwd of the task are not handled properly after MAPREDUCE-896 > > > Key: MAPREDUCE-1435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1435 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1435.patch, 1435.v1.patch, 1435.v2.patch > > > With JVM reuse, TaskRunner.setupWorkDir() lists the contents of workDir and > does a fs.delete on each path listed. If the listed file is a symlink to > directory, it will delete the contents of those linked directories. This > would delete files from distributed cache and jars directory,if > mapred.create.symlink is true. > Changing ownership/permissions of symlinks through ENABLE_TASK_FOR_CLEANUP > would change ownership/permissions of underlying files. > This is observed by Karam while running streaming jobs with DistributedCache > and jvm reuse. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1435) symlinks in cwd of the task are not handled properly after MAPREDUCE-896
[ https://issues.apache.org/jira/browse/MAPREDUCE-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829760#action_12829760 ] Ravi Gummadi commented on MAPREDUCE-1435: - TestFileArgs failed(known issue MAPREDUCE-1375) and all other unit tests passed on my local machine. > symlinks in cwd of the task are not handled properly after MAPREDUCE-896 > > > Key: MAPREDUCE-1435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1435 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1435.patch, 1435.v1.patch, 1435.v2.patch > > > With JVM reuse, TaskRunner.setupWorkDir() lists the contents of workDir and > does a fs.delete on each path listed. If the listed file is a symlink to > directory, it will delete the contents of those linked directories. This > would delete files from distributed cache and jars directory,if > mapred.create.symlink is true. > Changing ownership/permissions of symlinks through ENABLE_TASK_FOR_CLEANUP > would change ownership/permissions of underlying files. > This is observed by Karam while running streaming jobs with DistributedCache > and jvm reuse. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1435) symlinks in cwd of the task are not handled properly after MAPREDUCE-896
[ https://issues.apache.org/jira/browse/MAPREDUCE-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828941#action_12828941 ] Hemanth Yamijala commented on MAPREDUCE-1435: - As discussed offline, we probably need a unit test coverage for the changes in TaskRunner.setupWorkDir as well. Can you please include that ? Also, we may want to add a public distributed cache file in the streaming test case and ensure its permissions do not change too. > symlinks in cwd of the task are not handled properly after MAPREDUCE-896 > > > Key: MAPREDUCE-1435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1435 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1435.patch, 1435.v1.patch > > > With JVM reuse, TaskRunner.setupWorkDir() lists the contents of workDir and > does a fs.delete on each path listed. If the listed file is a symlink to > directory, it will delete the contents of those linked directories. This > would delete files from distributed cache and jars directory,if > mapred.create.symlink is true. > Changing ownership/permissions of symlinks through ENABLE_TASK_FOR_CLEANUP > would change ownership/permissions of underlying files. > This is observed by Karam while running streaming jobs with DistributedCache > and jvm reuse. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1435) symlinks in cwd of the task are not handled properly after MAPREDUCE-896
[ https://issues.apache.org/jira/browse/MAPREDUCE-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828165#action_12828165 ] Arun C Murthy commented on MAPREDUCE-1435: -- We suspect this was caused by MAPREDUCE-896. > symlinks in cwd of the task are not handled properly after MAPREDUCE-896 > > > Key: MAPREDUCE-1435 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1435 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu > Fix For: 0.22.0 > > > With JVM reuse, TaskRunner.setupWorkDir() lists the contents of workDir and > does a fs.delete on each path listed. If the listed file is a symlink to > directory, it will delete the contents of those linked directories. This > would delete files from distributed cache and jars directory,if > mapred.create.symlink is true. > Changing ownership/permissions of symlinks through ENABLE_TASK_FOR_CLEANUP > would change ownership/permissions of underlying files. > This is observed by Karam while running streaming jobs with DistributedCache > and jvm reuse. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.