[jira] [Commented] (MAPREDUCE-2238) Undeletable build directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13050073#comment-13050073 ] Kihwal Lee commented on MAPREDUCE-2238: --- We've seen this in 0.20-security builds. May be we should put HADOOP-7110 and this to 0.20-security. I see they are in CDH3. Undeletable build directories -- Key: MAPREDUCE-2238 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, test Affects Versions: 0.22.0 Reporter: Eli Collins Assignee: Todd Lipcon Priority: Critical Fix For: 0.22.0 Attachments: mapreduce-2238.txt, mapreduce-2238.txt, mapreduce-2238.txt The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir. https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console Building remotely on hadoop7 hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7 at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537) at hudson.model.AbstractProject.checkout(AbstractProject.java:1116) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411) at hudson.model.Run.run(Run.java:1324) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_00_0 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12986169#action_12986169 ] Hudson commented on MAPREDUCE-2238: --- Integrated in Hadoop-Mapreduce-trunk-Commit #583 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/583/]) MAPREDUCE-2238. Fix permissions handling to avoid leaving undeletable directories in local dirs. Contributed by Todd Lipcon Undeletable build directories -- Key: MAPREDUCE-2238 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, test Affects Versions: 0.22.0 Reporter: Eli Collins Assignee: Todd Lipcon Priority: Critical Fix For: 0.22.0 Attachments: mapreduce-2238.txt, mapreduce-2238.txt, mapreduce-2238.txt The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir. https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console Building remotely on hadoop7 hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7 at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537) at hudson.model.AbstractProject.checkout(AbstractProject.java:1116) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411) at hudson.model.Run.run(Run.java:1324) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_00_0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12983812#action_12983812 ] Todd Lipcon commented on MAPREDUCE-2238: Unit tests pass except known failures Undeletable build directories -- Key: MAPREDUCE-2238 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, test Affects Versions: 0.22.0 Reporter: Eli Collins Assignee: Todd Lipcon Priority: Critical Fix For: 0.22.0 Attachments: mapreduce-2238.txt, mapreduce-2238.txt, mapreduce-2238.txt The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir. https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console Building remotely on hadoop7 hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7 at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537) at hudson.model.AbstractProject.checkout(AbstractProject.java:1116) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411) at hudson.model.Run.run(Run.java:1324) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_00_0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12983267#action_12983267 ] Todd Lipcon commented on MAPREDUCE-2238: Spent some time adding logging and looping the tests to figure out this problem. I think I have it cracked. The issue is not multiple threads calling setPermission() on the same process, but rather a case where one thread is calling setPermission on the *parent* directory of a file where another thread (actually another entire process) is calling setPermission. In particular, these two invocations race: 2011-01-18 09:00:40,958 INFO tasktracker.Localizer (Localizer.java:setPermissions(129)) - Thread[TaskLauncher for MAP tasks,5,main]: About to set permissions on /data/1/todd/cdh/repos/cdh3/hadoop-0.20/build/test/logs/userlogs/job_20110118090037816_0001 java.lang.Exception at org.apache.hadoop.mapreduce.server.tasktracker.Localizer$PermissionsHandler.setPermissions(Localizer.java:129) at org.apache.hadoop.mapreduce.server.tasktracker.Localizer.initializeJobLogDir(Localizer.java:429) at org.apache.hadoop.mapred.TaskTracker.initializeJobLogDir(TaskTracker.java:1072) at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:969) at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2209) 2011-01-18 09:00:40,985 INFO tasktracker.Localizer (Localizer.java:setPermissions(129)) - Thread[Thread-213,5,main]: About to set permissions on /data/1/todd/cdh/repos/cdh3/hadoop-0.20/build/test/logs/userlogs/job_20110118090037816_0001/attempt_20110118090037816_0001_m_05_0 java.lang.Exception at org.apache.hadoop.mapreduce.server.tasktracker.Localizer$PermissionsHandler.setPermissions(Localizer.java:129) at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:285) at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198) The above traces are from an 0.20 branch but I imagine it's the same deal on trunk. The issue is that the top invocation flips the job_id directory to 000 momentarily. During that time, the stat/chmod calls for the attempt directory fail with EACCES, which can leave the attempt directory with the wrong permissions. I have strace output which shows this as well. I think we should do away with this Java API nonsense altogether, link in a normal chmod call, and use fork by default when native isn't available. Undeletable build directories -- Key: MAPREDUCE-2238 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, test Affects Versions: 0.23.0 Reporter: Eli Collins Attachments: mapreduce-2238.txt The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir. https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console Building remotely on hadoop7 hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7 at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537) at hudson.model.AbstractProject.checkout(AbstractProject.java:1116) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411) at hudson.model.Run.run(Run.java:1324) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_00_0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12983421#action_12983421 ] Eli Collins commented on MAPREDUCE-2238: +1 latest patch looks good. Undeletable build directories -- Key: MAPREDUCE-2238 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, test Affects Versions: 0.22.0 Reporter: Eli Collins Assignee: Todd Lipcon Priority: Critical Fix For: 0.22.0 Attachments: mapreduce-2238.txt, mapreduce-2238.txt The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir. https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console Building remotely on hadoop7 hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7 at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537) at hudson.model.AbstractProject.checkout(AbstractProject.java:1116) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411) at hudson.model.Run.run(Run.java:1324) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_00_0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12982755#action_12982755 ] Todd Lipcon commented on MAPREDUCE-2238: Bummer, I had this patch in a branch on our internal Hudson and the problem happened anyway... so it doesn't look like it's an issue with thread safety on setPermissions Undeletable build directories -- Key: MAPREDUCE-2238 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, test Affects Versions: 0.23.0 Reporter: Eli Collins Attachments: mapreduce-2238.txt The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir. https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console Building remotely on hadoop7 hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7 at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537) at hudson.model.AbstractProject.checkout(AbstractProject.java:1116) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411) at hudson.model.Run.run(Run.java:1324) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_00_0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12982913#action_12982913 ] Todd Lipcon commented on MAPREDUCE-2238: Another clue on this one: root@ubuntu64-build01:/home/hudson/production/workspace/CDH3-todd-security/build/test# stat logs/userlogs/job_20110116195813351_0001/attempt_20110116195813351_0001_m_01_0/ File: `logs/userlogs/job_20110116195813351_0001/attempt_20110116195813351_0001_m_01_0/' Size: 4096Blocks: 8 IO Block: 4096 directory Device: 801h/2049d Inode: 4376820 Links: 2 Access: (0311/d-wx--x--x) Uid: ( 1065/ hudson) Gid: ( 1065/ hudson) Access: 2011-01-17 15:10:22.0 -0800 Modify: 2011-01-16 19:58:49.0 -0800 Change: 2011-01-16 19:58:49.0 -0800 Note the change time. Now looking at what tests ran during that change time: root@ubuntu64-build01:/home/hudson/production/workspace/CDH3-todd-security/build/test# grep -l '2011-01-16 20:58:4' * TEST-org.apache.hadoop.mapred.TestTaskLogsTruncater.xml Seems like TestTaskLogsTruncater is probably the culprit. Also suspicious is that that same test failed a test run a few builds prior to this. Undeletable build directories -- Key: MAPREDUCE-2238 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, test Affects Versions: 0.23.0 Reporter: Eli Collins Attachments: mapreduce-2238.txt The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir. https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console Building remotely on hadoop7 hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7 at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537) at hudson.model.AbstractProject.checkout(AbstractProject.java:1116) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411) at hudson.model.Run.run(Run.java:1324) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_00_0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12982915#action_12982915 ] Todd Lipcon commented on MAPREDUCE-2238: Oops scratch that, it's 19:58, not 20:58... so not TestTaskLogsTruncater. Interestingly, the tests running around that area are: TestJobStatusPersistency: Last log message at 19:58:09 TestJobTrackerInstrumentation: First log msg at 19:59:03 So we have a gap between those two. According to the test log, this test ran in between them: [junit] Running org.apache.hadoop.mapred.TestJobSysDirWithDFS [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 50.818 sec So I think *that* was the test that made the undeletable dir. Undeletable build directories -- Key: MAPREDUCE-2238 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, test Affects Versions: 0.23.0 Reporter: Eli Collins Attachments: mapreduce-2238.txt The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir. https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console Building remotely on hadoop7 hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7 at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537) at hudson.model.AbstractProject.checkout(AbstractProject.java:1116) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411) at hudson.model.Run.run(Run.java:1324) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_00_0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12981520#action_12981520 ] Todd Lipcon commented on MAPREDUCE-2238: Saw this again on a build here. This time the undeletable userlog directory was created by TestMiniMRWithDFSWithDistinctUsers. Undeletable build directories -- Key: MAPREDUCE-2238 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, test Affects Versions: 0.23.0 Reporter: Eli Collins The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir. https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console Building remotely on hadoop7 hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.Channel@2545938c:hadoop7 at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537) at hudson.model.AbstractProject.checkout(AbstractProject.java:1116) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411) at hudson.model.Run.run(Run.java:1324) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_00_0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12978209#action_12978209 ] Todd Lipcon commented on MAPREDUCE-2238: I guess it could be a test timing out right as a setPermissions is done, interrupting in the middle... but seems pretty unlikely, don't you think? I agree we could work around it for the tests, but I'm nervous whether we will see this issue crop up in production. Have you guys at Yahoo seen this on any clusters running secure YDH? Undeletable build directories -- Key: MAPREDUCE-2238 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, test Affects Versions: 0.23.0 Reporter: Eli Collins The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir. https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console Building remotely on hadoop7 hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.chan...@2545938c:hadoop7 at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537) at hudson.model.AbstractProject.checkout(AbstractProject.java:1116) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411) at hudson.model.Run.run(Run.java:1324) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_00_0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12978222#action_12978222 ] Todd Lipcon commented on MAPREDUCE-2238: Linked MAPREDUCE-1972 which has some logs which might be related Undeletable build directories -- Key: MAPREDUCE-2238 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, test Affects Versions: 0.23.0 Reporter: Eli Collins The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir. https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console Building remotely on hadoop7 hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.chan...@2545938c:hadoop7 at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537) at hudson.model.AbstractProject.checkout(AbstractProject.java:1116) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411) at hudson.model.Run.run(Run.java:1324) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_00_0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12978551#action_12978551 ] Greg Roelofs commented on MAPREDUCE-2238: - bq. I guess it could be a test timing out right as a setPermissions is done, interrupting in the middle... but seems pretty unlikely, don't you think? Yes. I'm guessing it's more subtle than that and lies within the core MR code or the JVM. The fact that I see it semi-frequently on NFS (that is, more frequent than Hudson or production) suggests either timing (NFS is slow), perhaps via an erroneous assumption of synchronous behavior, or else an erroneous assumption of an infallible system call. It could be other things as well, of course, but those seem to me like the most probable candidates. bq. I agree we could work around it for the tests, but I'm nervous whether we will see this issue crop up in production. Have you guys at Yahoo seen this on any clusters running secure YDH? To clarify, I was suggesting working around it in the MR code itself, not realizing that the Hudson backtrace wasn't using MR code at all. (Well, apparently.) So I'm not sure where that leaves us, other than trying to fix the actual set-permissions problem. Seems like no one's basic deleteRecursive() implementation includes an option to attempt a chmod() before failing on bad permissions? Anyway, yes, I _think_ we've seen it in production with 0.20S or later, but it wasn't while I was on call, so I might be remembering a different issue with similar symptoms. Sorry...there are lots of interesting failure modes in Hadoop, and my memory is finite. :-) Undeletable build directories -- Key: MAPREDUCE-2238 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, test Affects Versions: 0.23.0 Reporter: Eli Collins The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir. https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console Building remotely on hadoop7 hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.chan...@2545938c:hadoop7 at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537) at hudson.model.AbstractProject.checkout(AbstractProject.java:1116) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411) at hudson.model.Run.run(Run.java:1324) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_00_0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12978561#action_12978561 ] Todd Lipcon commented on MAPREDUCE-2238: I think we should do the following: - change out the implementation of setPermissions to not use the Java APIs, but rather the old style system(chmod) approach - implement a proper chmod call in libhadoop (JNI) to avoid the fork for production systems where the fork is too expensive bq. Sorry...there are lots of interesting failure modes in Hadoop, and my memory is finite :) But that's why Hadoop's so much fun to work on! If it just worked all the time we'd be bored. Undeletable build directories -- Key: MAPREDUCE-2238 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, test Affects Versions: 0.23.0 Reporter: Eli Collins The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir. https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console Building remotely on hadoop7 hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.chan...@2545938c:hadoop7 at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537) at hudson.model.AbstractProject.checkout(AbstractProject.java:1116) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411) at hudson.model.Run.run(Run.java:1324) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_00_0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12978593#action_12978593 ] Greg Roelofs commented on MAPREDUCE-2238: - bq. But that's why Hadoop's so much fun to work on! If it just worked all the time we'd be bored. We are entirely in agreement. :-) Undeletable build directories -- Key: MAPREDUCE-2238 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, test Affects Versions: 0.23.0 Reporter: Eli Collins The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir. https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console Building remotely on hadoop7 hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.chan...@2545938c:hadoop7 at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537) at hudson.model.AbstractProject.checkout(AbstractProject.java:1116) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411) at hudson.model.Run.run(Run.java:1324) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_00_0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12978109#action_12978109 ] Greg Roelofs commented on MAPREDUCE-2238: - Race condition/lack of locking? FWIW, I've seen this exact same bug several times when running ant test on an NFS tree. (IIRC, it screws up all subsequent tests, too, until the bogus directory is manually removed or chmod'd.) If the bug proves too elusive--and it's definitely intermittent and not at all frequent--a hackish workaround would be just to catch the IOException, attempt a chmod 700, and reattempt the deletion. Undeletable build directories -- Key: MAPREDUCE-2238 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, test Affects Versions: 0.23.0 Reporter: Eli Collins The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir. https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console Building remotely on hadoop7 hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.chan...@2545938c:hadoop7 at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537) at hudson.model.AbstractProject.checkout(AbstractProject.java:1116) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411) at hudson.model.Run.run(Run.java:1324) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_00_0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12977492#action_12977492
]
Nigel Daley commented on MAPREDUCE-2238:
Hudson slave runs as the user 'hudson'. Here is the directory listing that is
failing. Why would the permissions on the dir be changed to 311?
{code}
r...@h7:/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001#
ls -al
total 36
drwx-- 9 hudson hudson 4096 2010-12-30 13:14 .
drwxr-xr-x 3 hudson hudson 4096 2010-12-30 13:14 ..
d-wx--x--x 2 hudson hudson 4096 2010-12-30 13:11
attempt_20101230131139886_0001_m_00_0
drwx-- 2 hudson hudson 4096 2010-12-30 13:12
attempt_20101230131139886_0001_m_00_1
drwx-- 2 hudson hudson 4096 2010-12-30 13:11
attempt_20101230131139886_0001_m_01_0
drwx-- 2 hudson hudson 4096 2010-12-30 13:12
attempt_20101230131139886_0001_m_03_1
drwx-- 2 hudson hudson 4096 2010-12-30 13:11
attempt_20101230131139886_0001_m_05_0
drwx-- 2 hudson hudson 4096 2010-12-30 13:12
attempt_20101230131139886_0001_m_06_0
drwx-- 2 hudson hudson 4096 2010-12-30 13:11
attempt_20101230131139886_0001_m_07_0
r...@h7:/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001#
ls -al attempt_20101230131139886_0001_m_00_0
total 16
d-wx--x--x 2 hudson hudson 4096 2010-12-30 13:11 .
drwx-- 9 hudson hudson 4096 2010-12-30 13:14 ..
-rw-r--r-- 1 hudson hudson 205 2010-12-30 13:11 log.index
-rw-r--r-- 1 hudson hudson0 2010-12-30 13:11 stderr
-rw-r--r-- 1 hudson hudson 1687 2010-12-30 13:11 stdout
{code}
Undeletable build directories
--
Key: MAPREDUCE-2238
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build, test
Affects Versions: 0.23.0
Reporter: Eli Collins
The MR hudson job is failing, looks like it's due to a test chmod'ing a build
directory so the checkout can't clean the build dir.
https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console
Building remotely on hadoop7
hudson.util.IOException2: remote file operation failed:
/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at
hudson.remoting.chan...@2545938c:hadoop7
at hudson.FilePath.act(FilePath.java:749)
at hudson.FilePath.act(FilePath.java:735)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1116)
at
hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
at hudson.model.Run.run(Run.java:1324)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:139)
Caused by: java.io.IOException: Unable to delete
/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_00_0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2238) Undeletable build directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12977503#action_12977503 ] Todd Lipcon commented on MAPREDUCE-2238: My hunch is that this is caused by some kind of bug in the new Localizer.PermissionsHandler.setPermissions originally introduced by MAPREDUCE-842. Undeletable build directories -- Key: MAPREDUCE-2238 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2238 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, test Affects Versions: 0.23.0 Reporter: Eli Collins The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir. https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/549/console Building remotely on hadoop7 hudson.util.IOException2: remote file operation failed: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk at hudson.remoting.chan...@2545938c:hadoop7 at hudson.FilePath.act(FilePath.java:749) at hudson.FilePath.act(FilePath.java:735) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:589) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:537) at hudson.model.AbstractProject.checkout(AbstractProject.java:1116) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411) at hudson.model.Run.run(Run.java:1324) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) Caused by: java.io.IOException: Unable to delete /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/test/logs/userlogs/job_20101230131139886_0001/attempt_20101230131139886_0001_m_00_0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
