[ https://issues.jenkins-ci.org/browse/JENKINS-10905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=161682#comment-161682 ]
pjdarton edited comment on JENKINS-10905 at 4/17/12 2:24 PM: ------------------------------------------------------------- I'm glad it's not just me - I keep getting these as well. In my case, it's from the Accurev plugin calling hudson.Util.deleteContentsRecursive and getting spurious errors from the filesystem. I believe that the underlying cause is an operating system bug, either that or a very common misconfiguration, as I (sometimes) get the same problems doing "rd /s /q xxx" from a CMD window, and (sometimes) find that Explorer fails to delete everything when doing a Shift+Delete on directories. i.e. I think that Windows sometimes locks files briefly, causing deletes to fail. Note: I've configured the Windows Search service to leave well alone, and also the anti-virus, so I don't think it's them - I've had these problems from both of those on previous versions of Windows, but on Windows 7 it doesn't seem to be enough. However, regardless of the cause of the problem, the workaround is probably to delete everything that can be deleted, and to retry a number of times until either all files are deleted, or we've exhausted the retries. I wouldn't call it a blocker though - just triggering the build again usually makes the problem disappear. i.e. it's as annoying as hell, but it isn't a blocker. Note: JENKINS-3053 appears to be reporting the same problem. was (Author: pjdarton): I'm glad it's not just me - I keep getting these as well. In my case, it's from the Accurev plugin calling hudson.Util.deleteContentsRecursive and getting spurious errors from the filesystem. I believe that the underlying cause is an operating system bug, either that or a very common misconfiguration, as I (sometimes) get the same problems doing "rd /s /q xxx" from a CMD window, and (sometimes) find that Explorer fails to delete everything when doing a Shift+Delete on directories. i.e. I think that Windows sometimes locks files briefly, causing deletes to fail. Note: I've configured the Windows Search service to leave well alone, and also the anti-virus, so I don't think it's them - I've had these problems from both of those on previous versions of Windows, but on Windows 7 it doesn't seem to be enough. However, regardless of the cause of the problem, the workaround is probably to delete everything that can be deleted, and to retry a number of times until either all files are deleted, or we've exhausted the retries. I wouldn't call it a blocker though - just triggering the build again usually makes the problem disappear. i.e. it's as annoying as hell, but it isn't a blocker. Note: JENKINS-3052 appears to be reporting the same problem. > Deleting previous project files regularly fails the build > --------------------------------------------------------- > > Key: JENKINS-10905 > URL: https://issues.jenkins-ci.org/browse/JENKINS-10905 > Project: Jenkins > Issue Type: Bug > Components: core > Affects Versions: current > Environment: Window 7 operating in master/slave environment (the > failing machine is the slave) > Reporter: nyoung02 > Priority: Blocker > > I have a project that is quite large. I'm not sure if it's coincidental or > not, but since upgrading to 1.428, I've been getting a number of intermittent > build failures, that actually fail in the cleanup stage deleting files from > the previous build. My svn settings are set to 'clean checkout' although I > also get this problem if I select 'revert and update'. Is it possible to make > this stage more tolerant of locked files (perhaps by increasing the number of > retries?) > 09:51:40 Started by user niy > 09:51:40 Building remotely on nwb-r5win32b > 09:51:40 Cleaning workspace C:\jenkins\workspace\R5_Win_Overnight > 09:52:29 hudson.util.IOException2: remote file operation failed: > c:\jenkins\workspace\R5_Win_Overnight at > hudson.remoting.Channel@470d6c27:nwb-r5win32b > 09:52:29 at hudson.FilePath.act(FilePath.java:754) > 09:52:29 at hudson.FilePath.act(FilePath.java:740) > 09:52:29 at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:731) > 09:52:29 at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:676) > 09:52:29 at > hudson.model.AbstractProject.checkout(AbstractProject.java:1193) > 09:52:29 at > hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:555) > 09:52:29 at > hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:443) > 09:52:29 at hudson.model.Run.run(Run.java:1376) > 09:52:29 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) > 09:52:29 at > hudson.model.ResourceController.execute(ResourceController.java:88) > 09:52:29 at hudson.model.Executor.run(Executor.java:230) > 09:52:29 Caused by: java.io.IOException: Unable to delete > c:\jenkins\workspace\R5_Win_Overnight\viscob\coretech\checker\src\tools\.svn > - files in dir: > [c:\jenkins\workspace\R5_Win_Overnight\viscob\coretech\checker\src\tools\.svn\tmp] > 09:52:29 at hudson.Util.deleteFile(Util.java:265) > 09:52:29 at hudson.Util.deleteRecursive(Util.java:316) > 09:52:29 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:52:29 at hudson.Util.deleteRecursive(Util.java:307) > 09:52:29 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:52:29 at hudson.Util.deleteRecursive(Util.java:307) > 09:52:29 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:52:29 at hudson.Util.deleteRecursive(Util.java:307) > 09:52:29 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:52:29 at hudson.Util.deleteRecursive(Util.java:307) > 09:52:29 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:52:29 at hudson.Util.deleteRecursive(Util.java:307) > 09:52:29 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:52:29 at > hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:74) > 09:52:29 at > hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:136) > 09:52:29 at > hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:773) > 09:52:29 at > hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:754) > 09:52:29 at > hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:738) > 09:52:29 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1994) > 09:52:29 at hudson.remoting.UserRequest.perform(UserRequest.java:118) > 09:52:29 at hudson.remoting.UserRequest.perform(UserRequest.java:48) > 09:52:29 at hudson.remoting.Request$2.run(Request.java:287) > 09:52:29 at java.util.concurrent.Executors$RunnableAdapter.call(Unknown > Source) > 09:52:29 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) > 09:52:29 at java.util.concurrent.FutureTask.run(Unknown Source) > 09:52:29 at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) > 09:52:29 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > 09:52:29 at hudson.remoting.Engine$1$1.run(Engine.java:60) > 09:52:29 at java.lang.Thread.run(Unknown Source) > 09:52:29 Retrying after 10 seconds > 09:52:39 Cleaning workspace C:\jenkins\workspace\R5_Win_Overnight > 09:55:02 hudson.util.IOException2: remote file operation failed: > c:\jenkins\workspace\R5_Win_Overnight at > hudson.remoting.Channel@470d6c27:nwb-r5win32b > 09:55:02 at hudson.FilePath.act(FilePath.java:754) > 09:55:02 at hudson.FilePath.act(FilePath.java:740) > 09:55:02 at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:731) > 09:55:02 at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:676) > 09:55:02 at > hudson.model.AbstractProject.checkout(AbstractProject.java:1193) > 09:55:02 at > hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:555) > 09:55:02 at > hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:443) > 09:55:02 at hudson.model.Run.run(Run.java:1376) > 09:55:02 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) > 09:55:02 at > hudson.model.ResourceController.execute(ResourceController.java:88) > 09:55:02 at hudson.model.Executor.run(Executor.java:230) > 09:55:02 Caused by: java.io.IOException: Unable to delete > c:\jenkins\workspace\R5_Win_Overnight\viscob\eclipseide\feature.builder\core\build\plugins\com.microfocus.eclipse.core\@dot\com\microfocus\eclipse\core > - files in dir: > [c:\jenkins\workspace\R5_Win_Overnight\viscob\eclipseide\feature.builder\core\build\plugins\com.microfocus.eclipse.core\@dot\com\microfocus\eclipse\core\wizards] > 09:55:02 at hudson.Util.deleteFile(Util.java:265) > 09:55:02 at hudson.Util.deleteRecursive(Util.java:316) > 09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:02 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:02 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:02 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:02 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:02 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:02 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:02 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:02 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:02 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:02 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:02 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:02 at > hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:74) > 09:55:02 at > hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:136) > 09:55:02 at > hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:773) > 09:55:02 at > hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:754) > 09:55:02 at > hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:738) > 09:55:02 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1994) > 09:55:02 at hudson.remoting.UserRequest.perform(UserRequest.java:118) > 09:55:02 at hudson.remoting.UserRequest.perform(UserRequest.java:48) > 09:55:02 at hudson.remoting.Request$2.run(Request.java:287) > 09:55:02 at java.util.concurrent.Executors$RunnableAdapter.call(Unknown > Source) > 09:55:02 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) > 09:55:02 at java.util.concurrent.FutureTask.run(Unknown Source) > 09:55:02 at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) > 09:55:02 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > 09:55:02 at hudson.remoting.Engine$1$1.run(Engine.java:60) > 09:55:02 at java.lang.Thread.run(Unknown Source) > 09:55:02 Retrying after 10 seconds > 09:55:12 Cleaning workspace C:\jenkins\workspace\R5_Win_Overnight > 09:55:13 hudson.util.IOException2: remote file operation failed: > c:\jenkins\workspace\R5_Win_Overnight at > hudson.remoting.Channel@470d6c27:nwb-r5win32b > 09:55:13 at hudson.FilePath.act(FilePath.java:754) > 09:55:13 at hudson.FilePath.act(FilePath.java:740) > 09:55:13 at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:731) > 09:55:13 at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:676) > 09:55:13 at > hudson.model.AbstractProject.checkout(AbstractProject.java:1193) > 09:55:13 at > hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:555) > 09:55:13 at > hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:443) > 09:55:13 at hudson.model.Run.run(Run.java:1376) > 09:55:13 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) > 09:55:13 at > hudson.model.ResourceController.execute(ResourceController.java:88) > 09:55:13 at hudson.model.Executor.run(Executor.java:230) > 09:55:13 Caused by: java.io.IOException: Unable to delete > c:\jenkins\workspace\R5_Win_Overnight\viscob\eclipseide\feature.builder\core\build\plugins\com.microfocus.eclipse.core\src\com\microfocus\eclipse\core > - files in dir: > [c:\jenkins\workspace\R5_Win_Overnight\viscob\eclipseide\feature.builder\core\build\plugins\com.microfocus.eclipse.core\src\com\microfocus\eclipse\core\wizards] > 09:55:13 at hudson.Util.deleteFile(Util.java:265) > 09:55:13 at hudson.Util.deleteRecursive(Util.java:316) > 09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:13 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:13 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:13 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:13 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:13 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:13 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:13 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:13 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:13 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:13 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:13 at hudson.Util.deleteRecursive(Util.java:307) > 09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227) > 09:55:13 at > hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:74) > 09:55:13 at > hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:136) > 09:55:13 at > hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:773) > 09:55:13 at > hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:754) > 09:55:13 at > hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:738) > 09:55:13 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1994) > 09:55:13 at hudson.remoting.UserRequest.perform(UserRequest.java:118) > 09:55:13 at hudson.remoting.UserRequest.perform(UserRequest.java:48) > 09:55:13 at hudson.remoting.Request$2.run(Request.java:287) > 09:55:13 at java.util.concurrent.Executors$RunnableAdapter.call(Unknown > Source) > 09:55:13 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) > 09:55:13 at java.util.concurrent.FutureTask.run(Unknown Source) > 09:55:13 at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) > 09:55:13 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > 09:55:13 at hudson.remoting.Engine$1$1.run(Engine.java:60) > 09:55:13 at java.lang.Thread.run(Unknown Source) > 09:55:13 Archiving artifacts > 10:12:03 Recording fingerprints > 10:12:03 Email was triggered for: Failure > 10:12:03 Sending email for trigger: Failure -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira