[JIRA] (JENKINS-37293) ECS Agents are not deleted because Jenkins is unable to delete logs due to NFS locks
Title: Message Title Philipp Garbe closed an issue as Done No feedback since months Jenkins / JENKINS-37293 ECS Agents are not deleted because Jenkins is unable to delete logs due to NFS locks Change By: Philipp Garbe Status: Open Closed Resolution: Done Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37293) ECS Agents are not deleted because Jenkins is unable to delete logs due to NFS locks
Title: Message Title Evan Borgstrom commented on JENKINS-37293 Re: ECS Agents are not deleted because Jenkins is unable to delete logs due to NFS locks Just upgraded from 1.4 to 1.5. The issue persists. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37293) ECS Agents are not deleted because Jenkins is unable to delete logs due to NFS locks
Title: Message Title Evan Borgstrom updated an issue Jenkins / JENKINS-37293 ECS Agents are not deleted because Jenkins is unable to delete logs due to NFS locks Change By: Evan Borgstrom Environment: jenkins 2.7.2amazon-ecs 1. 4 5 Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37293) ECS Agents are not deleted because Jenkins is unable to delete logs due to NFS locks
Title: Message Title Evan Borgstrom commented on JENKINS-37293 Re: ECS Agents are not deleted because Jenkins is unable to delete logs due to NFS locks This just happened again, so the noatime,nodiratime work around doesn't actually work... Aug 12, 2016 9:03:29 AM WARNING hudson.slaves.SlaveComputer kill Unable to delete agent logs java.io.IOException: Unable to delete '/var/lib/jenkins/logs/slaves/ecs-1aaa9aed01d09'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. ... Caused by: java.nio.file.FileSystemException: /var/lib/jenkins/logs/slaves/ecs-1aaa9aed01d09/.nfs8f2ea0ecf5e622fd0022: Device or resource busy Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37293) ECS Agents are not deleted because Jenkins is unable to delete logs due to NFS locks
Title: Message Title Evan Borgstrom commented on JENKINS-37293 Re: ECS Agents are not deleted because Jenkins is unable to delete logs due to NFS locks FWIW, I remounted our EFS files system with the noatime,nodiratime options today and it seems to be a solid workaround for the issue, slaves are being cleaned up as expected. While I intend to leave those mount options on, this should still be fixed. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37293) ECS Agents are not deleted because Jenkins is unable to delete logs due to NFS locks
Title: Message Title Evan Borgstrom created an issue Jenkins / JENKINS-37293 ECS Agents are not deleted because Jenkins is unable to delete logs due to NFS locks Issue Type: Bug Assignee: Jan Roehrich Components: amazon-ecs-plugin Created: 2016/Aug/09 11:41 PM Environment: jenkins 2.7.2 amazon-ecs 1.4 Labels: nfs logs lock Priority: Major Reporter: Evan Borgstrom We have our Jenkins master running on a NFS 4.1 mount (AWS EFS). When a job completes we see the following in the logs: Aug 09, 2016 4:21:09 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSService deleteTask Delete ECS Slave task: arn:aws:ecs:us-east-1::task/80b3bc38-774c-4bbe-9e79-f9db126a2b9b Aug 09, 2016 4:21:09 PM WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor Failed to monitor jenkins-ecs-d6d51f89f024 for Free Disk Space java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: hudson.remoting.Channel$OrderlyShutdown at hudson.remoting.Request$1.get(Request.java:282) at hudson.remoting.Request$1.get(Request.java:207) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: hudson.remoting.RequestAbortedException:
