[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title
Elmar Weber edited a comment on JENKINS-35246
Re: Kubernetes nodes not getting deleted when a pipeline job fails
We build a simple scriptlet that we scheduled to take care of this, nothing fancy, in case anyone looks for something similar:{{for (aSlave in hudson.model.Hudson.instance.slaves) \{}}{{ if (aSlave.getComputer().isOffline()) \{}} } {{ println(aSlave.name + ' Deleted');}} } {{ aSlave.getComputer().doDoDelete();}} } {{ }}}{{ }}}From [ http://stackoverflow.com/questions/24072354/jenkins-is-there-a-way-to-remove-all-offline-nodes-slaves-batch-remove-nod ]
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title
Elmar Weber edited a comment on JENKINS-35246
Re: Kubernetes nodes not getting deleted when a pipeline job fails
We build a simple scriptlet that we scheduled to take care of this, nothing fancy, in case anyone looks for something similar: \{code} {{ for (aSlave in hudson.model.Hudson.instance.slaves) \{ }} {{ if (aSlave.getComputer().isOffline()) \{ }} {{ println(aSlave.name + ' Deleted'); }} {{ aSlave.getComputer().doDoDelete(); }} {{ } } } \ { code { } }}From http://stackoverflow.com/questions/24072354/jenkins-is-there-a-way-to-remove-all-offline-nodes-slaves-batch-remove-nod
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title
Elmar Weber commented on JENKINS-35246
Re: Kubernetes nodes not getting deleted when a pipeline job fails
We build a simple scriptlet that we scheduled to take care of this, nothing fancy, in case anyone looks for something similar: {code} for (aSlave in hudson.model.Hudson.instance.slaves) { if (aSlave.getComputer().isOffline()) { println(aSlave.name + ' Deleted'); aSlave.getComputer().doDoDelete(); } } {code}
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title Elmar Weber commented on JENKINS-35246 Re: Kubernetes nodes not getting deleted when a pipeline job fails Hello, is there any update on this or an idea where to check? If there is anything we can do to help debug this or how to start looking to fix this would be great, we currently considering writing a script that cleans up these nodes, we may as well invest the effort in fixing it in the plugin =) a quick pointer of where to start looking would be great. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title Elmar Weber commented on JENKINS-35246 Re: Kubernetes nodes not getting deleted when a pipeline job fails To clarify: The pods are terminated and deleted in kubernetes the jenkins slave reference of them in Jenkins is not deleted when they are terminated and deleted in kubernetes Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title Elmar Weber commented on JENKINS-35246 Re: Kubernetes nodes not getting deleted when a pipeline job fails I forgot: They get however cleaned up after a while, I think around 12h or so automatically. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title Elmar Weber commented on JENKINS-35246 Re: Kubernetes nodes not getting deleted when a pipeline job fails Carlos Sanchez Sebastien Vas I have the same issue, on our side it does not seem to relate to a specific command but from what we gathered it happens when the job on the node (within a pipeline) takes less than 1-2 seconds. In that case the reference to the node does not seem to get cleaned from Jenkins. If the node actions we are scripting take longer it is cleanly removed from jenkins. It always get's removed from the kubernetes cluster when it ends in this state. Furthermore the build image we are using is rather large, around 1.5GB. On smaller ones we could not reproduce this behaviour. According to the Jenkins logs certain checks / actions are only done once a second, clould this be related to why the proper shutdown / kill of the pod is not realized within the plugin? I attached a screenshot of the situation. The log entries for all pods that end up like this follow the same pattern as below: Nov 10, 2016 12:13:40 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call INFO: Created Pod: cupenya-root-docker-1460271f3921 Nov 10, 2016 12:13:40 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call INFO: Waiting for Pod to be scheduled (0/100): cupenya-root-docker-1460271f3921 -- Nov 10, 2016 12:13:50 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate INFO: Terminating Kubernetes instance for slave cupenya-root-docker-1460271f3921 -- Nov 10, 2016 12:13:50 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate INFO: Terminated Kubernetes instance for slave cupenya-root-docker-1460271f3921 Nov 10, 2016 12:13:50 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate INFO: Disconnected computer cupenya-root-docker-1460271f3921 Nov 10, 2016 12:13:50 PM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed WARNING: Computer.threadPoolForRemoting [#138] for cupenya-root-docker-1460271f3921 terminated -- Nov 10, 2016 12:18:59 PM hudson.node_monitors.ResponseTimeMonitor$1 monitor WARNING: Making cupenya-root-docker-1460271f3921 offline because it’s not responding -- Nov 10, 2016 12:19:19 PM hudson.node_monitors.ResponseTimeMonitor$1 monitor WARNING: Making cupenya-root-docker-1460271f3921 offline because it’s not responding -- Nov 10, 2016 12:20:39 PM hudson.node_monitors.ResponseTimeMonitor$1 monitor WARNING: Making cupenya-root-docker-1460271f3921 offline because it’s not responding -- Nov 10, 2016 12:20:49 PM hudson.node_monitors.ResponseTimeMonitor$1 monitor WARNING: Making cupenya-root-docker-1460271f3921 offline because it’s not responding -- Nov 10, 2016 12:21:29 PM hudson.node_monitors.ResponseTimeMonitor$1 monitor WARNING: Making cupenya-root-docker-1460271f3921 offline because it’s not responding -- You can maybe try to reproduce it with this image as it is larger, in case that is the issue: cupenya/docker-jenkins-slave-cpy-root https://github.com/cupenya/docker-jenkins-slave-cpy-root/blob/master/Dockerfile I'm happy to provide more details or in case you can't reproduce this in your environment show you a live example when this happens.
[JIRA] (JENKINS-35246) Kubernetes nodes not getting deleted when a pipeline job fails
Title: Message Title Elmar Weber updated an issue Jenkins / JENKINS-35246 Kubernetes nodes not getting deleted when a pipeline job fails Change By: Elmar Weber Attachment: suspended_pods.png Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-22853) SEVERE: Trying to unexport an object that's already unexported
Elmar Weber commented on JENKINS-22853 SEVERE: Trying to unexport an object that's already unexported Same issue here, I also cannot upgrade to anything beyond 1.558, I'm getting the message reproducible on archiving artifacts from a remote build bia an agent. Only seems to happen for larger files though. Small projects run without issues for the most time, but larger ones (several artifacts with ~20-30MB) got stuck on "Archiving artifacts" with this message in the log. I have then to restart Jenkins to fix the slave. I don't have the time right now to do extensive tests / debug (after next week again), but I can quickly test stuff with a new war file if that helps. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
