[JIRA] (JENKINS-53045) Plugin trips FARGATE rate limit.
Title: Message Title Mike Kozell assigned an issue to Philipp Garbe Jenkins / JENKINS-53045 Plugin trips FARGATE rate limit. Change By: Mike Kozell Assignee: Jan Roehrich Philipp Garbe Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-56377) console log hangs in web browser after upgrade
Title: Message Title Mike Kozell created an issue Jenkins / JENKINS-56377 console log hangs in web browser after upgrade Issue Type: Bug Assignee: Unassigned Components: workflow-job-plugin Created: 2019-03-04 03:16 Environment: Jenkins v2.89.4 w/ workflow-job-plugin v2.25 Jenkins v2.150.3 w/ workflow-job-plugin v2.31 Priority: Minor Reporter: Mike Kozell We recently upgraded Jenkins from 2.89.4 to 2.150.3 and then upgraded the workflow-job plugin from 2.25 to 2.31 and are now seeing an issue where the full console log in the Jenkins UI is having issues loading. We run a parallel pipeline heavy workload which results in a 13mb console log in 2.150.3 and a 46mb console log in 2.89.4. In Jenkins 2.89.4, it took 45 seconds to load the full console log file in Chrome and even less time in Firefox. In Jenkins 2.150.3 the log file took 2m 30sec in Chrome and after 5 minutes never loaded in Firefox. I kept seeing "A web page is slowing down your browser. What would you like to do? [Stop It, Wait]" and eventually Firefox became unresponsive. I used Chrome 72 and Firefox 65 in my testing.
[JIRA] (JENKINS-47724) Pipeline with parallel jobs performance issue with EC2 slaves
Title: Message Title Mike Kozell commented on JENKINS-47724 Re: Pipeline with parallel jobs performance issue with EC2 slaves Jesse Glick I added -Dorg.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep.USE_WATCHING=true but it didn't seem to make a difference. Duration Branches Running Agents Connection Type 56s 190 6 trilead SSH 8m 3s 190 192 trilead SSH I'm also seeing a spike in load at the end of the build with and without the parameter. I should probably create new issue for it. The load average went to over 25 on the VM after the date command finished. In the build log, I saw: [Pipeline] } ... Add Comment
[JIRA] (JENKINS-55308) intermittent "terminated" messages using sh in Pipelines
Title: Message Title Mike Kozell commented on JENKINS-55308 Re: intermittent "terminated" messages using sh in Pipelines I upgraded Pipeline: Job (workflow-job) plugin from 2.25 to 2.31 and now see the issue. I'm running durable-task 1.27. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-54111) disableDeferredWipeout error with workspace cleanup plugin 0.35
Title: Message Title
Mike Kozell updated an issue
Jenkins / JENKINS-54111
disableDeferredWipeout error with workspace cleanup plugin 0.35
Change By:
Mike Kozell
After installing ws-cleanup plugin v0.35, I'm seeing the stack trace below in the jenkins.log file. In addition, when I go to the Pipeline Syntax page and select "CleanWs", I see the following for the code snippet. * step * Using v0.34 of the plugin, I see " * cleanWs() * "{code:java}WARNING: failed to uncoerce hudson.plugins.ws_cleanup.WsCleanup@3ec5cba9java.lang.UnsupportedOperationException: no public field ‘disableDeferredWipeout’ (or getter method) found in class hudson.plugins.ws_cleanup.WsCleanupat org.jenkinsci.plugins.structs.describable.DescribableParameter.getValue(DescribableParameter.java:161)at org.jenkinsci.plugins.structs.describable.DescribableParameter.inspect(DescribableParameter.java:142)at org.jenkinsci.plugins.structs.describable.DescribableModel.uninstantiate2(DescribableModel.java:584)at org.jenkinsci.plugins.structs.describable.DescribableModel.uninstantiate2_(DescribableModel.java:675)at org.jenkinsci.plugins.structs.describable.DescribableParameter.uncoerce(DescribableParameter.java:196)at org.jenkinsci.plugins.structs.describable.DescribableParameter.inspect(DescribableParameter.java:142)at org.jenkinsci.plugins.structs.describable.DescribableModel.uninstantiate2(DescribableModel.java:584)at org.jenkinsci.plugins.structs.describable.DescribableModel.uninstantiate2_(DescribableModel.java:675)at org.jenkinsci.plugins.workflow.steps.StepDescriptor.uninstantiate(StepDescriptor.java:246)at org.jenkinsci.plugins.workflow.cps.Snippetizer.object2Groovy(Snippetizer.java:139)at org.jenkinsci.plugins.workflow.cps.Snippetizer.step2Groovy(Snippetizer.java:81)at org.jenkinsci.plugins.workflow.cps.Snippetizer.doGenerateSnippet(Snippetizer.java:513)at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129)at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:734)at org.kohsuke.stapler.Stapler.invoke(Stapler.java:864)at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:734)at org.kohsuke.stapler.Stapler.invoke(Stapler.java:864)at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)at org.kohsuke.stapler.Stapler.service(Stapler.java:238){code} Both of these issues didn't occur for me in 0.34 version of the plugin.
[JIRA] (JENKINS-54111) disableDeferredWipeout error with workspace cleanup plugin 0.35
Title: Message Title Mike Kozell created an issue Jenkins / JENKINS-54111 disableDeferredWipeout error with workspace cleanup plugin 0.35 Issue Type: Bug Assignee: Oliver Gondža Components: workflow-cps-plugin, ws-cleanup-plugin Created: 2018-10-16 18:52 Environment: Jenkins 2.138.2 ws-cleanup 0.35 Priority: Major Reporter: Mike Kozell After installing ws-cleanup plugin v0.35, I'm seeing the stack trace below in the jenkins.log file. In addition, when I go to the Pipeline Syntax page and select "CleanWs", I see the following for the code snippet. step Using v0.34 of the plugin, I see "cleanWs()" WARNING: failed to uncoerce hudson.plugins.ws_cleanup.WsCleanup@3ec5cba9 java.lang.UnsupportedOperationException: no public field ‘disableDeferredWipeout’ (or getter method) found in class hudson.plugins.ws_cleanup.WsCleanup at org.jenkinsci.plugins.structs.describable.DescribableParameter.getValue(DescribableParameter.java:161) at org.jenkinsci.plugins.structs.describable.DescribableParameter.inspect(DescribableParameter.java:142) at org.jenkinsci.plugins.structs.describable.DescribableModel.uninstantiate2(DescribableModel.java:584) at org.jenkinsci.plugins.structs.describable.DescribableModel.uninstantiate2_(DescribableModel.java:675) at org.jenkinsci.plugins.structs.describable.DescribableParameter.uncoerce(DescribableParameter.java:196) at org.jenkinsci.plugins.structs.describable.DescribableParameter.inspect(DescribableParameter.java:142) at org.jenkinsci.plugins.structs.describable.DescribableModel.uninstantiate2(DescribableModel.j
[JIRA] (JENKINS-34256) Preparing Jenkins For Shutdown Hangs Running Pipelines
Title: Message Title Mike Kozell commented on JENKINS-34256 Re: Preparing Jenkins For Shutdown Hangs Running Pipelines Sam Van Oort we just came across this issue. Using the latest Jenkins pipeline plugins, we had a few shared groovy pipeline builds running when I clicked the "Prepare for shutdown" button. The running pipeline builds were running "steps.sh" commands. After waiting a long time, I logged into the Jenkins agents and discovered the processes had finished and the builds were just sitting there. I waited longer and finally clicked the pipeline "pause / resume" button twice which failed the build. I reviewed the Jenkins agents and didn't find anything unusual. The commands running inside the steps.sh() seemed to exit without issue. I checked the jenkins.log file on the master and didn't see any errors until I clicked pause / resume. Unfortunately this issue prevents us from being able to gracefully shutdown a busy Jenkins master to perform maintenance, upgrades, or even decommission. Below is the stack trace from the Jenkins log file after pause / resume was pressed on the hung builds. Sep 10, 2018 9:00:24 PM org.jenkinsci.plugins.workflow.cps.CpsStepContext completed WARNING: already completed CpsStepContext[14:timeout]:OwnerJenkins-Job-Name/1:Jenkins-Job-Name #1 java.lang.IllegalStateException: delivered here at org.jenkinsci.plugins.workflow.cps.CpsStepContext.completed(CpsStepContext.java:353) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.onSuccess(CpsStepContext.java:333) at org.jenkinsci.plugins.workflow.cps.CpsBodySubContext.onSuccess(CpsBodySubContext.java:68) at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:119) at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:366) at com.cloudbees.groovy.cps.impl.ValueBoundContinuation.receive(ValueBoundContinuation.java:21) at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:73) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:166) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at jav
[JIRA] (JENKINS-50199) Failed pipeline jobs stuck running after incorrect resume
Title: Message Title Mike Kozell reopened an issue Jenkins / JENKINS-50199 Failed pipeline jobs stuck running after incorrect resume Change By: Mike Kozell Resolution: Done Status: Closed Reopened Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-50199) Failed pipeline jobs stuck running after incorrect resume
Title: Message Title Mike Kozell updated an issue Jenkins / JENKINS-50199 Failed pipeline jobs stuck running after incorrect resume Change By: Mike Kozell Attachment: flownode-84.xml Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-50199) Failed pipeline jobs stuck running after incorrect resume
Title: Message Title Mike Kozell commented on JENKINS-50199 Re: Failed pipeline jobs stuck running after incorrect resume Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-50199) Failed pipeline jobs stuck running after incorrect resume
Title: Message Title Mike Kozell updated an issue Jenkins / JENKINS-50199 Failed pipeline jobs stuck running after incorrect resume Change By: Mike Kozell Comment: Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-50199) Failed pipeline jobs stuck running after incorrect resume
Title: Message Title Mike Kozell commented on JENKINS-50199 Re: Failed pipeline jobs stuck running after incorrect resume I have a dedicated test Jenkins master I can use for testing. I will just need to know where I can get the build and how to enable it. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-50223) Pipeline Jenkins master ran out of Java Metaspace
Title: Message Title
Mike Kozell commented on JENKINS-50223
Re: Pipeline Jenkins master ran out of Java Metaspace
We are able to reproduce the memory leak with two examples that is not specific to our code. In both examples and our potential workaround, the Jenkins jobs were all successful and the output in the console log was the same. The classes remained in the GC.class_stats query when using the examples and didn't when using the work around. Abstract Animal class Concrete Lion class extends Animal Example 1 We create a custom pipeline step and pass in an instantiated extended abstract object. The issue occurs when we use "runIdentify(AbstractObject)"
runZoo.groovy
def call() {
node {
def myLion = new Lion()
myLion.noise = "Roar"
myLion.sound(steps,myLion.noise)
runIdentify(myLion)
}
runIdentify.groovy
def call(Animal animal) {
if (animal.noise.toLowerCase().contains('roar')) {
println('This must be a Lion')
} else {
println('NOT a Lion')
}
}
Example 2 We instantiate an extended abstract object and pass it into a constructor object. The issue occurs when use "new Identity(AbstractObject)" to create the new object.
runZoo.groovy
def call() {
node {
def myLion = new Lion()
myLion.noise = "Roar"
myLion.sound(steps,myLion.noise)
new Identity(steps,myLion)
}
}
Identity.groovy
class Identity implements Serializable {
Identity(DSL steps, Animal animal) {
if (animal.noise.toLowerCase().contains('roar')) {
steps.echo('This must be a Lion')
} else {
steps.echo('NOT a Lion')
}
}
}
Work Around We were able to work around the issue by adding a new method to the abstract or concrete class.
runZoo.groovy
def call() {
node {
def myLion = new Lion()
myLion.noise = "Roar"
myLion.sound(steps,myLion.noise)
myLion.identifyAnimal(steps)
}
}
def ident
[JIRA] (JENKINS-50223) Pipeline Jenkins master ran out of Java Metaspace
Title: Message Title Mike Kozell commented on JENKINS-50223 Re: Pipeline Jenkins master ran out of Java Metaspace I switched a test Jenkins master back to MAX_SURVIVABILITY and still see the memory leak when failed builds occur. I'm going to test if the issue occurs on successful builds. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-50223) Pipeline Jenkins master ran out of Java Metaspace
Title: Message Title Mike Kozell commented on JENKINS-50223 Re: Pipeline Jenkins master ran out of Java Metaspace Yes these are the same masters. I will try to compare the memory usage from successful builds and build failures. Then I will try changing the durability level. We had another master run out of Metaspace today. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-50199) Failed pipeline jobs stuck running after incorrect resume
Title: Message Title Mike Kozell commented on JENKINS-50199 Re: Failed pipeline jobs stuck running after incorrect resume 1. I didn't see any IOExceptions in the jenkins.log file. There is one is the workflow-fallback/flowNodeStore.xml file which is included in this ticket. I did get an IOException after I aborted the build with the work around. java.io.IOException: Aborting build at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83) at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247) at Script1.run(Script1.groovy:1) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:585) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:623) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:594) at hudson.util.RemotingDiagnostics$Script.call(RemotingDiagnostics.java:142) at hudson.util.RemotingDiagnostics$Script.call(RemotingDiagnostics.java:114) at hudson.remoting.LocalChannel.call(LocalChannel.java:45) at hudson.util.RemotingDiagnostics.executeGroovy(RemotingDiagnostics.java:111) at jenkins.model.Jenkins._doScript(Jenkins.java:4360) at jenkins.model.Jenkins.doScript(Jenkins.java:4331) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletF
[JIRA] (JENKINS-50199) Failed pipeline jobs stuck running after incorrect resume
Title: Message Title Mike Kozell updated an issue Jenkins / JENKINS-50199 Failed pipeline jobs stuck running after incorrect resume Change By: Mike Kozell Attachment: workflow-fallback-flowNodeStore.xml Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-50199) Failed pipeline jobs stuck running after incorrect resume
Title: Message Title Mike Kozell updated an issue Jenkins / JENKINS-50199 Failed pipeline jobs stuck running after incorrect resume Change By: Mike Kozell Attachment: flowNodeStore.xml Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-50199) Failed pipeline jobs stuck running after incorrect resume
Title: Message Title Mike Kozell updated an issue Jenkins / JENKINS-50199 Failed pipeline jobs stuck running after incorrect resume Change By: Mike Kozell Attachment: build.xml Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-50199) Failed pipeline jobs stuck running after incorrect resume
Title: Message Title Mike Kozell updated an issue Jenkins / JENKINS-50199 Failed pipeline jobs stuck running after incorrect resume Change By: Mike Kozell Attachment: build.log Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-50223) Pipeline Jenkins master ran out of Java Metaspace
Title: Message Title Mike Kozell created an issue Jenkins / JENKINS-50223 Pipeline Jenkins master ran out of Java Metaspace Issue Type: Bug Assignee: Unassigned Components: pipeline Created: 2018-03-16 18:03 Priority: Major Reporter: Mike Kozell Setup Jenkins v2.89.3 / v2.89.4 LTS Pipeline API Plugin: 2.26 Pipeline Nodes and Processes Plugin: 2.19 Durable Task Plugin: 1.18 Pipeline Job Plugin: 2.17 Pipeline Shared Groovy Libraries Plugin: 2.9 Pipeline Supporting APIs Plugin: 2.18 Pipeline Groovy Plugin: 2.45 Script Security Plugin: 1.41 Pipeline Default Speed/Durability Level: Performance-Optimized "Do not allow the pipeline to resume if the master restarts": Enabled on all jobs Problem Shortly after upgrading from Jenkins 2.60.3 to Jenkins 2.89.3 LTS, we experienced a java.lang.OutOfMemoryError: Metaspace error. I'm unsure if this a bug or if increased Java Metaspace is to be expected in the newer version of Jenkins and pipeline plugins. Details Jenkins v2.60.3 In Jenkins 2.60.3 we used the memory parameters below and the Java nonheap space (Metaspace) stayed between 350mb and 550mb. We ran for months without restarting Jenkins. -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -XX:+UseStringDeduplication -XX:MaxMetaspaceSize=1g -XX:MetaspaceSize=256M -Xms4g -Xmx8g -Dgroovy.use.classvalue=true Jenkins v2.89.3 After upgrading to Jenkins 2.89.3 and upgrading all of the plugins, we used the Java memory settings below. We removed the "groovy.use.classvalue=true" parameter based on the recommendation here: https://support.cloudbees.com/hc/en-us/articles/115001505172-Jenkins-hangs-due-to-Groovy-ClassInfo-deadlock. We saw the Java nonheap memory fluctuate between 250mb and 750mb slowing trending upwards. After a few weeks, nonheap memory was at 1GB and Jenkins no longer responded to requests. The log file was full of java.lang.OutOfMemoryError: Metaspace errors. -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -XX:+UseStringDedu
[JIRA] (JENKINS-50199) Failed pipeline jobs stuck running after incorrect resume
Title: Message Title
Mike Kozell updated an issue
Jenkins / JENKINS-50199
Failed pipeline jobs stuck running after incorrect resume
Change By:
Mike Kozell
h2. SetupJenkins v2.89.4 LTS Pipeline API Plugin: 2.26 Pipeline Nodes and Processes Plugin: 2.19 Durable Task Plugin: 1.18 Pipeline Job Plugin: 2.17 Pipeline Shared Groovy Libraries Plugin: 2.9 Pipeline Supporting APIs Plugin: 2.18 Pipeline Groovy Plugin: 2.45 Script Security Plugin: 1.41Pipeline Default Speed/Durability Level: Performance-Optimized "Do not allow the pipeline to resume if the master restarts": Enabled on all jobsh2. ProblemI logged into a Jenkins master and saw no builds running but there was a queue of about 10 jobs. When mousing over the queued jobs, I saw "pending - Already running 2 builds across all nodes". This is strange because no jobs were showing as running and no Jenkins agents or executors were showing any running builds.I then ran "http://xx.xxx.xxx.xxx:8080/computer/api/xml?tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds" which did show 5 builds were running. I checked these builds and they were red (failure) and were not running.h2. ResearchI checked the console log of a build that showed as running but isn't and saw the line below near the top of the log._*Resume disabled by user, switching to high-performance, low-durability mode.*_At the end of the of the log I saw the following:*_Finished: FAILURE_* *_Resuming build at Tue Mar 13 23:04:52 UTC 2018 after Jenkins restart_*h2. Why Resume Build?The build failed on *Mar 12, 2018 6:02:37 PM*. +Why did the build try to resume almost a day later+? The job and system durability are configured to not resume builds. Below are some details taken from the API for the build._class "hudson.model.OneOffExecutor" id "41" keepLog false number 41 queueId 7178 result "FAILURE" timestamp 1520877757466I checked the Java process on the server and it was last restarted on *March 02 2018*.+What triggered the "Jenkins restart" identified on Mar 13 23:04:52 UTC 2018 since the Java process was not restarted?++Why does this get the build stuck in a "running" state when it's not running?+h2. ScopeThis issue can be seen across many of our Jenkins masters. In each case we see "Resuming build at x after Jenkins restart" occur a few days after the build failure or abort even though Java was not restarted. This issue didn't occur on Jenkins 2.60.3 running the older (pre-durability configurable) Pipeline plugins.h2. LogsI checked the jenkins.log file and saw the following when the build was attempting to be resumed. {code:java}Mar 13, 2018 9:29:56 PM org.jenkinsci.plugins.workflow.cps.CpsFlowExecution onLoad WARNING: Pipeline state not properly persisted, cannot resume job/JENKINS-JOB-NAME1/42/ Mar 13, 2018 9:29:56 PM org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem WARNING: Unexpected exception in CPS VM thread: CpsFlowExecution[OwnerJENKINS-JOB-NAME1/42:JENKINS-JOB-NAME1 #42] java.lang.NullPointe
[JIRA] (JENKINS-50199) Failed pipeline jobs stuck running after incorrect resume
Title: Message Title
Mike Kozell updated an issue
Jenkins / JENKINS-50199
Failed pipeline jobs stuck running after incorrect resume
Change By:
Mike Kozell
h2. SetupJenkins v2.89.4 LTS Pipeline API Plugin: 2.26 Pipeline Nodes and Processes Plugin: 2.19 Durable Task Plugin: 1.18 Pipeline Job Plugin: 2.17 Pipeline Shared Groovy Libraries Plugin: 2.9 Pipeline Supporting APIs Plugin: 2.18 Pipeline Groovy Plugin: 2.45 Script Security Plugin: 1.41Pipeline Default Speed/Durability Level: Performance-Optimized "Do not allow the pipeline to resume if the master restarts": Enabled on all jobsh2. ProblemI logged into a Jenkins master and saw no builds running but there was a queue of about 10 jobs. When mousing over the queued jobs, I saw "pending - Already running 2 builds across all nodes". This is strange because no jobs were showing as running and no Jenkins agents or executors were showing any running builds.I then ran "http://xx.xxx.xxx.xxx:8080/computer/api/xml?tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds" which did show 5 builds were running. I checked these builds and they were red (failure) and were not running.h2. ResearchI checked the console log of a build that showed as running but isn't and saw the line below near the top of the log._*Resume disabled by user, switching to high-performance, low-durability mode.*_At the end of the of the log I saw the following:*_Finished: FAILURE_* *_Resuming build at Tue Mar 13 23:04:52 UTC 2018 after Jenkins restart_*h2. Why Resume Build?The build failed on *Mar 12, 2018 6:02:37 PM*. +Why did the build try to resume almost a day later+? The job and system durability are configured to not resume builds. Below are some details taken from the API for the build._class "hudson.model.OneOffExecutor" id "41" keepLog false number 41 queueId 7178 result "FAILURE" timestamp 1520877757466I checked the Java process on the server and it was last restarted on *March 02 2018*.+What triggered the "Jenkins restart" identified on Mar 13 23:04:52 UTC 2018 since the Java process was not restarted?++Why does this get the build stuck in a "running" state when it's not running?+h2. ScopeThis issue can be seen across many of our Jenkins masters. In each case we see "Resuming build at x after Jenkins restart" occur a few days after the build failure or abort even though Java was not restarted. This issue didn't occur on Jenkins 2.60.3 running the older (pre-durability configurable) Pipeline plugins.h2. LogsI checked the jenkins.log file and saw the following when the build was attempting to be resumed. {code:java}Mar 13, 2018 9:29:56 PM org.jenkinsci.plugins.workflow.cps.CpsFlowExecution onLoad WARNING: Pipeline state not properly persisted, cannot resume job/JENKINS-JOB-NAME1/42/ Mar 13, 2018 9:29:56 PM org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem WARNING: Unexpected exception in CPS VM thread: CpsFlowExecution[OwnerJENKINS-JOB-NAME1/42:JENKINS-JOB-NAME1 #42] java.lang.NullPoin
[JIRA] (JENKINS-50199) Failed pipeline jobs stuck running after incorrect resume
Title: Message Title
Mike Kozell updated an issue
Jenkins / JENKINS-50199
Failed pipeline jobs stuck running after incorrect resume
Change By:
Mike Kozell
h2. SetupJenkins v2.89.4 LTS Pipeline API Plugin: 2.26 Pipeline Nodes and Processes Plugin: 2.19 Durable Task Plugin: 1.18 Pipeline Job Plugin: 2.17 Pipeline Shared Groovy Libraries Plugin: 2.9 Pipeline Supporting APIs Plugin: 2.18 Script Security Plugin: 1.41Pipeline Default Speed/Durability Level: Performance-Optimized "Do not allow the pipeline to resume if the master restarts": Enabled on all jobsh2. ProblemI logged into a Jenkins master and saw no builds running but there was a queue of about 10 jobs. When mousing over the queued jobs, I saw "pending - Already running 2 builds across all nodes". This is strange because no jobs were showing as running and no Jenkins agents or executors were showing any running builds.I then ran "http://xx.xxx.xxx.xxx:8080/computer/api/xml?tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds" which did show 5 builds were running. I checked these builds and they were red (failure) and were not running.h2. ResearchI checked the console log of a build that showed as running but isn't and saw the line below near the top of the log._*Resume disabled by user, switching to high-performance, low-durability mode.*_At the end of the of the log I saw the following:*_Finished: FAILURE_* *_Resuming build at Tue Mar 13 23:04:52 UTC 2018 after Jenkins restart_*h2. Why Resume Build?The build failed on *Mar 12, 2018 6:02:37 PM*. +Why did the build try to resume almost a day later+? The job and system durability are configured to not resume builds. Below are some details taken from the API for the build._class "hudson.model.OneOffExecutor" id "41" keepLog false number 41 queueId 7178 result "FAILURE" timestamp 1520877757466I checked the Java process on the server and it was last restarted on *March 02 2018*.+What triggered the "Jenkins restart" identified on Mar 13 23:04:52 UTC 2018 since the Java process was not restarted?++Why does this get the build stuck in a "running" state when it's not running?+h2. ScopeThis issue can be seen across many of our Jenkins masters. In each case we see "Resuming build at x after Jenkins restart" occur a few days after the build failure or abort even though Java was not restarted. This issue didn't occur on Jenkins 2.60.3 running the older (pre-durability configurable) Pipeline plugins.h2. LogsI checked the jenkins.log file and saw the following when the build was attempting to be resumed. {code:java}Mar 13, 2018 9:29:56 PM org.jenkinsci.plugins.workflow.cps.CpsFlowExecution onLoad WARNING: Pipeline state not properly persisted, cannot resume job/JENKINS-JOB-NAME1/42/ Mar 13, 2018 9:29:56 PM org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem WARNING: Unexpected exception in CPS VM thread: CpsFlowExecution[OwnerJENKINS-JOB-NAME1/42:JENKINS-JOB-NAME1 #42] java.lang.NullPointerException at org.jenkinsci
[JIRA] (JENKINS-50199) Failed pipeline jobs stuck running after incorrect resume
Title: Message Title
Mike Kozell updated an issue
Jenkins / JENKINS-50199
Failed pipeline jobs stuck running after incorrect resume
Change By:
Mike Kozell
h2. SetupJenkins v2.89.4 LTS Pipeline API Plugin: 2.26 Pipeline Nodes and Processes Plugin: 2.19 Durable Task Plugin: 1.18 Pipeline Job Plugin: 2.17 Pipeline Shared Groovy Libraries Plugin: 2.9 Pipeline Supporting APIs Plugin: 2.18 Script Security Plugin: 1.41Pipeline Default Speed/Durability Level: Performance-Optimized "Do not allow the pipeline to resume if the master restarts": Enabled on all jobsh2. ProblemI logged into a Jenkins master and saw no builds running but there was a queue of about 10 jobs. When mousing over the queued jobs, I saw "pending - Already running 2 builds across all nodes". This is strange because no jobs were showing as running and no Jenkins agents or executors were showing any running builds.I then ran "http://xx.xxx.xxx.xxx:8080/computer/api/xml?tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds" which did show 5 builds were running. I checked these builds and they were red (failure) and were not running.h2. ResearchI checked the console log of a build that showed as running but isn't and saw the line below near the top of the log._*Resume disabled by user, switching to high-performance, low-durability mode.*_At the end of the of the log I saw the following:*_Finished: FAILURE_* *_Resuming build at Tue Mar 13 23:04:52 UTC 2018 after Jenkins restart_*h2. Why Resume Build?The build failed on *Mar 12, 2018 6:02:37 PM*. +Why did the build try to resume almost a day later+? The job and system durability are configured to not resume builds. Below are some details taken from the API for the build._class "hudson.model.OneOffExecutor" id "41" keepLog false number 41 queueId 7178 result "FAILURE" timestamp 1520877757466I checked the Java process on the server and it was last restarted on *March 02 2018*.+What triggered the "Jenkins restart" identified on Mar 13 23:04:52 UTC 2018 since the Java process was not restarted?++Why does this get the build stuck in a "running" state when it's not running?+h2. ScopeThis issue can be seen across many of our Jenkins masters. In each case we see "Resuming build at x after Jenkins restart" occur a few days after the build failure or abort even though Java was not restarted. This issue didn't occur on Jenkins 2.60.3 running the older (pre-durability configurable) Pipeline plugins.h2. LogsI checked the jenkins.log file and saw the following when the build was attempting to be resumed. {code:java}Mar 13, 2018 9:29:56 PM org.jenkinsci.plugins.workflow.cps.CpsFlowExecution onLoad WARNING: Pipeline state not properly persisted, cannot resume job/JENKINS-JOB-NAME1/42/ Mar 13, 2018 9:29:56 PM org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem WARNING: Unexpected exception in CPS VM thread: CpsFlowExecution[OwnerJENKINS-JOB-NAME1/42:JENKINS-JOB-NAME1 #42] java.lang.NullPointerException at org.jenkinsci.p
[JIRA] (JENKINS-50199) Failed pipeline jobs stuck running after incorrect resume
Title: Message Title Mike Kozell created an issue Jenkins / JENKINS-50199 Failed pipeline jobs stuck running after incorrect resume Issue Type: Bug Assignee: Unassigned Components: pipeline, workflow-api-plugin, workflow-cps-plugin Created: 2018-03-15 17:18 Priority: Critical Reporter: Mike Kozell Setup Jenkins v2.89.4 LTS Pipeline API Plugin: 2.26 Pipeline Nodes and Processes Plugin: 2.19 Durable Task Plugin: 1.18 Pipeline Job Plugin: 2.17 Pipeline Shared Groovy Libraries Plugin: 2.9 Pipeline Supporting APIs Plugin: 2.18 Script Security Plugin: 1.41 Pipeline Default Speed/Durability Level: Performance-Optimized "Do not allow the pipeline to resume if the master restarts": Enabled on all jobs Problem I logged into a Jenkins master and saw no builds running but there was a queue of about 10 jobs. When mousing over the queued jobs, I saw "pending - Already running 2 builds across all nodes". This is strange because no jobs were showing as running and no Jenkins agents or executors were showing any running builds. I then ran "http://xx.xxx.xxx.xxx:8080/computer/api/xml?tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds" which did show 5 builds were running. I checked these builds and they were red (failure) and were not running. Research I checked the console log of a build that showed as running but isn't and saw the line below near the top of the log. Resume disabled by user, switching to high-performance, low-durability mode. At the end of the of the log I saw the following: Finished: FAILURE Resuming build at Tue Mar 13 23:04:52 UTC 2018 after Jenkins restart Why Resume Build? The build failed on Mar 12, 2018 6:02:37 PM. Why did the build try to resume almost a day later? The job and system durability are configured to not resume builds. Below are some details taken from the API for the build. _class "hudson.model.OneOffExecutor" id "41" keepLog false number 41 queueId 7178 result "FAILURE" timestamp 1520
[JIRA] (JENKINS-33761) Ability to disable Pipeline durability and "resume" build.
Title: Message Title
Mike Kozell edited a comment on JENKINS-33761
Re: Ability to disable Pipeline durability and "resume" build.
[~svanoort]After upgrading Jenkins with the following, I was not able to reproduce the issue after a build timeout, cancelling a build, and restarting Jenkins in the middle of a build.Jenkins 2.89.4Pipeline 2.5Pipeline API 2.26Pipeline Nodes and Processes 2.19Pipeline Step API 2.14Scripts Security 1.41durabilityHint=PERFORMANCE_OPTIMIZEDorg.jenkinsci.plugins.workflow.job.properties.DisableResumeJobPropertyGroovy Sandbox = disabledJava = 1.8.0_162Although my jobs correctly didn't resume after Jenkins restart, I did see the message below in the build logs. {code:java} Resuming build at Sat Feb 24 06:38:10 UTC 2018 after Jenkins restart[Pipeline] End of Pipelinejava.io.IOException: Cannot resume build -- – was not cleanly saved when Jenkins shut down. {code}
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-33761) Ability to disable Pipeline durability and "resume" build.
Title: Message Title Mike Kozell commented on JENKINS-33761 Re: Ability to disable Pipeline durability and "resume" build. Sam Van Oort After upgrading Jenkins with the following, I was not able to reproduce the issue after a build timeout, cancelling a build, and restarting Jenkins in the middle of a build. Jenkins 2.89.4 Pipeline 2.5 Pipeline API 2.26 Pipeline Nodes and Processes 2.19 Pipeline Step API 2.14 Scripts Security 1.41 durabilityHint=PERFORMANCE_OPTIMIZED org.jenkinsci.plugins.workflow.job.properties.DisableResumeJobProperty Groovy Sandbox = disabled Java = 1.8.0_162 Although my jobs correctly didn't resume after Jenkins restart, I did see the message below in the build logs. Resuming build at Sat Feb 24 06:38:10 UTC 2018 after Jenkins restart [Pipeline] End of Pipeline java.io.IOException: Cannot resume build – was not cleanly saved when Jenkins shut down. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-47868) Pipeline durability hang when slave node disconnected
Title: Message Title Mike Kozell commented on JENKINS-47868 Re: Pipeline durability hang when slave node disconnected This issue still occurs on: Jenkins 2.89.4 Pipeline 2.5 Pipeline API 2.26 Pipeline Nodes and Processes 2.19 Pipeline Step API 2.14 Scripts Security 1.41 durabilityHint=PERFORMANCE_OPTIMIZED org.jenkinsci.plugins.workflow.job.properties.DisableResumeJobProperty Groovy Sandbox = disabled Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-47724) Pipeline with parallel jobs performance issue with EC2 slaves
Title: Message Title Mike Kozell commented on JENKINS-47724 Re: Pipeline with parallel jobs performance issue with EC2 slaves Sam Van Oort, I apologize for taking so long to get back to you. I upgraded Jenkins and all of the plugins and retested the items in this ticket. Jenkins 2.89.4 Pipeline 2.5 Pipeline API 2.26 Pipeline Nodes and Processes 2.19 Pipeline Step API 2.14 Scripts Security 1.41 durabilityHint=PERFORMANCE_OPTIMIZED org.jenkinsci.plugins.workflow.job.properties.DisableResumeJobProperty Groovy Sandbox = disabled Java = 1.8.0_162 3. Parallel pipeline job appears to pause when starting agents This issue still occurs. I started 100 agents and used the sample pipeline code in the description of this ticket (except I used 100 branches instead of 150) and the build took 14 minutes to complete. This is consistent with what I saw last year. 2. The first parallel pipeline job on an agent is slower This also still occurs on the newest version of the plugins. As I stated above it took 14 minutes for the pipeline job to complete after starting 100 Jenkins agents. I then ran the exact same build immediately afterwards and it took 26 seconds. I'm unsure why there is a 14 minute pause on the first build ran on our Jenkins agents after they are started. This doesn't occur with freestyle and the sample code just does "sh date +%c". 4. High CPU on Jenkins Masters This issue appears to have been resolved. In previous tests, our Jenkins masters reached a load average of 10 when they were busy. After performing the upgrade and setting durability to performance optimized, I see the load average hover around 1. On a different Jenkins installation that uses a lot of master executors but no agents, we also saw the load average hit 10 on Jenkins 2.60.3. That installation was recently upgraded and the load average there also remains under 1 most of the time. Java memory usage may have also decreased. Our build durations remain about the same between Jenkins versions with 2.89.4 maybe being slightly faster. With the pause that occurs on the first build on the agents, throwing more agents at our builds doesn't yield faster build times. Please let me know if you have any suggestions. Add Comment
