Hello Sam,
Looks like something gone wrong here (in the pipeline script), extracted
from the stacktrace in your post.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
at
com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
at WorkflowScript.run(WorkflowScript:343)
at ___cps.transform___(Native Method)
/Ram
On Friday, March 2, 2018 at 11:26:46 PM UTC+5:30, Sam K wrote:
>
> Hello
>
> I have about 20 nodes of various RHEL versions and one team's nodes
> always pose certain problems and cause their pipelines to fail. For a
> while, I was able to get them to pass by 'bringing nodes online on-demand'
> because it would refresh the connection. But lately that strategy has also
> failed. The slaves had a higher version of Java than the master and after
> updating the master the problem seemed to go away only to come back again
> like this. If I disconnect, offline the node and bring it back, the logs
> are clean. But once the pipeline starts running again, it fails. Any help
> would be much appreciated. Thanks.
>
>
> I'm running Jenkins master version: 2.91.
>
> JDK version on master RHEL-7
> [sup...@10.0.156.53 <javascript:> ~]$ java -version
> openjdk version "1.8.0_141"
> OpenJDK Runtime Environment (build 1.8.0_141-b16)
> OpenJDK 64-Bit Server VM (build 25.141-b16, mixed mode)
>
> Slave -
> RHEL 7.3
> openjdk version "1.8.0_131"
> OpenJDK Runtime Environment (build 1.8.0_131-b11)
> OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)
>
> Pipeline fails with such errors:
>
> [Pipeline] End of Pipelinejava.io.EOFException
> at
> java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
> at
> java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
> at
> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
> at
> hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
> at
> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
> at
> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
> Caused: java.io.IOException: Unexpected termination of the channel
> at
> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
> Caused: hudson.remoting.ChannelClosedException: Remote call on 10.0.135.15
> failed. The channel is closing down or has closed down
> at hudson.remoting.Channel.call(Channel.java:901)
> at hudson.FilePath.act(FilePath.java:986)
> at hudson.FilePath.act(FilePath.java:975)
> at hudson.FilePath.mkdirs(FilePath.java:1158)
> at
> org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.<init>(FileMonitoringTask.java:113)
> at
> org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java:198)
> at
> org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java:190)
> at
> org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:111)
> at
> org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:64)
> at
> org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:177)
> at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:224)
> at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:150)
> at
> org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
> at sun.reflect.GeneratedMethodAccessor5286.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
> at
> org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
> at
> org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
> at
> org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:133)
> at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:153)
> at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:157)
> at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127)
> at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127)
> at
> com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
> Caused: java.io.IOException: remote file operation failed:
> /mnt/grandprix/homes/axsbuild/workspace/Sensor/Sensor_4.x_Groovy at
> hudson.remoting.Channel@3735998a:10.0.135.15
> at hudson.FilePath.act(FilePath.java:993)
> at hudson.FilePath.act(FilePath.java:975)
> at hudson.FilePath.mkdirs(FilePath.java:1158)
> at
> org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.<init>(FileMonitoringTask.java:113)
> at
> org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java:198)
> at
> org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java:190)
> at
> org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:111)
> at
> org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:64)
> at
> org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:177)
> at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:224)
> at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:150)
> at
> org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
> at sun.reflect.GeneratedMethodAccessor5286.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
> at
> org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
> at
> org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
> at
> org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:133)
> at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:153)
> at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:157)
> at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127)
> at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:127)
> at
> com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
> at WorkflowScript.run(WorkflowScript:343)
> at ___cps.transform___(Native Method)
> at
> com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
> at
> com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
> at
> com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
> at sun.reflect.GeneratedMethodAccessor196.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
> at
> com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:60)
> at
> com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
> at
> com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
> at sun.reflect.GeneratedMethodAccessor196.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
> at
> com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:39)
> at
> com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
> at
> com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:28)
> at
> com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55)
> at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16)
> at com.cloudbees.groovy.cps.Next.step(Next.java:83)
> at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
> at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
> at
> org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
> at
> org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
> at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
> at
> org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
> at
> org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35)
> at
> org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32)
> at
> org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
> at
> org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32)
> at
> org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
> at
> org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:330)
> at
> org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82)
> at
> org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242)
> at
> org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230)
> at
> org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
> at
> jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Finished: FAILURE
>
>
> And the slave has logs like this:
>
> Log from the node 10.0.135.15:
>
> Mar 01, 2018 9:21:54 PM hudson.util.ProcessTree getKillers
>
> WARNING: Failed to obtain killers
>
> java.io.IOException: Remote call on channel failed
>
> at hudson.remoting.Channel.call(Channel.java:916)
>
> at hudson.util.ProcessTree.getKillers(ProcessTree.java:159)
>
> at
> hudson.util.ProcessTree$OSProcess.killByKiller(ProcessTree.java:218)
>
> at hudson.util.ProcessTree$UnixProcess.kill(ProcessTree.java:660)
>
> at
> hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:668)
>
> at
> hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:667)
>
> at
> hudson.util.ProcessTree$UnixProcess.killRecursively(ProcessTree.java:667)
>
> at hudson.util.ProcessTree$Unix.killAll(ProcessTree.java:589)
>
> at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1091)
>
> at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1082)
>
> at hudson.remoting.UserRequest.perform(UserRequest.java:207)
>
> at hudson.remoting.UserRequest.perform(UserRequest.java:53)
>
> at hudson.remoting.Request$2.run(Request.java:358)
>
> at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:748)
>
> Caused by: java.lang.NoClassDefFoundError:
> hudson/remoting/RemoteClassLoader$RemoteIClassLoader
>
> at
> hudson.remoting.RemoteClassLoader.export(RemoteClassLoader.java:744)
>
> at hudson.remoting.UserRequest.<init>(UserRequest.java:104)
>
> at hudson.remoting.Channel.call(Channel.java:906)
>
> ... 17 more
>
>
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/03261a3c-61fe-4243-a638-b33bb65799c7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
