[JIRA] [core] (JENKINS-14332) Repeated channel/timeout errors from Jenkins slave

Wed, 05 Nov 2014 02:07:14 -0800

I have the same kind of issue.

Working combination:
Jenkins version: 1.588
Remoting version: 2.47
Master JVM: 1.7.0_65
Slave JVM: 1.6.0_33/1.7.0_65)
Linux kernel master: 3.13.0-37-generic
Linux kernel slave: 3.2.0-69-virtual

Not Working combination:
Jenkins version: 1.588
Remoting version: 2.47
Master JVM: 1.7.0_65
Slave JVM: 1.6.0_33/1.7.0_65(Tested both)
Linux kernel master: 3.13.0-37-generic
Linux kernel slave: 3.13.0-35-generic/3.13.0-39-generic(Tested both)

The slave hangs on the POM parsing

stack trace slave.jar:
Full thread dump OpenJDK 64-Bit Server VM (24.65-b04 mixed mode):

"Attach Listener" daemon prio=10 tid=0x00007f7eb0001800 nid=0x645 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Stream reader: maven process at Socket[addr=/127.0.0.1,port=46056,localport=55623]" prio=10 tid=0x00007f7ec004f800 nid=0x611 runnable [0x00007f7ea4825000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at hudson.util.StreamCopyThread.run(StreamCopyThread.java:60)

"java -XX:MaxPermSize=1024m -cp /tmp/maven3-agent.jar:/usr/share/maven/boot/plexus-classworlds-2.x.jar org.jvnet.hudson.maven3.agent.Maven3Main /usr/share/maven /tmp/slave.jar /tmp/maven3-interceptor.jar /tmp/maven3-interceptor-commons.jar 55623: stdout copier" prio=10 tid=0x00007f7ec0046800 nid=0x607 runnable [0x00007f7ea4623000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:272)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)

  • locked <0x00000000bbbc43f0> (a java.lang.UNIXProcess$ProcessPipeInputStream)
    at java.io.FilterInputStream.read(FilterInputStream.java:107)
    at hudson.util.StreamCopyThread.run(StreamCopyThread.java:60)

"Proc.executor 1" daemon prio=10 tid=0x00007f7eb81e6000 nid=0x570 waiting on condition [0x00007f7ea4724000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x00000000cec5cef0> (a java.util.concurrent.SynchronousQueue$TransferStack)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

"process reaper" daemon prio=10 tid=0x00007f7eb81ec800 nid=0x56c runnable [0x00007f7ea485e000]
java.lang.Thread.State: RUNNABLE
at java.lang.UNIXProcess.waitForProcessExit(Native Method)
at java.lang.UNIXProcess.access$500(UNIXProcess.java:54)
at java.lang.UNIXProcess$4.run(UNIXProcess.java:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-6" prio=10 tid=0x00007f7eac011800 nid=0x557 waiting on condition [0x00007f7ea5cba000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x00000000ce80c268> (a java.util.concurrent.SynchronousQueue$TransferStack)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-4" prio=10 tid=0x00007f7eac00e800 nid=0x555 waiting on condition [0x00007f7ea5ebc000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x00000000ce80c268> (a java.util.concurrent.SynchronousQueue$TransferStack)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-3" prio=10 tid=0x00007f7eac00d000 nid=0x554 waiting on condition [0x00007f7ec419b000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x00000000ce80c268> (a java.util.concurrent.SynchronousQueue$TransferStack)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

"Ping thread for channel hudson.remoting.Channel@b920c51:channel" daemon prio=10 tid=0x00007f7eb42cd800 nid=0x551 waiting on condition [0x00007f7ec5685000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at hudson.remoting.PingThread.run(PingThread.java:91)

"Ping thread for channel hudson.remoting.Channel@b920c51:channel" daemon prio=10 tid=0x00007f7ecc1a7800 nid=0x54f waiting on condition [0x00007f7ec6eaf000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at hudson.remoting.PingThread.run(PingThread.java:91)

"Channel reader thread: channel" prio=10 tid=0x00007f7ecc19c800 nid=0x54e runnable [0x00007f7ec6fb0000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:272)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)

  • locked <0x00000000ce80e768> (a java.io.BufferedInputStream)
    at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
    at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
    at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
    at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

"Service Thread" daemon prio=10 tid=0x00007f7ecc0cc800 nid=0x54c runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x00007f7ecc0ca000 nid=0x54b waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x00007f7ecc0c8000 nid=0x54a waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00007f7ecc0c5800 nid=0x549 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00007f7ecc099000 nid=0x548 in Object.wait() [0x00007f7ec773f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

  • waiting on <0x00000000ce80ea08> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
  • locked <0x00000000ce80ea08> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" daemon prio=10 tid=0x00007f7ecc097000 nid=0x547 in Object.wait() [0x00007f7ec7840000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

  • waiting on <0x00000000ce80eab0> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:503)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
  • locked <0x00000000ce80eab0> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x00007f7ecc00a000 nid=0x545 in Object.wait() [0x00007f7ed24b4000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

  • waiting on <0x00000000ce80c310> (a hudson.remoting.Channel)
    at hudson.remoting.Channel.join(Channel.java:902)
  • locked <0x00000000ce80c310> (a hudson.remoting.Channel)
    at hudson.remoting.Launcher.main(Launcher.java:527)
    at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:459)
    at hudson.remoting.Launcher.run(Launcher.java:239)
    at hudson.remoting.Launcher.main(Launcher.java:192)

"VM Thread" prio=10 tid=0x00007f7ecc092800 nid=0x546 runnable

"VM Periodic Task Thread" prio=10 tid=0x00007f7ecc0d7800 nid=0x54d waiting on condition

JNI global references: 221

Stack trace maven.jar:

Full thread dump OpenJDK 64-Bit Server VM (24.65-b04 mixed mode):

"Attach Listener" daemon prio=10 tid=0x00007f6308046000 nid=0x650 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"pool-1-thread-1 for channel / waiting for hudson.remoting.Channel@2855e552:channel" prio=10 tid=0x00007f6304002000 nid=0x614 in Object.wait() [0x00007f62ff89a000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

  • waiting on <0x0000000085272700> (a hudson.remoting.UserRequest)
    at hudson.remoting.Request.call(Request.java:146)
  • locked <0x0000000085272700> (a hudson.remoting.UserRequest)
    at hudson.remoting.Channel.call(Channel.java:751)
    at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:173)
    at com.sun.proxy.$Proxy5.setExecutedMojos(Unknown Source)
    at hudson.maven.MavenBuildProxy$Filter.setExecutedMojos(MavenBuildProxy.java:267)
    at hudson.maven.Maven3Builder$MavenExecutionListener.recordMojoEnded(Maven3Builder.java:630)
    at hudson.maven.Maven3Builder$MavenExecutionListener.mojoSucceeded(Maven3Builder.java:615)
    at org.apache.maven.lifecycle.internal.DefaultExecutionEventCatapult.fire(DefaultExecutionEventCatapult.java:87)
    at org.apache.maven.lifecycle.internal.DefaultExecutionEventCatapult.fire(DefaultExecutionEventCatapult.java:42)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:228)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
    at org.jvnet.hudson.maven3.launcher.Maven3Launcher.main(Maven3Launcher.java:117)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
    at org.jvnet.hudson.maven3.agent.Maven3Main.launch(Maven3Main.java:178)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at hudson.maven.Maven3Builder.call(Maven3Builder.java:136)
    at hudson.maven.Maven3Builder.call(Maven3Builder.java:71)
    at hudson.remoting.UserRequest.perform(UserRequest.java:121)
    at hudson.remoting.UserRequest.perform(UserRequest.java:49)
    at hudson.remoting.Request$2.run(Request.java:324)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

"Channel reader thread: channel" prio=10 tid=0x00007f63201ad000 nid=0x613 runnable [0x00007f62ff99d000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)

  • locked <0x0000000098dc0720> (a java.io.BufferedInputStream)
    at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
    at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
    at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
    at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

"Service Thread" daemon prio=10 tid=0x00007f63200a5800 nid=0x60f runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x00007f63200a3000 nid=0x60e waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x00007f63200a1000 nid=0x60d waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00007f632009e800 nid=0x60c runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00007f6320072000 nid=0x60b in Object.wait() [0x00007f6324886000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

  • waiting on <0x0000000098dc09c0> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
  • locked <0x0000000098dc09c0> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" daemon prio=10 tid=0x00007f6320070000 nid=0x60a in Object.wait() [0x00007f6324987000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

  • waiting on <0x0000000098dc0a68> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:503)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
  • locked <0x0000000098dc0a68> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x00007f632000a000 nid=0x608 in Object.wait() [0x00007f632895c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

  • waiting on <0x0000000098dbe408> (a hudson.remoting.Channel)
    at hudson.remoting.Channel.join(Channel.java:902)
  • locked <0x0000000098dbe408> (a hudson.remoting.Channel)
    at hudson.remoting.Launcher.main(Launcher.java:527)
    at hudson.remoting.Launcher.main(Launcher.java:501)
    at hudson.remoting.Launcher.main(Launcher.java:493)
    at hudson.remoting.Launcher.main(Launcher.java:489)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.jvnet.hudson.maven3.agent.Maven3Main.main(Maven3Main.java:136)
    at org.jvnet.hudson.maven3.agent.Maven3Main.main(Maven3Main.java:64)

"VM Thread" prio=10 tid=0x00007f632006b800 nid=0x609 runnable

"VM Periodic Task Thread" prio=10 tid=0x00007f63200b1000 nid=0x610 waiting on condition

JNI global references: 180
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to