|
The problem started occurring after upgrading from maven plugin 2.1.
The node becomes entirely unresponsive and any other builds or scm pollings also freeze on that node. The jenkins node need to be manually reconnected to clear the fault.
If a git SCM poll was active on the node then the jenkins master must be restarted for the polling status page to clear.
The problem does not seem to be intermittent and is easy to reproduce for us.
Attaching jstacks and a mocked multimodule job that replicates the problem.
Using SSH slaves plugin 1.6 and Jenkins LTS 1.554.1.
Java 7u51 x86-64.
Running on the master node without ssh slave plugin does not cause a freeze.
Running without the -T flag does not cause a freeze.
Running with maven-plugin 2.1 does not cause a freeze.
Have not tested with maven plugin version 2.2.
Tested with maven 3.1.1. Maven 3.2.1 with -T flag does not seem supported by the plugin at all.
Goals: compile -T 4
MAVEN_OPTS=-Xms128M -Xmx1G -verbose:gc
Jenkins is started via ubuntu upstart with the "java -jar jenkins.war" method. It is however manually installed.
Jenkins JVM options: -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Xloggc:/tmp/jenkins-test/loggc.txt -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Xmx1024m -Xms1024m -XX:SoftRefLRUPolicyMSPerMB=2000 -XX:MaxPermSize=192m -server -Djava.awt.headless=true -Dhudson.master.headless=true -Dhudson.webstart.headless=true -Dorg.jenkinsci.plugins.gitclient.Git.useCLI=true -DJENKINS_HOME=/local/jenkins-test -jar /local/jenkins-test/jenkins.war --javaHome=/opt/local/dev_tools/java/x64/jdk1.7.0_51/jre --prefix=/jenkins --logfile=/local/jenkins-test/jenkins.log --httpListenAddress=127.0.0.1 --httpPort=18080 --httpsListenAddress=127.0.0.1 --httpsPort=-1 --ajp13ListenAddress=127.0.0.1 --ajp13Port=-1 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20
Node JVM options: -Xms8m -Xmx128m -Dorg.jenkinsci.plugins.gitclient.Git.useCLI=true -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Xloggc:/local/jenkins-test-agents/jenkinsadm/loggc.txt -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
No out of memory problem indications in -Xloggc or -verbose:gc.
I have tried with and without these, no effect:
Use private Maven repository: Local to workspace
Run headless
Disable triggering of downstream projects
Disable automatic site documentation artifact archiving
Disable automatic artifact archiving
The "Build modules in parallel" functionality is not a viable option for us, we require that the build be done within the same job.
Tried both with and without the "chrt -b 0" batch scheduler. Normally we start the nodes with the batch scheduler as a prefix.
Tried to run nodes both on RHEL 6.2 over lan and Ubuntu 14.04 on localhost. Both are multicore machines.
I have renamed some server and usernames for security reasons.
Hanging job name: ThomasHerrlin_ticket_3348
Hanging slave node: server030
Jenkins console output:
...
Executing Maven: -B -f /ssd/jenkins-test-agents/jenkinsadm/workspace/ThomasHerrlin_ticket_3348/pom.xml -T4 compile
[GC 33024K->4405K(125952K), 0.0085580 secs]
[GC 37429K->6340K(125952K), 0.0109360 secs]
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] my-app1
[INFO] my-app2
[INFO] my-app3
[INFO] my-app4
[INFO] buildall_test
[HUDSON] Collecting dependencies info
[HUDSON] Collecting dependencies info[INFO] Building with 4 threads
[HUDSON] Collecting dependencies info
[HUDSON] Collecting dependencies info
At this point it freezes.
|
