Here is a small sample of the types of logs we’re seeing: cml@jenkins-master:~$ cat camel-20180215.lsof | grep camel | grep log | wc -l 2277
java 18713 jenkins 4931w REG 8,17 0 215356692 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-twitter-salesforce/builds/1328/log java 18713 jenkins 4932w REG 8,17 0 215356694 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-twitter-websocket/builds/1328/log java 18713 jenkins 4933w REG 8,17 0 215356696 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-twitter-websocket-blueprint/builds/1328/log java 18713 jenkins 4934w REG 8,17 0 215356698 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-validator-spring-boot/builds/1328/log java 18713 jenkins 4935w REG 8,17 0 215356701 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-widget-gadget-cdi/builds/1328/log java 18713 jenkins 4936w REG 8,17 0 215356706 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-widget-gadget-java/builds/1328/log java 18713 jenkins 4937w REG 8,17 0 215356709 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-widget-gadget-xml/builds/1328/log java 18713 jenkins 4938w REG 8,17 0 215356714 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-zipkin/builds/1328/log java 18713 jenkins 4939w REG 8,17 0 215356725 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-zipkin-client/builds/1328/log > On Feb 16, 2018, at 3:56 AM, Zoran Regvart <zo...@regvart.com> wrote: > > Hi Chris, > thank you for troubleshooting this, can you clarify one thing for me, > when you mention log files, are these the `.log` files generated > during the test phase of the build or the XML/TXT files with JUnit > reports? > > I would think that the job type being Maven and the automatic > gathering of JUnit test reports is the culprit but would like a > confirmation. > > If it is so, I think we one possible solution is to migrate to > Pipeline/Freestyle job type as, as far as I'm aware, there is no way > to prevent Maven job type from gathering JUnit reports. > > zoran > > On Fri, Feb 16, 2018 at 2:37 AM, Chris Lambertus <c...@apache.org> wrote: >> >> Hi Camel PMC, >> >> We have been having an ongoing problem with Jenkins for quite some time, >> where the CPU usage and IOPS skyrocket on the master. Each time this has >> happened, the jenkins build nodes lose all of their associated labels, and >> all new builds are unable to start. >> >> In the times I’ve been able to investigate this, there has in each case been >> several Camel builds running, and in each case, the builds are opening >> somewhere between 1500 and 2200 log files, which seems to be killing the >> jenkins master. For comparison, some very large build jobs for Hadoop only >> open ~15 or so log files. >> >> I have had to take the rather drastic step of disabling the Camel Jenkins >> jobs (many of which have been failing for awhile now) while we continue to >> investigate this issue. Before we re-enable the jobs, we’re going to have to >> figure out how to get your builds to open a sane number of log files — the >> current situation where the builds are creating thousands of log files is >> not sustainable, and we believe this may be one of the causative factors of >> the ongoing jenkins outages. While I cannot say with any certainty that this >> is what’s been killing the master, it’s far enough out of the norm that I >> need to rule it out. >> >> Please do not re-enable any of the disabled builds until we have had a >> chance to work on this together. Can you please identify someone from the >> project to act as a liaison with Infra to troubleshoot the issues with these >> builds? >> >> Since I have sent this to a private list, you have my permission to forward >> this message on to your devs or other public lists as you deem appropriate. >> >> >> Thanks, >> >> -Chris >> ASF Infra >> > > > > -- > Zoran Regvart
signature.asc
Description: Message signed with OpenPGP
