The problem is entirely in the per module files on the master - with Pipeline, you're only going to call junit once, which will read all the relevant files on the agent and write just one file on the master. It'll definitely work much, much better. Feel free to reach out to me (my day job is working on Jenkins Pipeline internals and usability) if you need any help with Pipeline.
A. On Feb 16, 2018 15:22, "Zoran Regvart" <zo...@regvart.com> wrote: > Thank Chris you for clarifying, > those log files are created by Jenkins Maven support[1], and they are > created for each Maven module of the project that's being built, with > Camel having 840 maven modules. I don't think this approach scales > with such big number of Maven modules. > > So Camel devs, seems like we need to switch to a different Jenkins job > type, I propose that we switch to Jenkins pipeline job type. At some > point I've been experimenting with that[2], and I can lead this > effort. > > I wonder, though, if the same issue will pop up with the JUnit test > reports archiving in the pipeline job, there will still be roughly the > same amount of files transferred to master, it could be that there has > been some work on Jenkins side to optimize that with pipeline. > > zoran > > [1] https://github.com/jenkinsci/maven-plugin/blob/ > [2] https://github.com/zregvart/camel/blob/jenkinsfile/Jenkinsfile > > On Fri, Feb 16, 2018 at 6:39 PM, Chris Lambertus <c...@apache.org> wrote: > > Here is a small sample of the types of logs we’re seeing: > > > > cml@jenkins-master:~$ cat camel-20180215.lsof | grep camel | grep log | > wc -l > > 2277 > > > > > > java 18713 jenkins 4931w REG 8,17 0 > 215356692 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/ > modules/org.apache.camel.example$camel-example-twitter- > salesforce/builds/1328/log > > java 18713 jenkins 4932w REG 8,17 0 > 215356694 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/ > modules/org.apache.camel.example$camel-example-twitter- > websocket/builds/1328/log > > java 18713 jenkins 4933w REG 8,17 0 > 215356696 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/ > modules/org.apache.camel.example$camel-example-twitter- > websocket-blueprint/builds/1328/log > > java 18713 jenkins 4934w REG 8,17 0 > 215356698 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/ > modules/org.apache.camel.example$camel-example- > validator-spring-boot/builds/1328/log > > java 18713 jenkins 4935w REG 8,17 0 > 215356701 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/ > modules/org.apache.camel.example$camel-example-widget- > gadget-cdi/builds/1328/log > > java 18713 jenkins 4936w REG 8,17 0 > 215356706 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/ > modules/org.apache.camel.example$camel-example-widget- > gadget-java/builds/1328/log > > java 18713 jenkins 4937w REG 8,17 0 > 215356709 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/ > modules/org.apache.camel.example$camel-example-widget- > gadget-xml/builds/1328/log > > java 18713 jenkins 4938w REG 8,17 0 > 215356714 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/ > modules/org.apache.camel.example$camel-example-zipkin/builds/1328/log > > java 18713 jenkins 4939w REG 8,17 0 > 215356725 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/ > modules/org.apache.camel.example$camel-example-zipkin- > client/builds/1328/log > > > > > > > >> On Feb 16, 2018, at 3:56 AM, Zoran Regvart <zo...@regvart.com> wrote: > >> > >> Hi Chris, > >> thank you for troubleshooting this, can you clarify one thing for me, > >> when you mention log files, are these the `.log` files generated > >> during the test phase of the build or the XML/TXT files with JUnit > >> reports? > >> > >> I would think that the job type being Maven and the automatic > >> gathering of JUnit test reports is the culprit but would like a > >> confirmation. > >> > >> If it is so, I think we one possible solution is to migrate to > >> Pipeline/Freestyle job type as, as far as I'm aware, there is no way > >> to prevent Maven job type from gathering JUnit reports. > >> > >> zoran > >> > >> On Fri, Feb 16, 2018 at 2:37 AM, Chris Lambertus <c...@apache.org> > wrote: > >>> > >>> Hi Camel PMC, > >>> > >>> We have been having an ongoing problem with Jenkins for quite some > time, where the CPU usage and IOPS skyrocket on the master. Each time this > has happened, the jenkins build nodes lose all of their associated labels, > and all new builds are unable to start. > >>> > >>> In the times I’ve been able to investigate this, there has in each > case been several Camel builds running, and in each case, the builds are > opening somewhere between 1500 and 2200 log files, which seems to be > killing the jenkins master. For comparison, some very large build jobs for > Hadoop only open ~15 or so log files. > >>> > >>> I have had to take the rather drastic step of disabling the Camel > Jenkins jobs (many of which have been failing for awhile now) while we > continue to investigate this issue. Before we re-enable the jobs, we’re > going to have to figure out how to get your builds to open a sane number of > log files — the current situation where the builds are creating thousands > of log files is not sustainable, and we believe this may be one of the > causative factors of the ongoing jenkins outages. While I cannot say with > any certainty that this is what’s been killing the master, it’s far enough > out of the norm that I need to rule it out. > >>> > >>> Please do not re-enable any of the disabled builds until we have had a > chance to work on this together. Can you please identify someone from the > project to act as a liaison with Infra to troubleshoot the issues with > these builds? > >>> > >>> Since I have sent this to a private list, you have my permission to > forward this message on to your devs or other public lists as you deem > appropriate. > >>> > >>> > >>> Thanks, > >>> > >>> -Chris > >>> ASF Infra > >>> > >> > >> > >> > >> -- > >> Zoran Regvart > > > > > > -- > Zoran Regvart >
