I'm forwarding this email from Chris Lambertus of Apache Infra. Please, for who is able to do this, avoid to re-enable the Camel jobs until we find a solution. Thanks. --Andrea Cosentino ----------------------------------Apache Camel PMC MemberApache Karaf CommitterApache Servicemix PMC MemberEmail: ancosen1985@yahoo.comTwitter: @oscerd2Github: oscerd
----- Forwarded Message ----- From: Chris Lambertus <c...@apache.org>To: "priv...@camel.apache.org" <priv...@camel.apache.org>Cc: "priv...@infra.apache.org" <priv...@infra.apache.org>Sent: Friday, February 16, 2018, 2:37:24 AM GMT+1Subject: camel builds causing problems with jenkins Hi Camel PMC, We have been having an ongoing problem with Jenkins for quite some time, where the CPU usage and IOPS skyrocket on the master. Each time this has happened, the jenkins build nodes lose all of their associated labels, and all new builds are unable to start. In the times I’ve been able to investigate this, there has in each case been several Camel builds running, and in each case, the builds are opening somewhere between 1500 and 2200 log files, which seems to be killing the jenkins master. For comparison, some very large build jobs for Hadoop only open ~15 or so log files. I have had to take the rather drastic step of disabling the Camel Jenkins jobs (many of which have been failing for awhile now) while we continue to investigate this issue. Before we re-enable the jobs, we’re going to have to figure out how to get your builds to open a sane number of log files — the current situation where the builds are creating thousands of log files is not sustainable, and we believe this may be one of the causative factors of the ongoing jenkins outages. While I cannot say with any certainty that this is what’s been killing the master, it’s far enough out of the norm that I need to rule it out. Please do not re-enable any of the disabled builds until we have had a chance to work on this together. Can you please identify someone from the project to act as a liaison with Infra to troubleshoot the issues with these builds? Since I have sent this to a private list, you have my permission to forward this message on to your devs or other public lists as you deem appropriate. Thanks, -Chris ASF Infra
