We upgraded our Jenkins instance yesterday (from version 2.190.1 to 2.249.1) and the heavy load on the master "magically disappeared". The exact same test suite now runs (and 'spams' all the agents), but the load on master stays below a value of 3 (with 4 CPUs).
On Friday, September 25, 2020 at 5:05:11 PM UTC+2 Dominik wrote: > Hi Kamil, > > Thanks for the link. Although we do use pipelines for many projects, the > set of test jobs I mentioned are plain old 'freestyle' jobs. > They include one or multiple Maven build steps which run some more or less > heavy unit and integration tests (which all happens on agents anyway). > When looking at top/htop on the server during those tests, it's really > just the Jenkins main process rotating at 380-400% CPU usage (of 4 CPUs). > > Regards, > Dominik > > On Friday, September 25, 2020 at 3:28:23 PM UTC+2 kamil.dzi...@vimn.com > wrote: > >> Hi Dominik, >> >> High load on the master might be caused by incorrectly implemented >> Jenkinsfiles which contain complex logic, thus inducing load on the master >> instead of the nodes. Furthermore, this load will not be visible in the >> node overview. I think this is the best point to start investigating this >> issue from, see this piece of documentation for explanation: >> https://www.jenkins.io/doc/book/pipeline/pipeline-best-practices/#making-sure-to-use-groovy-code-in-pipelines-as-glue >> >> Best regards, >> Kamil >> >> On Friday, September 25, 2020 at 2:26:47 PM UTC+2 Dominik wrote: >> >>> Hi, >>> >>> We are running Jenkins with about 20 build agents on separate VMs (with >>> up to 4 executors each). The master has 4 CPUs, so I would expect a load of >>> about 4 when there's "a lot of work" to be done. >>> >>> However, no job is ever scheduled on the master itself, and still we can >>> observe very high load levels on the master when many jobs are running >>> concurrently on the agents. For example, one of our test suites launches >>> about 30 jobs (lasting 5-20min each) in a 1h time slot and at that moment >>> the master reaches a load of up to 40, with all CPUs fully occupied! >>> However, looking at the UI, all jobs clearly run on the agents, and the >>> master simply shows as 'Idle'. >>> >>> When I look at Jenkins' own monitoring, I see a huge amount of >>> *Computer.threadPoolForRemoting >>> *and similar threads with a summary stating *Threads on master: *Number >>> = 415, Maximum = 511, Total started = 7,731,717. >>> >>> Can anyone give some insight on what could be going on? What can be the >>> reasons for high load on the master when all the jobs are running on agents? >>> >>> Regards, >>> Dominik >>> >> -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/f170a0e9-2e47-451a-9483-d2fd363aced9n%40googlegroups.com.
