Hi Gabriel, It seems there is no response yet.
Perhaps, you can give a summary of what are the types of the 1750 threads after startup? (start with a thread dump using jstack for example) And similarly what are the types of the 150 threads created each day? Have you tried to define the -Xss or the -XX:ThreadStackSize parameters when starting the JVM? For example, - XX:ThreadStackSize=256 ? And may I ask where do you have found 350 slaves? bye, Emeric On Jan 23, 7:08 pm, Gabriel Peisah <[email protected]> wrote: > Hi All, > > We are using Jenkins CI application for about 2 years. > 3 weeks ago we upgraded it to version 1.443. > > Since upgrade we started to face CPU performance issues. > I would appreciate if you will be able to advise with the problem resolution. > > Our Jenkins is configured and installed with following parameters: > > 1) Jenkins has about 350 nodes > > 2) Jenkins has up to 1000 jobs > > 3) Jenkins has up about 30-40 jobs executed concurrently > > 4) Jenkins is restarted on weekly basis on Sunday 06:00 AM > > 5) Jenkins is installed on Virtual Linux machine platform with: > > a. operation system Redhat 5.6 > > b. 4 CPU > > c. 16 GB memory > > 6) Jenkins using SUN JAVA version 1.6.0.17 > > 7) Jenkins application is started with 8GB memory > > The problem: > > 1) After the restart Jenkins starts with about 1750 Threads which uses > about 40% total CPU capacity. > > 2) Each day Java Threads number grows in 150 threads and CPU consumption > increasing by 20-30% > > 3) After few days of Jenkins run, it starts consuming about 100% of CPU > which cause Jenkins application to stuck > > 4) Each action on the UI stuck as well > > We tried below solutions by unfortunately it didn't help: > > 1) Node status change per request > > From the beginning all nodes were online, we thought that it was the problem > since per node there is a thread that trying to ping a node for checking its > status on ongoing basis. > > So we changed their status to be offline by default. > > When there is a Job to run, so the node's status become online and once Job's > execution is completed the node's status changed back to offline. > > But this resolution didn't help. Number of threads after Jenkins restart > remained the same as it was before changing nodes status. > > 2) Increase Number of CPUs > > From the beginning on the Virtual Machine where Jenkins installed we had 1 > CPU. > > We thought it can cause the problem, so we increased to 2 CPUs and after to 4 > CPUs but the problem still remained. > > 3) Fixed windows servers slave definition, removed double listeners > > 4) Found all jobs without specific node and fixed > > 5) Defined all nodes to accept job originated to them only > > 6) Turn offline all windows servers that are not in use for long time > > 7) Found zombie scm polling jobs and kill them > > 8) Fixed all scm polling jobs that were utilizing remote proxy server > instead a local > > Thanks In Adance, > Gabriel > > This message and the information contained herein is proprietary and > confidential and subject to the Amdocs policy statement, > you may review athttp://www.amdocs.com/email_disclaimer.asp
