Re: Re: Re: Re: Server usage pegged at 99%
On Wed, 06 Feb 2013 21:13:42 +0100, Andrew Melo andrew.m...@gmail.com wrote: Hey everyone, Sorry for resurrecting that thread, but are there any solutions for this problem? It's basically killing our instance often. best, Andrew Hi, subversion-plugin in version 1.45 does not longer have MailAddressResolver implemented. It should not try to resolve your mail addresses. Could you confirm there is any difference four you after the upgrade? For more details see: https://issues.jenkins-ci.org/browse/JENKINS-16437. -- oliver -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Re: Re: Re: Server usage pegged at 99%
On Thu, Feb 7, 2013 at 4:10 AM, oliver gondža ogon...@gmail.com wrote: On Wed, 06 Feb 2013 21:13:42 +0100, Andrew Melo andrew.m...@gmail.com wrote: Hey everyone, Sorry for resurrecting that thread, but are there any solutions for this problem? It's basically killing our instance often. best, Andrew Hi, subversion-plugin in version 1.45 does not longer have MailAddressResolver implemented. It should not try to resolve your mail addresses. Could you confirm there is any difference four you after the upgrade? For more details see: https://issues.jenkins-ci.org/browse/JENKINS-16437. I'm upgrading now, then I'll report back. FWIW, is the git repository plugin also patched in the same way? That's the one with the most commits for us. thanks, Andrew -- oliver -- -- Andrew Melo -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Re: Re: Re: Server usage pegged at 99%
Hi Oliver, It doesn't appear to have helped. Hitting the asyncPeople link pegs the CPU and just stalls. FWIW, I'm on the LTS release and have updated all the plugins to their latest versions. -Andrew On Thu, Feb 7, 2013 at 10:46 AM, Andrew Melo andrew.m...@gmail.com wrote: On Thu, Feb 7, 2013 at 4:10 AM, oliver gondža ogon...@gmail.com wrote: On Wed, 06 Feb 2013 21:13:42 +0100, Andrew Melo andrew.m...@gmail.com wrote: Hey everyone, Sorry for resurrecting that thread, but are there any solutions for this problem? It's basically killing our instance often. best, Andrew Hi, subversion-plugin in version 1.45 does not longer have MailAddressResolver implemented. It should not try to resolve your mail addresses. Could you confirm there is any difference four you after the upgrade? For more details see: https://issues.jenkins-ci.org/browse/JENKINS-16437. I'm upgrading now, then I'll report back. FWIW, is the git repository plugin also patched in the same way? That's the one with the most commits for us. thanks, Andrew -- oliver -- -- Andrew Melo -- -- Andrew Melo -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Re: Re: Re: Server usage pegged at 99%
I'm also having an intermuitant 100% CPU problem. I found that the Jenkins Monitoring plugin was a big help in identifiying the offending threads. In my case, it is casued by looking at an EMMA code coverage report in the browser. -Richard On Thursday, February 7, 2013 9:47:20 AM UTC-8, Andrew Melo wrote: Hi Oliver, It doesn't appear to have helped. Hitting the asyncPeople link pegs the CPU and just stalls. FWIW, I'm on the LTS release and have updated all the plugins to their latest versions. -Andrew On Thu, Feb 7, 2013 at 10:46 AM, Andrew Melo andre...@gmail.comjavascript: wrote: On Thu, Feb 7, 2013 at 4:10 AM, oliver gondža ogo...@gmail.comjavascript: wrote: On Wed, 06 Feb 2013 21:13:42 +0100, Andrew Melo andre...@gmail.comjavascript: wrote: Hey everyone, Sorry for resurrecting that thread, but are there any solutions for this problem? It's basically killing our instance often. best, Andrew Hi, subversion-plugin in version 1.45 does not longer have MailAddressResolver implemented. It should not try to resolve your mail addresses. Could you confirm there is any difference four you after the upgrade? For more details see: https://issues.jenkins-ci.org/browse/JENKINS-16437. I'm upgrading now, then I'll report back. FWIW, is the git repository plugin also patched in the same way? That's the one with the most commits for us. thanks, Andrew -- oliver -- -- Andrew Melo -- -- Andrew Melo -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Re: Re: Re: Server usage pegged at 99%
Hey everyone, Sorry for resurrecting that thread, but are there any solutions for this problem? It's basically killing our instance often. best, Andrew On Wed, Aug 1, 2012 at 11:47 AM, Andrew Melo andrew.m...@gmail.com wrote: Oh, wow, I didn't notice, but jenkins has autopopulated a user for everyone that ever committed on the project. There's like 30 people, ~4000 commits, so I could see why that would take a while :) On Wed, Aug 1, 2012 at 11:42 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:37 AM, Slide slide.o@gmail.com wrote: Can you gist your global config.xml and something from one of your jobs as well? Please remember to sanitize it. We actually keep it stored in SCM. https://github.com/dmwm/jenkins/ And the following is the gist for the job we run each commit (didn't make it in for some reason...) https://gist.github.com/3228599 On Wed, Aug 1, 2012 at 9:34 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:26 AM, Slide slide.o@gmail.com wrote: No, because its only looking for the email address because it wants to send an email to that user. I don't know who's getting emailed. I don't remember setting it up for anything, and we actually wrote some scripts that turn jenkins success/failures into Github issues, so having jenkins also send emails would be redundant. I don't supposed there's a global flag to disable email? (I don't see one at manage jenkins) -Andrew On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:22 AM, Slide slide.o@gmail.com wrote: This is a huge issue with the email-ext plugin as well when it does email address resolution. Quite a number of people have complained about how long it takes. I have yet to come up with a good solution. The perforce plugin has a similar issue. If I just stick a dummy address in every user's profile, will that work? slide On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7
Re: Re: Re: Server usage pegged at 99%
On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-)
Re: Re: Re: Server usage pegged at 99%
On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java Then if I run jstack, I get the following backtrace: https://gist.github.com/3228105 Does that look useful at all? Thanks, Andrew -- -- Andrew Melo
Re: Re: Re: Re: Server usage pegged at 99%
Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java Then if I run jstack, I get the following backtrace: https://gist.github.com/3228105 Does that look useful at all? Thanks, Andrew
Re: Re: Re: Re: Server usage pegged at 99%
This is a huge issue with the email-ext plugin as well when it does email address resolution. Quite a number of people have complained about how long it takes. I have yet to come up with a good solution. The perforce plugin has a similar issue. slide On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java Then if I run jstack, I get the following backtrace: https://gist.github.com/3228105 Does that look useful at all? Thanks, Andrew -- Website: http://earl-of-code.com
Re: Re: Re: Re: Server usage pegged at 99%
On Wed, Aug 1, 2012 at 11:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly So, if I understand right, jenkins is looking for email addresses, so I need to make sure that every user that registers has a valid address? On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java Then if I run jstack, I get the following backtrace: https://gist.github.com/3228105 Does that look useful at all? Thanks, Andrew -- -- Andrew Melo
Re: Re: Re: Re: Server usage pegged at 99%
On Wed, Aug 1, 2012 at 11:22 AM, Slide slide.o@gmail.com wrote: This is a huge issue with the email-ext plugin as well when it does email address resolution. Quite a number of people have complained about how long it takes. I have yet to come up with a good solution. The perforce plugin has a similar issue. If I just stick a dummy address in every user's profile, will that work? slide On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java Then if I run jstack, I get the following backtrace: https://gist.github.com/3228105 Does that look useful at all? Thanks, Andrew -- Website: http://earl-of-code.com -- -- Andrew Melo
Re: Re: Re: Re: Server usage pegged at 99%
No, because its only looking for the email address because it wants to send an email to that user. On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:22 AM, Slide slide.o@gmail.com wrote: This is a huge issue with the email-ext plugin as well when it does email address resolution. Quite a number of people have complained about how long it takes. I have yet to come up with a good solution. The perforce plugin has a similar issue. If I just stick a dummy address in every user's profile, will that work? slide On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java Then if I run jstack, I get the following backtrace: https://gist.github.com/3228105 Does that look useful at all? Thanks, Andrew -- Website: http://earl-of-code.com -- -- Andrew Melo -- Website: http://earl-of-code.com
Re: Re: Re: Re: Server usage pegged at 99%
On Wed, Aug 1, 2012 at 11:26 AM, Slide slide.o@gmail.com wrote: No, because its only looking for the email address because it wants to send an email to that user. I don't know who's getting emailed. I don't remember setting it up for anything, and we actually wrote some scripts that turn jenkins success/failures into Github issues, so having jenkins also send emails would be redundant. I don't supposed there's a global flag to disable email? (I don't see one at manage jenkins) -Andrew On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:22 AM, Slide slide.o@gmail.com wrote: This is a huge issue with the email-ext plugin as well when it does email address resolution. Quite a number of people have complained about how long it takes. I have yet to come up with a good solution. The perforce plugin has a similar issue. If I just stick a dummy address in every user's profile, will that work? slide On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7
Re: Re: Re: Re: Server usage pegged at 99%
Can you gist your global config.xml and something from one of your jobs as well? Please remember to sanitize it. On Wed, Aug 1, 2012 at 9:34 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:26 AM, Slide slide.o@gmail.com wrote: No, because its only looking for the email address because it wants to send an email to that user. I don't know who's getting emailed. I don't remember setting it up for anything, and we actually wrote some scripts that turn jenkins success/failures into Github issues, so having jenkins also send emails would be redundant. I don't supposed there's a global flag to disable email? (I don't see one at manage jenkins) -Andrew On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:22 AM, Slide slide.o@gmail.com wrote: This is a huge issue with the email-ext plugin as well when it does email address resolution. Quite a number of people have complained about how long it takes. I have yet to come up with a good solution. The perforce plugin has a similar issue. If I just stick a dummy address in every user's profile, will that work? slide On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7
Re: Re: Re: Re: Server usage pegged at 99%
On Wed, Aug 1, 2012 at 11:37 AM, Slide slide.o@gmail.com wrote: Can you gist your global config.xml and something from one of your jobs as well? Please remember to sanitize it. We actually keep it stored in SCM. https://github.com/dmwm/jenkins/ And the following is the gist for the job we run each commit (didn't make it in for some reason...) https://gist.github.com/3228599 On Wed, Aug 1, 2012 at 9:34 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:26 AM, Slide slide.o@gmail.com wrote: No, because its only looking for the email address because it wants to send an email to that user. I don't know who's getting emailed. I don't remember setting it up for anything, and we actually wrote some scripts that turn jenkins success/failures into Github issues, so having jenkins also send emails would be redundant. I don't supposed there's a global flag to disable email? (I don't see one at manage jenkins) -Andrew On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:22 AM, Slide slide.o@gmail.com wrote: This is a huge issue with the email-ext plugin as well when it does email address resolution. Quite a number of people have complained about how long it takes. I have yet to come up with a good solution. The perforce plugin has a similar issue. If I just stick a dummy address in every user's profile, will that work? slide On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java 24973 jenkins 15 0 3246m 743m 18m S
Re: Re: Re: Re: Server usage pegged at 99%
Oh, wow, I didn't notice, but jenkins has autopopulated a user for everyone that ever committed on the project. There's like 30 people, ~4000 commits, so I could see why that would take a while :) On Wed, Aug 1, 2012 at 11:42 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:37 AM, Slide slide.o@gmail.com wrote: Can you gist your global config.xml and something from one of your jobs as well? Please remember to sanitize it. We actually keep it stored in SCM. https://github.com/dmwm/jenkins/ And the following is the gist for the job we run each commit (didn't make it in for some reason...) https://gist.github.com/3228599 On Wed, Aug 1, 2012 at 9:34 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:26 AM, Slide slide.o@gmail.com wrote: No, because its only looking for the email address because it wants to send an email to that user. I don't know who's getting emailed. I don't remember setting it up for anything, and we actually wrote some scripts that turn jenkins success/failures into Github issues, so having jenkins also send emails would be redundant. I don't supposed there's a global flag to disable email? (I don't see one at manage jenkins) -Andrew On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:22 AM, Slide slide.o@gmail.com wrote: This is a huge issue with the email-ext plugin as well when it does email address resolution. Quite a number of people have complained about how long it takes. I have yet to come up with a good solution. The perforce plugin has a similar issue. If I just stick a dummy address in every user's profile, will that work? slide On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0
