Re: Server usage pegged at 99%
I just see these: Feb 6, 2013 9:50:10 PM hudson.plugins.im.IMConnectionProvider$ConnectorRunnable run INFO: Reconnect failed. Next connection attempt in 16 minutes Feb 6, 2013 9:51:40 PM hudson.slaves.SlaveComputer tryReconnect INFO: Attempting to reconnect dmwm-agent-int-old Feb 6, 2013 10:01:53 PM winstone.Logger logInternal WARNING: Called getInputStream after getParameter ... error Feb 6, 2013 10:02:19 PM winstone.Logger logInternal WARNING: Called getInputStream after getParameter ... error Feb 6, 2013 10:02:21 PM winstone.Logger logInternal WARNING: Called getInputStream after getParameter ... error (UTC timestamps) While running a 3 matrix jobs with 10 configurations each and 10 slaves (with the web browser open), I get the following utilization PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 15209 jenkins 18 0 3366m 690m 18m S 177.2 22.9 203:51.39 java and the web interface is painfully slow. thanks, Andrew On Wed, Feb 6, 2013 at 2:57 PM, Sami Tikka sjti...@gmail.com wrote: Are there any errors/exceptions in Jenkins log files or stdout/stderr? -- Sami Andrew Melo andrew.m...@gmail.com kirjoitti 6.2.2013 kello 22.13: Hey everyone, Sorry for resurrecting that thread, but are there any solutions for this problem? It's basically killing our instance often. best, Andrew On Wed, Aug 1, 2012 at 11:47 AM, Andrew Melo andrew.m...@gmail.com wrote: Oh, wow, I didn't notice, but jenkins has autopopulated a user for everyone that ever committed on the project. There's like 30 people, ~4000 commits, so I could see why that would take a while :) On Wed, Aug 1, 2012 at 11:42 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:37 AM, Slide slide.o@gmail.com wrote: Can you gist your global config.xml and something from one of your jobs as well? Please remember to sanitize it. We actually keep it stored in SCM. https://github.com/dmwm/jenkins/ And the following is the gist for the job we run each commit (didn't make it in for some reason...) https://gist.github.com/3228599 On Wed, Aug 1, 2012 at 9:34 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:26 AM, Slide slide.o@gmail.com wrote: No, because its only looking for the email address because it wants to send an email to that user. I don't know who's getting emailed. I don't remember setting it up for anything, and we actually wrote some scripts that turn jenkins success/failures into Github issues, so having jenkins also send emails would be redundant. I don't supposed there's a global flag to disable email? (I don't see one at manage jenkins) -Andrew On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:22 AM, Slide slide.o@gmail.com wrote: This is a huge issue with the email-ext plugin as well when it does email address resolution. Quite a number of people have complained about how long it takes. I have yet to come up with a good solution. The perforce plugin has a similar issue. If I just stick a dummy address in every user's profile, will that work? slide On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins
Re: Re: Re: Re: Server usage pegged at 99%
On Wed, 06 Feb 2013 21:13:42 +0100, Andrew Melo andrew.m...@gmail.com wrote: Hey everyone, Sorry for resurrecting that thread, but are there any solutions for this problem? It's basically killing our instance often. best, Andrew Hi, subversion-plugin in version 1.45 does not longer have MailAddressResolver implemented. It should not try to resolve your mail addresses. Could you confirm there is any difference four you after the upgrade? For more details see: https://issues.jenkins-ci.org/browse/JENKINS-16437. -- oliver -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Re: Re: Re: Server usage pegged at 99%
On Thu, Feb 7, 2013 at 4:10 AM, oliver gondža ogon...@gmail.com wrote: On Wed, 06 Feb 2013 21:13:42 +0100, Andrew Melo andrew.m...@gmail.com wrote: Hey everyone, Sorry for resurrecting that thread, but are there any solutions for this problem? It's basically killing our instance often. best, Andrew Hi, subversion-plugin in version 1.45 does not longer have MailAddressResolver implemented. It should not try to resolve your mail addresses. Could you confirm there is any difference four you after the upgrade? For more details see: https://issues.jenkins-ci.org/browse/JENKINS-16437. I'm upgrading now, then I'll report back. FWIW, is the git repository plugin also patched in the same way? That's the one with the most commits for us. thanks, Andrew -- oliver -- -- Andrew Melo -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Re: Re: Re: Server usage pegged at 99%
Hi Oliver, It doesn't appear to have helped. Hitting the asyncPeople link pegs the CPU and just stalls. FWIW, I'm on the LTS release and have updated all the plugins to their latest versions. -Andrew On Thu, Feb 7, 2013 at 10:46 AM, Andrew Melo andrew.m...@gmail.com wrote: On Thu, Feb 7, 2013 at 4:10 AM, oliver gondža ogon...@gmail.com wrote: On Wed, 06 Feb 2013 21:13:42 +0100, Andrew Melo andrew.m...@gmail.com wrote: Hey everyone, Sorry for resurrecting that thread, but are there any solutions for this problem? It's basically killing our instance often. best, Andrew Hi, subversion-plugin in version 1.45 does not longer have MailAddressResolver implemented. It should not try to resolve your mail addresses. Could you confirm there is any difference four you after the upgrade? For more details see: https://issues.jenkins-ci.org/browse/JENKINS-16437. I'm upgrading now, then I'll report back. FWIW, is the git repository plugin also patched in the same way? That's the one with the most commits for us. thanks, Andrew -- oliver -- -- Andrew Melo -- -- Andrew Melo -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Re: Re: Re: Server usage pegged at 99%
I'm also having an intermuitant 100% CPU problem. I found that the Jenkins Monitoring plugin was a big help in identifiying the offending threads. In my case, it is casued by looking at an EMMA code coverage report in the browser. -Richard On Thursday, February 7, 2013 9:47:20 AM UTC-8, Andrew Melo wrote: Hi Oliver, It doesn't appear to have helped. Hitting the asyncPeople link pegs the CPU and just stalls. FWIW, I'm on the LTS release and have updated all the plugins to their latest versions. -Andrew On Thu, Feb 7, 2013 at 10:46 AM, Andrew Melo andre...@gmail.comjavascript: wrote: On Thu, Feb 7, 2013 at 4:10 AM, oliver gondža ogo...@gmail.comjavascript: wrote: On Wed, 06 Feb 2013 21:13:42 +0100, Andrew Melo andre...@gmail.comjavascript: wrote: Hey everyone, Sorry for resurrecting that thread, but are there any solutions for this problem? It's basically killing our instance often. best, Andrew Hi, subversion-plugin in version 1.45 does not longer have MailAddressResolver implemented. It should not try to resolve your mail addresses. Could you confirm there is any difference four you after the upgrade? For more details see: https://issues.jenkins-ci.org/browse/JENKINS-16437. I'm upgrading now, then I'll report back. FWIW, is the git repository plugin also patched in the same way? That's the one with the most commits for us. thanks, Andrew -- oliver -- -- Andrew Melo -- -- Andrew Melo -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Re: Re: Re: Server usage pegged at 99%
Hey everyone, Sorry for resurrecting that thread, but are there any solutions for this problem? It's basically killing our instance often. best, Andrew On Wed, Aug 1, 2012 at 11:47 AM, Andrew Melo andrew.m...@gmail.com wrote: Oh, wow, I didn't notice, but jenkins has autopopulated a user for everyone that ever committed on the project. There's like 30 people, ~4000 commits, so I could see why that would take a while :) On Wed, Aug 1, 2012 at 11:42 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:37 AM, Slide slide.o@gmail.com wrote: Can you gist your global config.xml and something from one of your jobs as well? Please remember to sanitize it. We actually keep it stored in SCM. https://github.com/dmwm/jenkins/ And the following is the gist for the job we run each commit (didn't make it in for some reason...) https://gist.github.com/3228599 On Wed, Aug 1, 2012 at 9:34 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:26 AM, Slide slide.o@gmail.com wrote: No, because its only looking for the email address because it wants to send an email to that user. I don't know who's getting emailed. I don't remember setting it up for anything, and we actually wrote some scripts that turn jenkins success/failures into Github issues, so having jenkins also send emails would be redundant. I don't supposed there's a global flag to disable email? (I don't see one at manage jenkins) -Andrew On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:22 AM, Slide slide.o@gmail.com wrote: This is a huge issue with the email-ext plugin as well when it does email address resolution. Quite a number of people have complained about how long it takes. I have yet to come up with a good solution. The perforce plugin has a similar issue. If I just stick a dummy address in every user's profile, will that work? slide On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7
Re: Server usage pegged at 99%
Are there any errors/exceptions in Jenkins log files or stdout/stderr? -- Sami Andrew Melo andrew.m...@gmail.com kirjoitti 6.2.2013 kello 22.13: Hey everyone, Sorry for resurrecting that thread, but are there any solutions for this problem? It's basically killing our instance often. best, Andrew On Wed, Aug 1, 2012 at 11:47 AM, Andrew Melo andrew.m...@gmail.com wrote: Oh, wow, I didn't notice, but jenkins has autopopulated a user for everyone that ever committed on the project. There's like 30 people, ~4000 commits, so I could see why that would take a while :) On Wed, Aug 1, 2012 at 11:42 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:37 AM, Slide slide.o@gmail.com wrote: Can you gist your global config.xml and something from one of your jobs as well? Please remember to sanitize it. We actually keep it stored in SCM. https://github.com/dmwm/jenkins/ And the following is the gist for the job we run each commit (didn't make it in for some reason...) https://gist.github.com/3228599 On Wed, Aug 1, 2012 at 9:34 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:26 AM, Slide slide.o@gmail.com wrote: No, because its only looking for the email address because it wants to send an email to that user. I don't know who's getting emailed. I don't remember setting it up for anything, and we actually wrote some scripts that turn jenkins success/failures into Github issues, so having jenkins also send emails would be redundant. I don't supposed there's a global flag to disable email? (I don't see one at manage jenkins) -Andrew On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:22 AM, Slide slide.o@gmail.com wrote: This is a huge issue with the email-ext plugin as well when it does email address resolution. Quite a number of people have complained about how long it takes. I have yet to come up with a good solution. The perforce plugin has a similar issue. If I just stick a dummy address in every user's profile, will that work? slide On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m
Re: Server usage pegged at 99%
On Tue, Jul 31, 2012 at 2:54 PM, Les Mikesell lesmikes...@gmail.com wrote: On Tue, Jul 31, 2012 at 2:38 PM, Andrew Melo andrew.m...@gmail.com wrote: But if it happened before June 30th or the system has been rebooted since, this is not the problem. Well, and it's only when i'm using the web interface (or if background stuff is happening) It affects the linux futex() system call that is used mostly in threaded applications (so you see it in java). And I think it is sort of a race condition where the extra CPU use happens at random. Well, I restarted it and reset the date and it didn't seem to help. I'm pretty helpless when it comes to java, but is there some sort of way I can attach a profiler to the process and see what it spins on? Thanks -- Les Mikesell lesmikes...@gmail.com -- -- Andrew Melo
Re: Re: Server usage pegged at 99%
quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ On Wednesday 01 August 2012 09:25:08 Andrew Melo wrote: On Tue, Jul 31, 2012 at 2:54 PM, Les Mikesell lesmikes...@gmail.com wrote: On Tue, Jul 31, 2012 at 2:38 PM, Andrew Melo andrew.m...@gmail.com wrote: But if it happened before June 30th or the system has been rebooted since, this is not the problem. Well, and it's only when i'm using the web interface (or if background stuff is happening) It affects the linux futex() system call that is used mostly in threaded applications (so you see it in java). And I think it is sort of a race condition where the extra CPU use happens at random. Well, I restarted it and reset the date and it didn't seem to help. I'm pretty helpless when it comes to java, but is there some sort of way I can attach a profiler to the process and see what it spins on? Thanks -- Les Mikesell lesmikes...@gmail.com
Re: Re: Server usage pegged at 99%
On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? On Wednesday 01 August 2012 09:25:08 Andrew Melo wrote: On Tue, Jul 31, 2012 at 2:54 PM, Les Mikesell lesmikes...@gmail.com wrote: On Tue, Jul 31, 2012 at 2:38 PM, Andrew Melo andrew.m...@gmail.com wrote: But if it happened before June 30th or the system has been rebooted since, this is not the problem. Well, and it's only when i'm using the web interface (or if background stuff is happening) It affects the linux futex() system call that is used mostly in threaded applications (so you see it in java). And I think it is sort of a race condition where the extra CPU use happens at random. Well, I restarted it and reset the date and it didn't seem to help. I'm pretty helpless when it comes to java, but is there some sort of way I can attach a profiler to the process and see what it spins on? Thanks -- Les Mikesell lesmikes...@gmail.com -- -- Andrew Melo
Re: Re: Re: Server usage pegged at 99%
On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-)
Re: Re: Re: Server usage pegged at 99%
On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java Then if I run jstack, I get the following backtrace: https://gist.github.com/3228105 Does that look useful at all? Thanks, Andrew -- -- Andrew Melo
Re: Re: Re: Re: Server usage pegged at 99%
Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java Then if I run jstack, I get the following backtrace: https://gist.github.com/3228105 Does that look useful at all? Thanks, Andrew
Re: Re: Re: Re: Server usage pegged at 99%
This is a huge issue with the email-ext plugin as well when it does email address resolution. Quite a number of people have complained about how long it takes. I have yet to come up with a good solution. The perforce plugin has a similar issue. slide On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java Then if I run jstack, I get the following backtrace: https://gist.github.com/3228105 Does that look useful at all? Thanks, Andrew -- Website: http://earl-of-code.com
Re: Re: Re: Re: Server usage pegged at 99%
On Wed, Aug 1, 2012 at 11:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly So, if I understand right, jenkins is looking for email addresses, so I need to make sure that every user that registers has a valid address? On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java Then if I run jstack, I get the following backtrace: https://gist.github.com/3228105 Does that look useful at all? Thanks, Andrew -- -- Andrew Melo
Re: Re: Re: Re: Server usage pegged at 99%
On Wed, Aug 1, 2012 at 11:22 AM, Slide slide.o@gmail.com wrote: This is a huge issue with the email-ext plugin as well when it does email address resolution. Quite a number of people have complained about how long it takes. I have yet to come up with a good solution. The perforce plugin has a similar issue. If I just stick a dummy address in every user's profile, will that work? slide On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java Then if I run jstack, I get the following backtrace: https://gist.github.com/3228105 Does that look useful at all? Thanks, Andrew -- Website: http://earl-of-code.com -- -- Andrew Melo
Re: Re: Re: Re: Server usage pegged at 99%
No, because its only looking for the email address because it wants to send an email to that user. On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:22 AM, Slide slide.o@gmail.com wrote: This is a huge issue with the email-ext plugin as well when it does email address resolution. Quite a number of people have complained about how long it takes. I have yet to come up with a good solution. The perforce plugin has a similar issue. If I just stick a dummy address in every user's profile, will that work? slide On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java Then if I run jstack, I get the following backtrace: https://gist.github.com/3228105 Does that look useful at all? Thanks, Andrew -- Website: http://earl-of-code.com -- -- Andrew Melo -- Website: http://earl-of-code.com
Re: Re: Re: Re: Server usage pegged at 99%
On Wed, Aug 1, 2012 at 11:26 AM, Slide slide.o@gmail.com wrote: No, because its only looking for the email address because it wants to send an email to that user. I don't know who's getting emailed. I don't remember setting it up for anything, and we actually wrote some scripts that turn jenkins success/failures into Github issues, so having jenkins also send emails would be redundant. I don't supposed there's a global flag to disable email? (I don't see one at manage jenkins) -Andrew On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:22 AM, Slide slide.o@gmail.com wrote: This is a huge issue with the email-ext plugin as well when it does email address resolution. Quite a number of people have complained about how long it takes. I have yet to come up with a good solution. The perforce plugin has a similar issue. If I just stick a dummy address in every user's profile, will that work? slide On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7
Re: Re: Re: Re: Server usage pegged at 99%
Can you gist your global config.xml and something from one of your jobs as well? Please remember to sanitize it. On Wed, Aug 1, 2012 at 9:34 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:26 AM, Slide slide.o@gmail.com wrote: No, because its only looking for the email address because it wants to send an email to that user. I don't know who's getting emailed. I don't remember setting it up for anything, and we actually wrote some scripts that turn jenkins success/failures into Github issues, so having jenkins also send emails would be redundant. I don't supposed there's a global flag to disable email? (I don't see one at manage jenkins) -Andrew On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:22 AM, Slide slide.o@gmail.com wrote: This is a huge issue with the email-ext plugin as well when it does email address resolution. Quite a number of people have complained about how long it takes. I have yet to come up with a good solution. The perforce plugin has a similar issue. If I just stick a dummy address in every user's profile, will that work? slide On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7
Re: Re: Re: Re: Server usage pegged at 99%
On Wed, Aug 1, 2012 at 11:37 AM, Slide slide.o@gmail.com wrote: Can you gist your global config.xml and something from one of your jobs as well? Please remember to sanitize it. We actually keep it stored in SCM. https://github.com/dmwm/jenkins/ And the following is the gist for the job we run each commit (didn't make it in for some reason...) https://gist.github.com/3228599 On Wed, Aug 1, 2012 at 9:34 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:26 AM, Slide slide.o@gmail.com wrote: No, because its only looking for the email address because it wants to send an email to that user. I don't know who's getting emailed. I don't remember setting it up for anything, and we actually wrote some scripts that turn jenkins success/failures into Github issues, so having jenkins also send emails would be redundant. I don't supposed there's a global flag to disable email? (I don't see one at manage jenkins) -Andrew On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:22 AM, Slide slide.o@gmail.com wrote: This is a huge issue with the email-ext plugin as well when it does email address resolution. Quite a number of people have complained about how long it takes. I have yet to come up with a good solution. The perforce plugin has a similar issue. If I just stick a dummy address in every user's profile, will that work? slide On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java 24973 jenkins 15 0 3246m 743m 18m S
Re: Re: Re: Re: Server usage pegged at 99%
Oh, wow, I didn't notice, but jenkins has autopopulated a user for everyone that ever committed on the project. There's like 30 people, ~4000 commits, so I could see why that would take a while :) On Wed, Aug 1, 2012 at 11:42 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:37 AM, Slide slide.o@gmail.com wrote: Can you gist your global config.xml and something from one of your jobs as well? Please remember to sanitize it. We actually keep it stored in SCM. https://github.com/dmwm/jenkins/ And the following is the gist for the job we run each commit (didn't make it in for some reason...) https://gist.github.com/3228599 On Wed, Aug 1, 2012 at 9:34 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:26 AM, Slide slide.o@gmail.com wrote: No, because its only looking for the email address because it wants to send an email to that user. I don't know who's getting emailed. I don't remember setting it up for anything, and we actually wrote some scripts that turn jenkins success/failures into Github issues, so having jenkins also send emails would be redundant. I don't supposed there's a global flag to disable email? (I don't see one at manage jenkins) -Andrew On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo andrew.m...@gmail.com wrote: On Wed, Aug 1, 2012 at 11:22 AM, Slide slide.o@gmail.com wrote: This is a huge issue with the email-ext plugin as well when it does email address resolution. Quite a number of people have complained about how long it takes. I have yet to come up with a good solution. The perforce plugin has a similar issue. If I just stick a dummy address in every user's profile, will that work? slide On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek vjura...@redhat.com wrote: Looks like you it does search for user's email: hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek vjura...@redhat.com wrote: On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek vjura...@redhat.com wrote: quick way how to look what the thread consuming CPU is doing is to do thread dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0
Server usage pegged at 99%
Hello, For some reason, if I watch top on my master while I browse around on the web interface, the java process stays pegged at 99-100% and it takes several seconds to render. Is there a common reason for that? I have 2Gb allocated to jenkins and the res is only 750MB, so I don't think it's GC churn. Thanks, andrew -- -- Andrew Melo
Re: Server usage pegged at 99%
On Tue, Jul 31, 2012 at 1:46 PM, Andrew Melo andrew.m...@gmail.com wrote: Hello, For some reason, if I watch top on my master while I browse around on the web interface, the java process stays pegged at 99-100% and it takes several seconds to render. Is there a common reason for that? I have 2Gb allocated to jenkins and the res is only 750MB, so I don't think it's GC churn. Is it running on a linux host that has been up since at least June 30th? If so it could be the leap-second bug. If so, resetting the time will fix it. -- Les Mikesell lesmikes...@gmail.com
Re: Server usage pegged at 99%
On Tue, Jul 31, 2012 at 2:16 PM, Les Mikesell lesmikes...@gmail.com wrote: On Tue, Jul 31, 2012 at 1:46 PM, Andrew Melo andrew.m...@gmail.com wrote: Hello, For some reason, if I watch top on my master while I browse around on the web interface, the java process stays pegged at 99-100% and it takes several seconds to render. Is there a common reason for that? I have 2Gb allocated to jenkins and the res is only 750MB, so I don't think it's GC churn. Is it running on a linux host that has been up since at least June 30th? If so it could be the leap-second bug. If so, resetting the time will fix it. Resetting the time (ntp?) or the jenkins server? Either way, it's been an issue since before then, but I'll give whichever one a try Thanks, Andrew -- Les Mikesell lesmikes...@gmail.com -- -- Andrew Melo
Re: Server usage pegged at 99%
On Tue, Jul 31, 2012 at 2:17 PM, Andrew Melo andrew.m...@gmail.com wrote: For some reason, if I watch top on my master while I browse around on the web interface, the java process stays pegged at 99-100% and it takes several seconds to render. Is there a common reason for that? I have 2Gb allocated to jenkins and the res is only 750MB, so I don't think it's GC churn. Is it running on a linux host that has been up since at least June 30th? If so it could be the leap-second bug. If so, resetting the time will fix it. Resetting the time (ntp?) or the jenkins server? Either way, it's been an issue since before then, but I'll give whichever one a try It is a linux kernel bug triggered by ntp on the day of a leap second. Resetting the system time any way other than ntp will fix it. For example: date -s `date` But if it happened before June 30th or the system has been rebooted since, this is not the problem. -- Les Mikesell lesmikes...@gmail.com
Re: Server usage pegged at 99%
On Tue, Jul 31, 2012 at 2:36 PM, Les Mikesell lesmikes...@gmail.com wrote: On Tue, Jul 31, 2012 at 2:17 PM, Andrew Melo andrew.m...@gmail.com wrote: For some reason, if I watch top on my master while I browse around on the web interface, the java process stays pegged at 99-100% and it takes several seconds to render. Is there a common reason for that? I have 2Gb allocated to jenkins and the res is only 750MB, so I don't think it's GC churn. Is it running on a linux host that has been up since at least June 30th? If so it could be the leap-second bug. If so, resetting the time will fix it. Resetting the time (ntp?) or the jenkins server? Either way, it's been an issue since before then, but I'll give whichever one a try It is a linux kernel bug triggered by ntp on the day of a leap second. Resetting the system time any way other than ntp will fix it. For example: date -s `date` But if it happened before June 30th or the system has been rebooted since, this is not the problem. Well, and it's only when i'm using the web interface (or if background stuff is happening) -- Les Mikesell lesmikes...@gmail.com -- -- Andrew Melo
Re: Server usage pegged at 99%
On Tue, Jul 31, 2012 at 2:38 PM, Andrew Melo andrew.m...@gmail.com wrote: But if it happened before June 30th or the system has been rebooted since, this is not the problem. Well, and it's only when i'm using the web interface (or if background stuff is happening) It affects the linux futex() system call that is used mostly in threaded applications (so you see it in java). And I think it is sort of a race condition where the extra CPU use happens at random. -- Les Mikesell lesmikes...@gmail.com
