Re: Jenkins Hung Load at 1.02 / 99.8% CPU Load And Holding For 1 Hour
Hello Nate, Thank you for the tip. The issue had gone on too long and I had to manually restart Jenkins. Unfortunately that was just before I received your message so I could not use the kill -QUIT command. If this problem happens again I'll reply back with the details you suggested. Eric On Monday, April 15, 2013 10:16:41 AM UTC-7, Nate Gelbard wrote: > > Get a thread dump from the jenkins process. > kill -QUIT > > Attach to jenkins with VisualVM. are you out of heap and Garbage > Collecting really hard? > Might need to increase your java heap. > > > > On Mon, Apr 15, 2013 at 9:40 AM, Eric Blom >wrote: > >> Hello Everyone, >> >> I'm new to Jenkins and Java and am looking for some tips on how to debug >> my Jenkins server hanging. When I say hanging I mean that the Jenkins web >> page does not load. No errors are returned the page just continuously loads >> but nothing appears. >> >> My Jenkins server current "hang time" is a little over an hour: >> * Load at 1.02 / 99.8% CPU Load And Holding For 1 Hour >> >> Configuration >> * Jenkins: 1.480.3 (LTS) >> * OS: Red Hat 4.1.2-46 >> * Nodes: ~20 distributed globally >> * Jobs: ~100 >> >> Since all HTTP interfaces are hung I've been looking for a helpful log >> file on the server but haven't found one. When I look in the Jenkins home >> directory I see that no files have been modified for the past hour (while >> the load is at ~1.0). The last file to be modified was ThinBackup Worker >> Thread.log. 17 minutes before that nearly all of the node logs were >> updated. >> >> Other services on the machine, such as Apache and MySQL, are responding >> normally. >> >> Any suggestions on how to debug this would be greatly appreciated. As >> soon as the server comes back I'm going to uninstall ThinBackup, but, I'd >> like to learn what exactly Jenkins has been trying to do for the past hour. >> >> Cheers, >> Eric >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to jenkinsci-use...@googlegroups.com . >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Jenkins Hung Load at 1.02 / 99.8% CPU Load And Holding For 1 Hour
Get a thread dump from the jenkins process. kill -QUIT Attach to jenkins with VisualVM. are you out of heap and Garbage Collecting really hard? Might need to increase your java heap. On Mon, Apr 15, 2013 at 9:40 AM, Eric Blom wrote: > Hello Everyone, > > I'm new to Jenkins and Java and am looking for some tips on how to debug > my Jenkins server hanging. When I say hanging I mean that the Jenkins web > page does not load. No errors are returned the page just continuously loads > but nothing appears. > > My Jenkins server current "hang time" is a little over an hour: > * Load at 1.02 / 99.8% CPU Load And Holding For 1 Hour > > Configuration > * Jenkins: 1.480.3 (LTS) > * OS: Red Hat 4.1.2-46 > * Nodes: ~20 distributed globally > * Jobs: ~100 > > Since all HTTP interfaces are hung I've been looking for a helpful log > file on the server but haven't found one. When I look in the Jenkins home > directory I see that no files have been modified for the past hour (while > the load is at ~1.0). The last file to be modified was ThinBackup Worker > Thread.log. 17 minutes before that nearly all of the node logs were > updated. > > Other services on the machine, such as Apache and MySQL, are responding > normally. > > Any suggestions on how to debug this would be greatly appreciated. As soon > as the server comes back I'm going to uninstall ThinBackup, but, I'd like > to learn what exactly Jenkins has been trying to do for the past hour. > > Cheers, > Eric > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to jenkinsci-users+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Jenkins Hung Load at 1.02 / 99.8% CPU Load And Holding For 1 Hour
Hello Everyone, I'm new to Jenkins and Java and am looking for some tips on how to debug my Jenkins server hanging. When I say hanging I mean that the Jenkins web page does not load. No errors are returned the page just continuously loads but nothing appears. My Jenkins server current "hang time" is a little over an hour: * Load at 1.02 / 99.8% CPU Load And Holding For 1 Hour Configuration * Jenkins: 1.480.3 (LTS) * OS: Red Hat 4.1.2-46 * Nodes: ~20 distributed globally * Jobs: ~100 Since all HTTP interfaces are hung I've been looking for a helpful log file on the server but haven't found one. When I look in the Jenkins home directory I see that no files have been modified for the past hour (while the load is at ~1.0). The last file to be modified was ThinBackup Worker Thread.log. 17 minutes before that nearly all of the node logs were updated. Other services on the machine, such as Apache and MySQL, are responding normally. Any suggestions on how to debug this would be greatly appreciated. As soon as the server comes back I'm going to uninstall ThinBackup, but, I'd like to learn what exactly Jenkins has been trying to do for the past hour. Cheers, Eric -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Jenkins Hung
Even better would be the system log that collects the output of the jvm process. On a Mac you can look at the system log using /Applications/Utilities/Console.app. The log is also available in the file /var/log/system.log -- Sami 2012/3/9 Didier Durand : > Hi, > > you can access the system log via Jenkins Homepage > Manage Jenkins > System > log > > regards > > didier > > Le mercredi 7 mars 2012 20:24:16 UTC+1, John Hinnegan a écrit : >> >> Jenkins hung on me again. It just stops responding to requests. I wish it >> would just die so it would be restarted. >> >> Anyway, following >> along: https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+is+hanging >> >> Here's the JStack: http://pastebin.com/ZSNQgCm7 >> Generated with sudo jstack -F 64871 >> >> Heres's the JMap: http://pastebin.com/Mytcsnpv >> Generated with sudo jmap -heap 64871 >> >> It's OSX. My version of JMap didn't seem to understand what the >> -dump:format=b,file=heap.hprof tip in the wiki was going for. Can someone >> give me the command line to see how that would work? >> >> Also, I tried to get the core dump, but it seems on OS X they are >> suppressed by default, at least according >> to: http://stackoverflow.com/questions/2080918/where-are-core-dumps-written-to-in-mac-os-x >> >> I didn't know to run the ulimit command first, so it seems I don't have a >> dump to share. >> >> Suggestions on what to do next time to help diagnose the issue >> appreciated. >> >> >
Re: Jenkins Hung
Hi, you can access the system log via Jenkins Homepage > Manage Jenkins > System log regards didier Le mercredi 7 mars 2012 20:24:16 UTC+1, John Hinnegan a écrit : > > Jenkins hung on me again. It just stops responding to requests. I wish it > would just die so it would be restarted. > > Anyway, following along: > https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+is+hanging > > Here's the JStack: http://pastebin.com/ZSNQgCm7 > Generated with sudo jstack -F 64871 > > Heres's the JMap: http://pastebin.com/Mytcsnpv > Generated with sudo jmap -heap 64871 > > It's OSX. My version of JMap didn't seem to understand what the > -dump:format=b,file=heap.hprof tip in the wiki was going for. Can someone > give me the command line to see how that would work? > > Also, I tried to get the core dump, but it seems on OS X they are > suppressed by default, at least according to: > http://stackoverflow.com/questions/2080918/where-are-core-dumps-written-to-in-mac-os-x > > I didn't know to run the ulimit command first, so it seems I don't have a > dump to share. > > Suggestions on what to do next time to help diagnose the issue appreciated. > > >
Re: Jenkins Hung
it is permgen out of memory, looks like it's similar to other people'e problems
Re: Jenkins Hung
Where is the system log located? On Wednesday, March 7, 2012 3:46:53 PM UTC-8, sti wrote: > > Do you find any exceptions from Jenkins in the system log? > > You should read what others have discussed here. You do not seem to be > alone with your problem. Far from it. > > -- Sami > > John Hinnegan kirjoitti 7.3.2012 kello 21.24: > > Jenkins hung on me again. It just stops responding to requests. I wish it > would just die so it would be restarted. > > Anyway, following along: > https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+is+hanging > > Here's the JStack: http://pastebin.com/ZSNQgCm7 > Generated with sudo jstack -F 64871 > > Heres's the JMap: http://pastebin.com/Mytcsnpv > Generated with sudo jmap -heap 64871 > > It's OSX. My version of JMap didn't seem to understand what the > -dump:format=b,file=heap.hprof tip in the wiki was going for. Can someone > give me the command line to see how that would work? > > Also, I tried to get the core dump, but it seems on OS X they are > suppressed by default, at least according to: > http://stackoverflow.com/questions/2080918/where-are-core-dumps-written-to-in-mac-os-x > > I didn't know to run the ulimit command first, so it seems I don't have a > dump to share. > > Suggestions on what to do next time to help diagnose the issue appreciated. > > >
Re: Jenkins Hung
Do you find any exceptions from Jenkins in the system log? You should read what others have discussed here. You do not seem to be alone with your problem. Far from it. -- Sami John Hinnegan kirjoitti 7.3.2012 kello 21.24: > Jenkins hung on me again. It just stops responding to requests. I wish it > would just die so it would be restarted. > > Anyway, following along: > https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+is+hanging > > Here's the JStack: http://pastebin.com/ZSNQgCm7 > Generated with sudo jstack -F 64871 > > Heres's the JMap: http://pastebin.com/Mytcsnpv > Generated with sudo jmap -heap 64871 > > It's OSX. My version of JMap didn't seem to understand what the > -dump:format=b,file=heap.hprof tip in the wiki was going for. Can someone > give me the command line to see how that would work? > > Also, I tried to get the core dump, but it seems on OS X they are suppressed > by default, at least according to: > http://stackoverflow.com/questions/2080918/where-are-core-dumps-written-to-in-mac-os-x > > I didn't know to run the ulimit command first, so it seems I don't have a > dump to share. > > Suggestions on what to do next time to help diagnose the issue appreciated. > >
Jenkins Hung
Jenkins hung on me again. It just stops responding to requests. I wish it would just die so it would be restarted. Anyway, following along: https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+is+hanging Here's the JStack: http://pastebin.com/ZSNQgCm7 Generated with sudo jstack -F 64871 Heres's the JMap: http://pastebin.com/Mytcsnpv Generated with sudo jmap -heap 64871 It's OSX. My version of JMap didn't seem to understand what the -dump:format=b,file=heap.hprof tip in the wiki was going for. Can someone give me the command line to see how that would work? Also, I tried to get the core dump, but it seems on OS X they are suppressed by default, at least according to: http://stackoverflow.com/questions/2080918/where-are-core-dumps-written-to-in-mac-os-x I didn't know to run the ulimit command first, so it seems I don't have a dump to share. Suggestions on what to do next time to help diagnose the issue appreciated.
