Re: Better Logging for Jenkins Shutdown
Nothing in the system logs? From: Rob Mandeville rmandevi...@dekaresearch.commailto:rmandevi...@dekaresearch.com Reply-To: jenkinsci-users@googlegroups.commailto:jenkinsci-users@googlegroups.com jenkinsci-users@googlegroups.commailto:jenkinsci-users@googlegroups.com Date: Wednesday, May 27, 2015 at 6:09 AM To: jenkinsci-users@googlegroups.commailto:jenkinsci-users@googlegroups.com jenkinsci-users@googlegroups.commailto:jenkinsci-users@googlegroups.com Cc: Rob Mandeville rmandevi...@dekaresearch.commailto:rmandevi...@dekaresearch.com Subject: Better Logging for Jenkins Shutdown My Jenkins server went down last night, and the log showed no signs of trouble. I am running: Jenkins 1.565.1.LTS on its own Winstone server on Java HotSpot 64 bit 1.8.0_25-b17 on Ubuntu 13.03 Saucy Salamander (yes, I know it’s out of date) on a vSphere VM. The log entries ended at 5:20 yesterday afternoon and started again when I ran ‘service jenkins start’ at 8:20 this morning. The relevant log segment is listed below the fold. I see no ‘JVM is terminating. Shutting down winstone’ message, so it doesn’t look like a user did an orderly shutdown. This instance is a memory hog; it takes about 15 minutes to bring up and has over 1,000 projects. It is possible that the JVM ran out of memory or something, but I can’t tell. The worst case scenario (though very unlikely) is that someone ran ‘kill -9’ against the server. In that case, there can’t be a log entry. But barring that, how can I increase the logging for typical shutdown conditions so that I have a better chance to know what happened next time? --Rob INFO: Branch development Project HSM DAL Manager Quick Static Code Analysis #129 main build action completed: SUCCESS May 26, 2015 5:20:10 PM hudson.plugins.logparser.LogParserParser parseLogBody INFO: LogParserParser: Start parsing : Branch development Project HSM DAL Manager Quick Static Code Analysis_build_129 May 26, 2015 5:20:13 PM hudson.plugins.logparser.LogParserParser parseLogBody INFO: LogParserParser: Parsing took 0 minutes (3) seconds. May 26, 2015 5:20:17 PM hudson.model.Run execute INFO: Branch development Run SoapUI Smoke Tests #213 main build action completed: SUCCESS May 26, 2015 5:20:18 PM hudson.model.Run execute INFO: Branch development Smoke Tests #230 main build action completed: UNSTABLE Running from: /usr/share/jenkins/jenkins.war.1.565.1.LTS May 27, 2015 8:22:14 AM winstone.Logger logInternal INFO: Beginning extraction from war file May 27, 2015 8:22:14 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: jetty-8.y.z-SNAPSHOT May 27, 2015 8:22:18 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: NO JSP Support for , did not find org.apache.jasper.servlet.JspServlet Jenkins home directory: /var/lib/jenkins found at: EnvVars.masterEnvVars.get(JENKINS_HOME) May 27, 2015 8:22:19 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: Started SelectChannelConnector@0.0.0.0mailto:SelectChannelConnector@0.0.0.0:8080 May 27, 2015 8:22:19 AM winstone.Logger logInternal INFO: Winstone Servlet Engine v2.0 running: controlPort=disabled May 27, 2015 8:22:21 AM jenkins.InitReactorRunner$1 onAttained INFO: Started initialization May 27, 2015 8:22:23 AM hudson.ClassicPluginStrategy createPluginWrapper INFO: Plugin subversion.jpi is disabled May 27, 2015 8:22:24 AM hudson.ClassicPluginStrategy createPluginWrapper INFO: Plugin cvs.jpi is disabled May 27, 2015 8:22:25 AM jenkins.InitReactorRunner$1 onAttained INFO: Listed all plugins This e-mail and the information, including any attachments it contains, are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message. Thank you. Please consider the environment before printing this email. -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.commailto:jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0A40042D85E7C84DB443060EC44B3FD36D9BCC181A%40dekaexchange07.deka.localhttps://groups.google.com/d/msgid/jenkinsci-users/0A40042D85E7C84DB443060EC44B3FD36D9BCC181A%40dekaexchange07.deka.local?utm_medium=emailutm_source=footer. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to
RE: Better Logging for Jenkins Shutdown
Thanks! It got hit by the OOM killer. I should have thought of that first. --Rob -Original Message- From: jenkinsci-users@googlegroups.com [mailto:jenkinsci-users@googlegroups.com] On Behalf Of Daniel Beck Sent: Wednesday, May 27, 2015 12:40 PM To: jenkinsci-users@googlegroups.com Subject: Re: Better Logging for Jenkins Shutdown Check the system logs. Too large of an Xmx may have Linux kill off your process (oom-killer). On 27.05.2015, at 15:09, Rob Mandeville rmandevi...@dekaresearch.com wrote: My Jenkins server went down last night, and the log showed no signs of trouble. I am running: Jenkins 1.565.1.LTS on its own Winstone server on Java HotSpot 64 bit 1.8.0_25-b17 on Ubuntu 13.03 Saucy Salamander (yes, I know it’s out of date) on a vSphere VM. The log entries ended at 5:20 yesterday afternoon and started again when I ran ‘service jenkins start’ at 8:20 this morning. The relevant log segment is listed below the fold. I see no ‘JVM is terminating. Shutting down winstone’ message, so it doesn’t look like a user did an orderly shutdown. This instance is a memory hog; it takes about 15 minutes to bring up and has over 1,000 projects. It is possible that the JVM ran out of memory or something, but I can’t tell. The worst case scenario (though very unlikely) is that someone ran ‘kill -9’ against the server. In that case, there can’t be a log entry. But barring that, how can I increase the logging for typical shutdown conditions so that I have a better chance to know what happened next time? --Rob INFO: Branch development Project HSM DAL Manager Quick Static Code Analysis #129 main build action completed: SUCCESS May 26, 2015 5:20:10 PM hudson.plugins.logparser.LogParserParser parseLogBody INFO: LogParserParser: Start parsing : Branch development Project HSM DAL Manager Quick Static Code Analysis_build_129 May 26, 2015 5:20:13 PM hudson.plugins.logparser.LogParserParser parseLogBody INFO: LogParserParser: Parsing took 0 minutes (3) seconds. May 26, 2015 5:20:17 PM hudson.model.Run execute INFO: Branch development Run SoapUI Smoke Tests #213 main build action completed: SUCCESS May 26, 2015 5:20:18 PM hudson.model.Run execute INFO: Branch development Smoke Tests #230 main build action completed: UNSTABLE Running from: /usr/share/jenkins/jenkins.war.1.565.1.LTS May 27, 2015 8:22:14 AM winstone.Logger logInternal INFO: Beginning extraction from war file May 27, 2015 8:22:14 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: jetty-8.y.z-SNAPSHOT May 27, 2015 8:22:18 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: NO JSP Support for , did not find org.apache.jasper.servlet.JspServlet Jenkins home directory: /var/lib/jenkins found at: EnvVars.masterEnvVars.get(JENKINS_HOME) May 27, 2015 8:22:19 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: Started SelectChannelConnector@0.0.0.0:8080 May 27, 2015 8:22:19 AM winstone.Logger logInternal INFO: Winstone Servlet Engine v2.0 running: controlPort=disabled May 27, 2015 8:22:21 AM jenkins.InitReactorRunner$1 onAttained INFO: Started initialization May 27, 2015 8:22:23 AM hudson.ClassicPluginStrategy createPluginWrapper INFO: Plugin subversion.jpi is disabled May 27, 2015 8:22:24 AM hudson.ClassicPluginStrategy createPluginWrapper INFO: Plugin cvs.jpi is disabled May 27, 2015 8:22:25 AM jenkins.InitReactorRunner$1 onAttained INFO: Listed all plugins This e-mail and the information, including any attachments it contains, are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message. Thank you. Please consider the environment before printing this email. -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0A40042D85E7C84DB443060EC44B3FD36D9BCC181A%40dekaexchange07.deka.local. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/20B968A5-8400-40A7-8E24-2B82B47C4D41%40beckweb.net. For more options, visit https://groups.google.com/d/optout. Click https://www.mailcontrol.com/sr
Re: Better Logging for Jenkins Shutdown
Check the system logs. Too large of an Xmx may have Linux kill off your process (oom-killer). On 27.05.2015, at 15:09, Rob Mandeville rmandevi...@dekaresearch.com wrote: My Jenkins server went down last night, and the log showed no signs of trouble. I am running: Jenkins 1.565.1.LTS on its own Winstone server on Java HotSpot 64 bit 1.8.0_25-b17 on Ubuntu 13.03 Saucy Salamander (yes, I know it’s out of date) on a vSphere VM. The log entries ended at 5:20 yesterday afternoon and started again when I ran ‘service jenkins start’ at 8:20 this morning. The relevant log segment is listed below the fold. I see no ‘JVM is terminating. Shutting down winstone’ message, so it doesn’t look like a user did an orderly shutdown. This instance is a memory hog; it takes about 15 minutes to bring up and has over 1,000 projects. It is possible that the JVM ran out of memory or something, but I can’t tell. The worst case scenario (though very unlikely) is that someone ran ‘kill -9’ against the server. In that case, there can’t be a log entry. But barring that, how can I increase the logging for typical shutdown conditions so that I have a better chance to know what happened next time? --Rob INFO: Branch development Project HSM DAL Manager Quick Static Code Analysis #129 main build action completed: SUCCESS May 26, 2015 5:20:10 PM hudson.plugins.logparser.LogParserParser parseLogBody INFO: LogParserParser: Start parsing : Branch development Project HSM DAL Manager Quick Static Code Analysis_build_129 May 26, 2015 5:20:13 PM hudson.plugins.logparser.LogParserParser parseLogBody INFO: LogParserParser: Parsing took 0 minutes (3) seconds. May 26, 2015 5:20:17 PM hudson.model.Run execute INFO: Branch development Run SoapUI Smoke Tests #213 main build action completed: SUCCESS May 26, 2015 5:20:18 PM hudson.model.Run execute INFO: Branch development Smoke Tests #230 main build action completed: UNSTABLE Running from: /usr/share/jenkins/jenkins.war.1.565.1.LTS May 27, 2015 8:22:14 AM winstone.Logger logInternal INFO: Beginning extraction from war file May 27, 2015 8:22:14 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: jetty-8.y.z-SNAPSHOT May 27, 2015 8:22:18 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: NO JSP Support for , did not find org.apache.jasper.servlet.JspServlet Jenkins home directory: /var/lib/jenkins found at: EnvVars.masterEnvVars.get(JENKINS_HOME) May 27, 2015 8:22:19 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: Started SelectChannelConnector@0.0.0.0:8080 May 27, 2015 8:22:19 AM winstone.Logger logInternal INFO: Winstone Servlet Engine v2.0 running: controlPort=disabled May 27, 2015 8:22:21 AM jenkins.InitReactorRunner$1 onAttained INFO: Started initialization May 27, 2015 8:22:23 AM hudson.ClassicPluginStrategy createPluginWrapper INFO: Plugin subversion.jpi is disabled May 27, 2015 8:22:24 AM hudson.ClassicPluginStrategy createPluginWrapper INFO: Plugin cvs.jpi is disabled May 27, 2015 8:22:25 AM jenkins.InitReactorRunner$1 onAttained INFO: Listed all plugins This e-mail and the information, including any attachments it contains, are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message. Thank you. Please consider the environment before printing this email. -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0A40042D85E7C84DB443060EC44B3FD36D9BCC181A%40dekaexchange07.deka.local. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/20B968A5-8400-40A7-8E24-2B82B47C4D41%40beckweb.net. For more options, visit https://groups.google.com/d/optout.
