RE: Better Logging for Jenkins Shutdown

2015-05-27 Thread Rob Mandeville 
Thanks!  It got hit by the OOM killer.  I should have thought of that first.

--Rob

-Original Message-
From: jenkinsci-users@googlegroups.com 
[mailto:jenkinsci-users@googlegroups.com] On Behalf Of Daniel Beck
Sent: Wednesday, May 27, 2015 12:40 PM
To: jenkinsci-users@googlegroups.com
Subject: Re: Better Logging for Jenkins Shutdown

Check the system logs. Too large of an Xmx may have Linux kill off your process 
(oom-killer).

On 27.05.2015, at 15:09, Rob Mandeville  wrote:

> My Jenkins server went down last night, and the log showed no signs of 
> trouble.  I am running:
>
> Jenkins 1.565.1.LTS on
> its own Winstone server on
> Java HotSpot 64 bit 1.8.0_25-b17 on
> Ubuntu 13.03 Saucy Salamander (yes, I know it’s out of date) on a vSphere VM.
>
> The log entries ended at 5:20 yesterday afternoon and started again when I 
> ran ‘service jenkins start’ at 8:20 this morning.  The relevant log segment 
> is listed below the fold.
>
> I see no ‘JVM is terminating.  Shutting down winstone’ message, so it doesn’t 
> look like a user did an orderly shutdown.
>
> This instance is a memory hog; it takes about 15 minutes to bring up and has 
> over 1,000 projects.  It is possible that the JVM ran out of memory or 
> something, but I can’t tell.
>
> The worst case scenario (though very unlikely) is that someone ran ‘kill -9’ 
> against the server.  In that case, there can’t be a log entry.  But barring 
> that, how can I increase the logging for typical shutdown conditions so that 
> I have a better chance to know what happened next time?
>
> --Rob
>
>
>
> INFO: Branch development Project HSM DAL Manager Quick Static Code
> Analysis #129 main build action completed: SUCCESS May 26, 2015
> 5:20:10 PM hudson.plugins.logparser.LogParserParser parseLogBody
> INFO: LogParserParser: Start parsing : Branch development Project HSM
> DAL Manager Quick Static Code Analysis_build_129 May 26, 2015 5:20:13
> PM hudson.plugins.logparser.LogParserParser parseLogBody
> INFO: LogParserParser: Parsing took 0 minutes (3) seconds.
> May 26, 2015 5:20:17 PM hudson.model.Run execute
> INFO: Branch development Run SoapUI Smoke Tests #213 main build action
> completed: SUCCESS May 26, 2015 5:20:18 PM hudson.model.Run execute
> INFO: Branch development Smoke Tests #230 main build action completed:
> UNSTABLE Running from: /usr/share/jenkins/jenkins.war.1.565.1.LTS
> May 27, 2015 8:22:14 AM winstone.Logger logInternal
> INFO: Beginning extraction from war file May 27, 2015 8:22:14 AM
> org.eclipse.jetty.util.log.JavaUtilLog info
> INFO: jetty-8.y.z-SNAPSHOT
> May 27, 2015 8:22:18 AM org.eclipse.jetty.util.log.JavaUtilLog info
> INFO: NO JSP Support for , did not find
> org.apache.jasper.servlet.JspServlet
> Jenkins home directory: /var/lib/jenkins found at:
> EnvVars.masterEnvVars.get("JENKINS_HOME")
> May 27, 2015 8:22:19 AM org.eclipse.jetty.util.log.JavaUtilLog info
> INFO: Started SelectChannelConnector@0.0.0.0:8080
> May 27, 2015 8:22:19 AM winstone.Logger logInternal
> INFO: Winstone Servlet Engine v2.0 running: controlPort=disabled May
> 27, 2015 8:22:21 AM jenkins.InitReactorRunner$1 onAttained
> INFO: Started initialization
> May 27, 2015 8:22:23 AM hudson.ClassicPluginStrategy
> createPluginWrapper
> INFO: Plugin subversion.jpi is disabled May 27, 2015 8:22:24 AM
> hudson.ClassicPluginStrategy createPluginWrapper
> INFO: Plugin cvs.jpi is disabled
> May 27, 2015 8:22:25 AM jenkins.InitReactorRunner$1 onAttained
> INFO: Listed all plugins
>
>
>
> This e-mail and the information, including any attachments it contains, are 
> intended to be a confidential communication only to the person or entity to 
> whom it is addressed and may contain information that is privileged. If the 
> reader of this message is not the intended recipient, you are hereby notified 
> that any dissemination, distribution or copying of this communication is 
> strictly prohibited. If you have received this communication in error, please 
> immediately notify the sender and destroy the original message.
>
> Thank you.
>
> Please consider the environment before printing this email.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkinsci-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/jenkinsci-users/0A40042D85E7C84DB443060EC44B3FD36D9BCC181A%40dekaexchange07.deka.local.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe fro

Re: Better Logging for Jenkins Shutdown

2015-05-27 Thread Daniel Beck 
Check the system logs. Too large of an Xmx may have Linux kill off your process 
(oom-killer).

On 27.05.2015, at 15:09, Rob Mandeville  wrote:

> My Jenkins server went down last night, and the log showed no signs of 
> trouble.  I am running:
>  
> Jenkins 1.565.1.LTS on
> its own Winstone server on
> Java HotSpot 64 bit 1.8.0_25-b17 on
> Ubuntu 13.03 Saucy Salamander (yes, I know it’s out of date) on a vSphere VM.
>  
> The log entries ended at 5:20 yesterday afternoon and started again when I 
> ran ‘service jenkins start’ at 8:20 this morning.  The relevant log segment 
> is listed below the fold.
>  
> I see no ‘JVM is terminating.  Shutting down winstone’ message, so it doesn’t 
> look like a user did an orderly shutdown.
>  
> This instance is a memory hog; it takes about 15 minutes to bring up and has 
> over 1,000 projects.  It is possible that the JVM ran out of memory or 
> something, but I can’t tell.
>  
> The worst case scenario (though very unlikely) is that someone ran ‘kill -9’ 
> against the server.  In that case, there can’t be a log entry.  But barring 
> that, how can I increase the logging for typical shutdown conditions so that 
> I have a better chance to know what happened next time?
>  
> --Rob
>  
>  
>  
> INFO: Branch development Project HSM DAL Manager Quick Static Code Analysis 
> #129 main build action completed: SUCCESS
> May 26, 2015 5:20:10 PM hudson.plugins.logparser.LogParserParser parseLogBody
> INFO: LogParserParser: Start parsing : Branch development Project HSM DAL 
> Manager Quick Static Code Analysis_build_129
> May 26, 2015 5:20:13 PM hudson.plugins.logparser.LogParserParser parseLogBody
> INFO: LogParserParser: Parsing took 0 minutes (3) seconds.
> May 26, 2015 5:20:17 PM hudson.model.Run execute
> INFO: Branch development Run SoapUI Smoke Tests #213 main build action 
> completed: SUCCESS
> May 26, 2015 5:20:18 PM hudson.model.Run execute
> INFO: Branch development Smoke Tests #230 main build action completed: 
> UNSTABLE
> Running from: /usr/share/jenkins/jenkins.war.1.565.1.LTS
> May 27, 2015 8:22:14 AM winstone.Logger logInternal
> INFO: Beginning extraction from war file
> May 27, 2015 8:22:14 AM org.eclipse.jetty.util.log.JavaUtilLog info
> INFO: jetty-8.y.z-SNAPSHOT
> May 27, 2015 8:22:18 AM org.eclipse.jetty.util.log.JavaUtilLog info
> INFO: NO JSP Support for , did not find org.apache.jasper.servlet.JspServlet
> Jenkins home directory: /var/lib/jenkins found at: 
> EnvVars.masterEnvVars.get("JENKINS_HOME")
> May 27, 2015 8:22:19 AM org.eclipse.jetty.util.log.JavaUtilLog info
> INFO: Started SelectChannelConnector@0.0.0.0:8080
> May 27, 2015 8:22:19 AM winstone.Logger logInternal
> INFO: Winstone Servlet Engine v2.0 running: controlPort=disabled
> May 27, 2015 8:22:21 AM jenkins.InitReactorRunner$1 onAttained
> INFO: Started initialization
> May 27, 2015 8:22:23 AM hudson.ClassicPluginStrategy createPluginWrapper
> INFO: Plugin subversion.jpi is disabled
> May 27, 2015 8:22:24 AM hudson.ClassicPluginStrategy createPluginWrapper
> INFO: Plugin cvs.jpi is disabled
> May 27, 2015 8:22:25 AM jenkins.InitReactorRunner$1 onAttained
> INFO: Listed all plugins
>  
>  
> 
> This e-mail and the information, including any attachments it contains, are 
> intended to be a confidential communication only to the person or entity to 
> whom it is addressed and may contain information that is privileged. If the 
> reader of this message is not the intended recipient, you are hereby notified 
> that any dissemination, distribution or copying of this communication is 
> strictly prohibited. If you have received this communication in error, please 
> immediately notify the sender and destroy the original message.
> 
> Thank you.
> 
> Please consider the environment before printing this email.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkinsci-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/jenkinsci-users/0A40042D85E7C84DB443060EC44B3FD36D9BCC181A%40dekaexchange07.deka.local.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/20B968A5-8400-40A7-8E24-2B82B47C4D41%40beckweb.net.
For more options, visit https://groups.google.com/d/optout.

Re: Better Logging for Jenkins Shutdown

2015-05-27 Thread Suckow, Thomas J 
Nothing in the system logs?

From: Rob Mandeville 
mailto:rmandevi...@dekaresearch.com>>
Reply-To: 
"jenkinsci-users@googlegroups.com" 
mailto:jenkinsci-users@googlegroups.com>>
Date: Wednesday, May 27, 2015 at 6:09 AM
To: "jenkinsci-users@googlegroups.com" 
mailto:jenkinsci-users@googlegroups.com>>
Cc: Rob Mandeville 
mailto:rmandevi...@dekaresearch.com>>
Subject: Better Logging for Jenkins Shutdown

My Jenkins server went down last night, and the log showed no signs of trouble. 
 I am running:

Jenkins 1.565.1.LTS on
its own Winstone server on
Java HotSpot 64 bit 1.8.0_25-b17 on
Ubuntu 13.03 Saucy Salamander (yes, I know it’s out of date) on a vSphere VM.

The log entries ended at 5:20 yesterday afternoon and started again when I ran 
‘service jenkins start’ at 8:20 this morning.  The relevant log segment is 
listed below the fold.

I see no ‘JVM is terminating.  Shutting down winstone’ message, so it doesn’t 
look like a user did an orderly shutdown.

This instance is a memory hog; it takes about 15 minutes to bring up and has 
over 1,000 projects.  It is possible that the JVM ran out of memory or 
something, but I can’t tell.

The worst case scenario (though very unlikely) is that someone ran ‘kill -9’ 
against the server.  In that case, there can’t be a log entry.  But barring 
that, how can I increase the logging for typical shutdown conditions so that I 
have a better chance to know what happened next time?

--Rob



INFO: Branch development Project HSM DAL Manager Quick Static Code Analysis 
#129 main build action completed: SUCCESS
May 26, 2015 5:20:10 PM hudson.plugins.logparser.LogParserParser parseLogBody
INFO: LogParserParser: Start parsing : Branch development Project HSM DAL 
Manager Quick Static Code Analysis_build_129
May 26, 2015 5:20:13 PM hudson.plugins.logparser.LogParserParser parseLogBody
INFO: LogParserParser: Parsing took 0 minutes (3) seconds.
May 26, 2015 5:20:17 PM hudson.model.Run execute
INFO: Branch development Run SoapUI Smoke Tests #213 main build action 
completed: SUCCESS
May 26, 2015 5:20:18 PM hudson.model.Run execute
INFO: Branch development Smoke Tests #230 main build action completed: UNSTABLE
Running from: /usr/share/jenkins/jenkins.war.1.565.1.LTS
May 27, 2015 8:22:14 AM winstone.Logger logInternal
INFO: Beginning extraction from war file
May 27, 2015 8:22:14 AM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: jetty-8.y.z-SNAPSHOT
May 27, 2015 8:22:18 AM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: NO JSP Support for , did not find org.apache.jasper.servlet.JspServlet
Jenkins home directory: /var/lib/jenkins found at: 
EnvVars.masterEnvVars.get("JENKINS_HOME")
May 27, 2015 8:22:19 AM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: Started 
SelectChannelConnector@0.0.0.0:8080
May 27, 2015 8:22:19 AM winstone.Logger logInternal
INFO: Winstone Servlet Engine v2.0 running: controlPort=disabled
May 27, 2015 8:22:21 AM jenkins.InitReactorRunner$1 onAttained
INFO: Started initialization
May 27, 2015 8:22:23 AM hudson.ClassicPluginStrategy createPluginWrapper
INFO: Plugin subversion.jpi is disabled
May 27, 2015 8:22:24 AM hudson.ClassicPluginStrategy createPluginWrapper
INFO: Plugin cvs.jpi is disabled
May 27, 2015 8:22:25 AM jenkins.InitReactorRunner$1 onAttained
INFO: Listed all plugins




This e-mail and the information, including any attachments it contains, are 
intended to be a confidential communication only to the person or entity to 
whom it is addressed and may contain information that is privileged. If the 
reader of this message is not the intended recipient, you are hereby notified 
that any dissemination, distribution or copying of this communication is 
strictly prohibited. If you have received this communication in error, please 
immediately notify the sender and destroy the original message.

Thank you.

Please consider the environment before printing this email.

--
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/0A40042D85E7C84DB443060EC44B3FD36D9BCC181A%40dekaexchange07.deka.local.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://gr