![]() |
|
|
|
|
|
|
Change By:
|
Stefan Verhoeff
(18/Aug/14 4:43 PM)
|
|
Affects Version/s:
|
current
|
|
Description:
|
We experienced a crashing Jenkins server after Walldisplay started to report "timeout". Requests to the jobs view API URL at /view/view-name/api/json where timing out. CPU was a 100% and there were many hanging RequestHandler threads. In the logs we found IOExceptions on a job that contains whitespace which lead to a StackOverflowError.
After renaming the job to not contain whitespace the issue was solved.
Detailed
Stacktrace
Stack trace
from jenkins.log:
{code}
Aug 18, 2014 10:21:34 AM hudson.model.RunMap retrieve
WARNING: could not load /var/lib/jenkins/jobs/Deploy to dev and run QG/builds/2014-07-22_02-59-43
java.io.IOException: Unable to read /var/lib/jenkins/jobs/Deploy to dev and run QG/builds/2014-07-22_02
-59-43/build.xml
at hudson.XmlFile.unmarshal(XmlFile.java:167)
at hudson.model.Run.reload(Run.java:323)
at hudson.model.Run.<init>(Run.java:311)
at hudson.model.AbstractBuild.<init>(AbstractBuild.java:177)
at hudson.model.Build.<init>(Build.java:103)
at hudson.model.FreeStyleBuild.<init>(FreeStyleBuild.java:38)
at sun.reflect.GeneratedConstructorAccessor94.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.
java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:155)
at jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:136)
at hudson.model.RunMap.retrieve(RunMap.java:218)
at hudson.model.RunMap.retrieve(RunMap.java:56)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:688)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:671)
at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:470)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:547)
at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:235)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:956)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:144)
at hudson.model.Cause$UpstreamCause.onLoad(Cause.java:189)
at hudson.model.CauseAction.onLoad(CauseAction.java:124)
at hudson.model.Run.onLoad(Run.java:342)
at hudson.model.RunMap.retrieve(RunMap.java:219)
at hudson.model.RunMap.retrieve(RunMap.java:56)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:688)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:671)
at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:470)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:547)
at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:235)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:956)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:144)
at hudson.model.Cause$UpstreamCause.onLoad(Cause.java:189)
at hudson.model.CauseAction.onLoad(CauseAction.java:124)
at hudson.model.Run.onLoad(Run.java:342)
at hudson.model.RunMap.retrieve(RunMap.java:219)
at hudson.model.RunMap.retrieve(RunMap.java:56)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:688)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:671)
at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:470)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:547)
at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:235)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:956)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:144)
at hudson.model.Cause$UpstreamCause.onLoad(Cause.java:189)
at hudson.model.CauseAction.onLoad(CauseAction.java:124)
at hudson.model.Run.onLoad(Run.java:342)
at hudson.model.RunMap.retrieve(RunMap.java:219)
at hudson.model.RunMap.retrieve(RunMap.java:56)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:688)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:671)
at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:470)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:547)
at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:235)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:956)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:144)
at hudson.model.Cause$UpstreamCause.onLoad(Cause.java:189)
at hudson.model.CauseAction.onLoad(CauseAction.java:124)
at hudson.model.Run.onLoad(Run.java:342)
at hudson.model.RunMap.retrieve(RunMap.java:219)
at hudson.model.RunMap.retrieve(RunMap.java:56)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:688)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:671)
at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:470)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:547)
at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:235)
...
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:688)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:671)
at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:470)
Caused by: java.lang.StackOverflowError
at com.thoughtworks.xstream.mapper.MapperWrapper.aliasForAttribute(MapperWrapper.java:50)
at com.thoughtworks.xstream.mapper.MapperWrapper.aliasForAttribute(MapperWrapper.java:50)
at com.thoughtworks.xstream.mapper.MapperWrapper.aliasForAttribute(MapperWrapper.java:50)
at com.thoughtworks.xstream.mapper.MapperWrapper.aliasForAttribute(MapperWrapper.java:50)
at com.thoughtworks.xstream.mapper.AttributeAliasingMapper.aliasForAttribute(AttributeAliasingM
apper.java:31)
at com.thoughtworks.xstream.mapper.MapperWrapper.aliasForAttribute(MapperWrapper.java:50)
at com.thoughtworks.xstream.mapper.SystemAttributeAliasingMapper.aliasForSystemAttribute(System
AttributeAliasingMapper.java:31)
at com.thoughtworks.xstream.mapper.MapperWrapper.aliasForSystemAttribute(MapperWrapper.java:58)
at com.thoughtworks.xstream.mapper.MapperWrapper.aliasForSystemAttribute(MapperWrapper.java:58)
at com.thoughtworks.xstream.mapper.MapperWrapper.aliasForSystemAttribute(MapperWrapper.java:58)
at com.thoughtworks.xstream.mapper.MapperWrapper.aliasForSystemAttribute(MapperWrapper.java:58)
at com.thoughtworks.xstream.mapper.MapperWrapper.aliasForSystemAttribute(MapperWrapper.java:58)
at com.thoughtworks.xstream.mapper.MapperWrapper.aliasForSystemAttribute(MapperWrapper.java:58)
at com.thoughtworks.xstream.mapper.MapperWrapper.aliasForSystemAttribute(MapperWrapper.java:58)
at com.thoughtworks.xstream.mapper.MapperWrapper.aliasForSystemAttribute(MapperWrapper.java:58)
at com.thoughtworks.xstream.mapper.MapperWrapper.aliasForSystemAttribute(MapperWrapper.java:58)
at com.thoughtworks.xstream.mapper.MapperWrapper.aliasForSystemAttribute(MapperWrapper.java:58)
at com.thoughtworks.xstream.mapper.MapperWrapper.aliasForSystemAttribute(MapperWrapper.java:58)
at com.thoughtworks.xstream.mapper.MapperWrapper.aliasForSystemAttribute(MapperWrapper.java:58)
at com.thoughtworks.xstream.mapper.MapperWrapper.aliasForSystemAttribute(MapperWrapper.java:58)
at hudson.util.xstream.MapperDelegate.aliasForSystemAttribute(MapperDelegate.java:71)
at com.thoughtworks.xstream.mapper.MapperWrapper.aliasForSystemAttribute(MapperWrapper.java:58)
at com.thoughtworks.xstream.core.util.HierarchicalStreams.readClassAttribute(HierarchicalStream
...
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:144)
at hudson.model.Cause$UpstreamCause.onLoad(Cause.java:189)
at hudson.model.CauseAction.onLoad(CauseAction.java:124)
at hudson.model.Run.onLoad(Run.java:342)
at hudson.model.RunMap.retrieve(RunMap.java:219)
at hudson.model.RunMap.retrieve(RunMap.java:56)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:688)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:671)
at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:470)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:547)
at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:235)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:956)
{code}
{code}
Aug 18, 2014 10:21:34 AM org.eclipse.jetty.util.log.JavaUtilLog warn
WARNING: Error while serving http://jenkins.webmaps.devbln.europe.nokia.com//view/HereWeb/api/json
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:210)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:631)
at org.kohsuke.stapler.Stapler.service(Stapler.java:225)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:202)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:180)
at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:85)
at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:89)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:79)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to write firstBuild
at org.kohsuke.stapler.export.Property.writeTo(Property.java:120)
at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187)
at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182)
at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182)
at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182)
at org.kohsuke.stapler.export.Property.writeValue(Property.java:236)
at org.kohsuke.stapler.export.Property.writeValue(Property.java:185)
at org.kohsuke.stapler.export.Property.writeValue(Property.java:137)
at org.kohsuke.stapler.export.Property.writeTo(Property.java:114)
at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187)
at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182)
at org.kohsuke.stapler.export.Model.writeTo(Model.java:154)
at org.kohsuke.stapler.ResponseImpl.serveExposedBean(ResponseImpl.java:267)
at hudson.model.Api.doJson(Api.java:204)
... 83 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor203.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.kohsuke.stapler.export.MethodProperty.getValue(MethodProperty.java:66)
at org.kohsuke.stapler.export.Property.writeTo(Property.java:114)
... 96 more
Caused by: java.lang.StackOverflowError
at java.io.UnixFileSystem.canonicalize0(Native Method)
at java.io.UnixFileSystem.canonicalize(UnixFileSystem.java:172)
at java.io.File.getCanonicalPath(File.java:618)
at java.io.File.getCanonicalFile(File.java:643)
at hudson.model.Run.parseTimestampFromBuildDir(Run.java:406)
at hudson.model.Run.<init>(Run.java:309)
at hudson.model.AbstractBuild.<init>(AbstractBuild.java:177)
at hudson.model.Build.<init>(Build.java:103)
at hudson.model.FreeStyleBuild.<init>(FreeStyleBuild.java:38)
at sun.reflect.GeneratedConstructorAccessor94.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.
java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:155)
at jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:136)
at hudson.model.RunMap.retrieve(RunMap.java:218)
at hudson.model.RunMap.retrieve(RunMap.java:56)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at com.chikli.hudson.plugin.naginator.NaginatorActionFactory.createFor(NaginatorActionFactory.java:20)
at hudson.model.Run.getTransientActions(Run.java:362)
... 129 more
{code}
|
|
|
|
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
|
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit
https://groups.google.com/d/optout.
Jenkins version is the latest: Jenkins ver. 1.575.
Browser is Chrome 36, although this looks like a backend issue.
I'm not convinced the issue is caused by Naginator. There are two exceptions in the trace I provided. I split them to make that more clear. The first is says "java.io.IOException: Unable to read" that seems to be caused by the directory named "Deploy to dev and run QG" contains whitespace.
The second exception is where Naginator shows up, but that could be just to retry the failed build. The exception that triggered it is "java.io.IOException: Failed to write firstBuild".
The NPE fix for Naginator is still a good one, but i'm not sure it fixed the root cause.