[JIRA] [core] (JENKINS-25742) Concurrent "job rename" and "edit view" results in deadlock that puts Jenkins in a non responsive state

Fri, 21 Nov 2014 20:31:15 -0800

Issue Type: Bug Bug
Assignee: Unassigned
Components: core
Created: 22/Nov/14 4:29 AM
Description:

When there are multiple users using Jenkins:

One user has clicked on "Yes" confirmation to rename a job.
Another user has clicked on "OK" on the "Edit View" screen at the same time.

Results in a race condition that results in a deadlock. Immediately after this deadlock occurs, Jenkins "create new job" and other operations are unresponsive. And after some more time Jenkins goes down.

Name: Handling GET /jenkins/view/ABCDEF/api/json : qtp817406040-243035
State: BLOCKED on hudson.model.ListView@57affa54 owned by: Handling POST /jenkins/view/ABCDEF/configSubmit : qtp817406040-42
Total blocked: 9 Total waited: 189

Stack trace:
hudson.model.ListView.getItems(ListView.java:164)
hudson.model.ListView.getItems(ListView.java:66)
sun.reflect.GeneratedMethodAccessor1929.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:483)
org.kohsuke.stapler.export.MethodProperty.getValue(MethodProperty.java:66)
org.kohsuke.stapler.export.Property.writeTo(Property.java:116)
org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:190)
org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:185)
org.kohsuke.stapler.export.Model.writeTo(Model.java:157)
org.kohsuke.stapler.ResponseImpl.serveExposedBean(ResponseImpl.java:267)
hudson.model.Api.doJson(Api.java:204)
sun.reflect.GeneratedMethodAccessor771.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:483)
org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:210)
org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:636)
org.kohsuke.stapler.Stapler.service(Stapler.java:225)
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:91)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
hudson.security.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:114)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)

Name: Handling POST /jenkins/view/ABCDEF/configSubmit : qtp817406040-42
State: BLOCKED on hudson.model.Hudson@715617bb owned by: Handling POST /jenkins/view/XYZXYZ/job/turkeytimeXYZ/doRename : qtp817406040-109
Total blocked: 2,007 Total waited: 12,613

Stack trace:
jenkins.model.Jenkins.save(Jenkins.java:2675)
hudson.model.View.save(View.java:341)
hudson.util.PersistedList.onModified(PersistedList.java:173)
hudson.util.PersistedList.replaceBy(PersistedList.java:85)
hudson.util.DescribableList.rebuildHetero(DescribableList.java:203)
hudson.model.ListView.submit(ListView.java:348)
hudson.model.View.doConfigSubmit(View.java:956)

  • locked hudson.model.ListView@57affa54
    sun.reflect.GeneratedMethodAccessor1966.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:483)
    org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
    org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46)
    org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399)
    org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
    org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
    org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
    org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
    org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
    org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:636)
    org.kohsuke.stapler.Stapler.service(Stapler.java:225)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
    hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
    hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66)
    hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
    hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:85)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    hudson.security.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:114)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)

Name: Handling POST /jenkins/view/XYZXYZ/job/turkeytimeXYZ/doRename : qtp817406040-109
State: BLOCKED on hudson.model.ListView@7e01a6a2 owned by: Handling POST /jenkins/view/XYZXYZ/configSubmit : qtp817406040-117
Total blocked: 54,155 Total waited: 11,725

Stack trace:
hudson.model.ListView$Listener.renameViewItem(ListView.java:421)
hudson.model.ListView$Listener.onLocationChanged(ListView.java:404)
hudson.model.listeners.ItemListener.fireLocationChange(ItemListener.java:199)
hudson.model.AbstractItem.renameTo(AbstractItem.java:317)

  • locked hudson.model.FreeStyleProject@5dafef50
  • locked hudson.model.Hudson@715617bb
    hudson.model.Job.renameTo(Job.java:617)
    hudson.model.Job.doDoRename(Job.java:1403)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:483)
    org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
    org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46)
    org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399)
    org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
    org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
    org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
    org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
    org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
    org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
    org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
    org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:636)
    org.kohsuke.stapler.Stapler.service(Stapler.java:225)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
    hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
    hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66)
    hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
    hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:85)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    hudson.security.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:114)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)

Name: Handling POST /jenkins/view/XYZXYZ/configSubmit : qtp817406040-117
State: BLOCKED on hudson.model.Hudson@715617bb owned by: Handling POST /jenkins/view/XYZXYZ/job/turkeytimeXYZ/doRename : qtp817406040-109
Total blocked: 2,045 Total waited: 12,350

Stack trace:
jenkins.model.Jenkins.save(Jenkins.java:2675)
hudson.model.View.save(View.java:341)
hudson.model.View.doConfigSubmit(View.java:967)

  • locked hudson.model.ListView@7e01a6a2
    sun.reflect.GeneratedMethodAccessor1966.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:483)
    org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
    org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46)
    org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399)
    org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
    org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
    org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
    org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
    org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
    org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:636)
    org.kohsuke.stapler.Stapler.service(Stapler.java:225)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
    hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
    hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66)
    hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
    hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:85)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    hudson.security.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:114)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
Environment: Summary: HP DL160 G6, 2 x Xeon E5645 2.40GHz, 93.8GB / 96GB 1333MHz DDR3
System: HP ProLiant DL160 G6, ySPEC 34.5
Processors: 2 x Xeon E5645 2.40GHz (HT enabled, 12 cores, 24 threads) - Gulftown B1, 64-bit, six-core, 32nm, L3: 12MB
Memory: 93.8GB / 96GB 1333MHz DDR3 == 6 x 8GB - 8GB PC3-10600 Samsung DDR3-1333 ECC Registered CL9 2Rx4
6 x 8GB - 8GB PC3-10600 Hynix DDR3-1333 ECC Registered CL9 2Rx4
6 x empty
Disk: sda (hpsa0): 1.2TB (0%) RAID-10 == 1 x 1.2TB HP-LOGICAL-VOLUME
Disk-Control: ahci0: HP/Intel ICH10 82801J 6 Port SATA/300 AHCI
Disk-Control: hpsa0: HP Smart Array P410
Chipset: Intel 5520 IOH-36D B3 (Tylersburg), 82801JIR A0 (ICH10R)
Network: eth0 (igb): HP NC362i/Intel 82576 Gigabit, 1Gb/s <full-duplex>
Network: eth1 (igb): HP NC362i/Intel 82576 Gigabit, no carrier
OS: RHEL Server 6.4, Linux 2.6.32-358, 64-bit
BIOS: HP O33 08/16/2010, rev 8.15
Hostname: xxx


Jenkins ver. 1.565.3
JDK 8u25 x.64
Jetty-9.2.2.v20140723


Project: Jenkins
Labels: jenkins
Priority: Critical Critical
Reporter: Manav Chimakurthi
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to