[JIRA] [support-core-plugin] (JENKINS-34269) Suspected deadlock in SupportLogHandler.publish

[msz...@wp.pl (JIRA)]

Title: Message Title

Marcin Zajączkowski created an issue   Jenkins / JENKINS-34269 Suspected deadlock in SupportLogHandler.publish Issue Type: Bug Assignee: Steven Christou Components: support-core-plugin Created: 2016/Apr/15 8:46 AM Environment: Jenkins 1.642.3, support-core-plugin 2.31 Priority: Critical Reporter: Marcin Zajączkowski We encountered a situation when 80+ threads were blocked on a lock acquired in SupportLogHandler.publish making the Jenkins instance irresponsible. Most of the threads go through the Pipeline Delivery plugin (which is heavily used by our users), but in the end everything tries to use Jenkins.getItem(java.lang.String). This is not the same as already fixed JENKINS-32622 . Analyzing thread dump I don't see a classic deadlock - all SupportLogHandler calls hang in the same place. Unfortunately the stack trace I have was taken with -F switch and does not have additional information about locks - like who owns the lock. The support-core plugin had been installed a few hours before the accident which happened twice separated by a restart and disappeared after the plugin was removed. I can send privately a more complete thread dump if needed. Thread 105325: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Compiled frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node, int) @bci=67, line=870 (Compiled frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int) @bci=17, line=1199 (Compiled frame) - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock() @bci=21, line=209 (Compiled frame) - java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=285 (Compiled frame) - com.cloudbees.jenkins.support.SupportLogHandler.publish(java.util.logging.LogRecord) @bci=31, line=103 (Compiled frame) - java.util.logging.Logger.log(java.util.logging.LogRecord) @bci=87, line=738 (Compiled frame) - java.util.logging.Logger.doLog(java.util.logging.LogRecord) @bci=46, line=765 (Compiled frame) - java.util.logging.Logger.log(java.util.logging.Level, java.lang.String) @bci=21, line=788 (Compiled frame) - java.util.logging.Logger.finer(java.lang.String) @bci=5, line=1528 (Compiled frame) - hudson.security.SidACL._hasPermission(org.acegisecurity.Authentication, hudson.security.Permission) @bci=183, line=82 (Compiled frame) - hudson.security.SidACL.hasPermission(org.acegisecurity.Authentication, hudson.security.Permission) @bci=63, line=52 (Compiled frame) - hudson.security.ACL.hasPermission(hudson.security.Permission) @bci=5, line=73 (Compiled frame) - hudson.model.AbstractItem.hasPermission(hudson.security.Permission) @bci=5, line=504 (Compiled frame) - jenkins.model.Jenkins.getItem(java.lang.String) @bci=30, line=2396 (Compiled frame) - jenkins.model.Jenkins.getItem(java.lang.String) @bci=2, line=306 (Compiled frame) - jenkins.model.Jenkins.getItemByFullName(java.lang.String, java.lang.Class) @bci=30, line=2504 (Compiled frame) - se.diabol.jenkins.pipeline.util.BuildUtil.getUpstreamBuild(hudson.model.AbstractBuild) @bci=85, line=42 (Compiled frame) - se.diabol.jenkins.pipeline.util.BuildUtil.getFirstUpstreamBuild(hudson.model.AbstractBuild, hudson.model.AbstractProject) @bci=20, line=68 (Compiled frame) - se.diabol.jenkins.pipeline.util.BuildUtil.match(hudson.util.RunList, hudson.model.AbstractBuild) @bci=33, line=87 (Compiled frame) - se.diabol.jenkins.pipeline.domain.task.Task.getAggregatedTask(hudson.model.AbstractBuild, hudson.model.ItemGroup) @bci=12, line=213 (Compiled frame) - se.diabol.jenkins.pipeline.domain.Stage.createAggregatedStage(hudson.model.ItemGroup, hudson.model.AbstractProject) @bci=74, line=206 (Compiled frame) - se.diabol.jenkins.pipeline.domain.Pipeline.createPipelineAggregated(hudson.model.ItemGroup) @bci=46, line=202 (Compiled frame) - se.diabol.jenkins.pipeline.DeliveryPipelineView.getComponent(java.lang.String, hudson.model.AbstractProject, hudson.model.AbstractProject, java.lang.String, boolean) @bci=32, line=451 (Compiled frame) - se.diabol.jenkins.pipeline.DeliveryPipelineView.getPipelines() @bci=109, line=418 (Compiled frame) - sun.reflect.GeneratedMethodAccessor302.invoke(java.lang.Object, java.lang.Object[]) @bci=36 (Compiled frame) - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame) - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=497 (Compiled frame) - org.kohsuke.stapler.export.MethodProperty.getValue(java.lang.Object) @bci=9, line=66 (Compiled frame) (...) Add Comment   This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.