[JIRA] [core] (JENKINS-31253) Lock contention on jenkins.model.Nodes.removeNode
Title: Message Title Ted commented on JENKINS-31253 Re: Lock contention on jenkins.model.Nodes.removeNode The trick part is the whole stack trace didn't show which thread is holding the lock, maybe the lock is held in java native code? I tried to take thread dumps on serval servers at different times, no luck to find the lock holder. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-31253) Lock contention on jenkins.model.Nodes.removeNode
Title: Message Title James Nord commented on JENKINS-31253 Re: Lock contention on jenkins.model.Nodes.removeNode Ted the stack dump is not complete - it only shows threads waiting on the lock. Can you attach a full stack dump of all threads which will include the thread which actually owns the lock. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-31253) Lock contention on jenkins.model.Nodes.removeNode
Title: Message Title Ted commented on JENKINS-31253 Re: Lock contention on jenkins.model.Nodes.removeNode workaround by pull quest https://github.com/jenkinsci/jenkins/pull/1895 Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-31253) Lock contention on jenkins.model.Nodes.removeNode
Title: Message Title Ted edited a comment on JENKINS-31253 Re: Lock contention on jenkins.model.Nodes.removeNode 100+ threads compete for getting the queue lock, that make the system throughput low stack trace on[ http://pastebin.com/iJaeUmJj] By the way, we are running jenkins 1.634 on jdk java version "1.8.0_45"Java(TM) SE Runtime Environment (build 1.8.0_45-b14)Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode) Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-31253) Lock contention on jenkins.model.Nodes.removeNode
Title: Message Title Ted commented on JENKINS-31253 Re: Lock contention on jenkins.model.Nodes.removeNode 100+ threads compete for getting the queue lock, that make the system throughput low stack trace on[ http://pastebin.com/iJaeUmJj] Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-31253) Lock contention on jenkins.model.Nodes.removeNode
Title: Message Title James Nord edited a comment on JENKINS-31253 Re: Lock contention on jenkins.model.Nodes.removeNode At this point what actually is inside the queue?The removal should be quick and as should things inside the queue - so there may be something else wrong here that is causing this. Which version of Jenkins did you observe this on? Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-31253) Lock contention on jenkins.model.Nodes.removeNode
Title: Message Title James Nord commented on JENKINS-31253 Re: Lock contention on jenkins.model.Nodes.removeNode At this point what actually is inside the queue? The removal should be quick and as should things inside the queue - so there may be something else wrong here that is causing this. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-31253) Lock contention on jenkins.model.Nodes.removeNode
Title: Message Title Ted commented on JENKINS-31253 Re: Lock contention on jenkins.model.Nodes.removeNode submitted pull request https://github.com/jenkinsci/jenkins/pull/1893 Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-31253) Lock contention on jenkins.model.Nodes.removeNode
Title: Message Title Ted created an issue Jenkins / JENKINS-31253 Lock contention on jenkins.model.Nodes.removeNode Issue Type: Bug Assignee: Unassigned Components: core Created: 29/Oct/15 10:09 AM Priority: Critical Reporter: Ted We run CI on cloud and the slaves are terminated after one time use, and new slaves are also created on demand. When the slaves number reaches 200 and the throuhput is very slow, and the stacktrace shows over 100 locks parking for java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) parking to wait for <0x7fd6f216d708> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at hudson.model.Queue._withLock(Queue.java:1291) at hudson.model.Queue.withLock(Queue.java:1176) at jenkins.model.Nodes.removeNode(Nodes.java:193) at jenkins.model.Jenkins.removeNode(Jenkins.java:1751) at hudson.plugins.ec2.EC2OndemandSlave.terminate(EC2OndemandSlave.java:72) at hudson.plugins.ec2.EC2O
