[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
Title: Message Title Oleg Nenashev commented on JENKINS-28492 Re: The server rejected the connection: *** is already connected to this master. Rejecting this connection. Some bits have been addressed in JENKINS-39835 Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
Title: Message Title Matthew Mitchell commented on JENKINS-28492 Re: The server rejected the connection: *** is already connected to this master. Rejecting this connection. This also appears to perhaps happen "near" a failure https://issues.jenkins-ci.org/browse/JENKINS-33358 Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
Title: Message Title Matthew Mitchell commented on JENKINS-28492 Re: The server rejected the connection: *** is already connected to this master. Rejecting this connection. Right, for sure. I think there are a few cases: 1) Cases where plugins/core usae computer.getChannel(). These work fine since if the listeners run properly, the channel is set to null and the code can do the right thing at that point depending on what its function is. 2) Cases where the channel object is held on an object (Ping thread, etc.) but there is error handling. These appear okay (ping thread exits). 3) Cases where the channel object is held on an object but no error handling - These need fixing. I don't know of any real cases of #3. #2 appears okay. #1 is a problem because the listeners which should null out the channel on the computer object don't actually run. That is why we see the "already connected" issue. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
Title: Message Title Oleg Nenashev commented on JENKINS-28492 Re: The server rejected the connection: *** is already connected to this master. Rejecting this connection. Maybe. I think we firstly need to implement correct handling of Errors and RuntimeExceptions in the core in order to avoid the case when the channel object leaks after the failure. It should help with some cases Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
Title: Message Title Matthew Mitchell commented on JENKINS-28492 Re: The server rejected the connection: *** is already connected to this master. Rejecting this connection. Yep, that correlates with what I've seen too. What is odd about this is that the actual usage doesn't appear that large. A heap dump before the failure doesn't show a heap that is anywhere close to full (3GB usage of 32GB). But I do see lots of "unable to allocate new native thread", etc and the number of threads grows very large from the steady state. Could this be related to a deadlock of some sort where we start to spawn threads till we die? Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
<<< text/html; charset=UTF-8: Unrecognized >>>
[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
Title: Message Title Oleg Nenashev edited a comment on JENKINS-28492 Re: The server rejected the connection: *** is already connected to this master. Rejecting this connection. Another *known* case when the issue happens - OutOfMemory exception - JENKINS-30823 Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
Title: Message Title Matthew Mitchell commented on JENKINS-28492 Re: The server rejected the connection: *** is already connected to this master. Rejecting this connection. Do you use Job DSL too? I think there might be something there that is related too. Based on the data seen, I think this may be a deadlock. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
Title: Message Title mariem baccar commented on JENKINS-28492 Re: The server rejected the connection: *** is already connected to this master. Rejecting this connection. You find below some explications for my case which can help you: -Before installing the plugin "Docker Slave v1.0.5", the slave operates correctly without any problems. -After installing this plugin, I encountered many problems. One of them is about slave: I always get this message "slave agent is already connected". For more details, you find attached the whole message. In fact, the problem is related to JENKINS-39078: There is a problem in Docker Slave Plugin 1.0.5 (Fix: https://github.com/jenkinsci/docker-slaves-plugin/commit/451929125fd8ff39c6f84c30476c26cccb912140) After discovering that this plugin recently installed is the source of all my new problems in Jenkins 2.19.1 LTS, I uninstall it and all my problems are resolved. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
Title: Message Title mariem baccar updated an issue Jenkins / JENKINS-28492 The server rejected the connection: *** is already connected to this master. Rejecting this connection. Change By: mariem baccar Attachment: Slave errors Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
Title: Message Title Matthew Mitchell commented on JENKINS-28492 Re: The server rejected the connection: *** is already connected to this master. Rejecting this connection. FYI when this problem occurs the number of threads starts to climb from steady state at ~800-1500 up to 11k. Most of the extra threads look like they are doing nothing (just a single frame on a thread named Thread-NN). Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
Title: Message Title Matthew Mitchell commented on JENKINS-28492 Re: The server rejected the connection: *** is already connected to this master. Rejecting this connection. Also, this Issue looks extremely close to what I am seeing. Perhaps there is a lock on the channel object? https://issues.jenkins-ci.org/browse/JENKINS-5413 Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
Title: Message Title Matthew Mitchell commented on JENKINS-28492 Re: The server rejected the connection: *** is already connected to this master. Rejecting this connection. I added a bunch more logging to the system to see why the channel would not get cleared out. I've got another couple pieces of info: For me, there seems to be some relation to Job DSL execution. Not exactly sure why. There's definitely something memory related, as we see a big jump in memory usage when this happens, and there are for sure some allocation failures. However, the overall heap usage is actually really low (compared to avialable size), so it might be the GC telling us unable to allocate. When the error happens, I see the following sequence. 1. Timeout failures monitoring for clock difference, disk space, etc. on the node. 2. Ping failure on the node. 3. already connected to this master. Now, some other things to note. I added logging to the listeners on the channel set up that would close the channel and null out the reference on the Computer object (onClosed). However, for the affected nodes, the onClosed methods are never executed, so the listeners never get run. Why that is, is the big mystery. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues"
[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
Title: Message Title Oleg Nenashev commented on JENKINS-28492 Re: The server rejected the connection: *** is already connected to this master. Rejecting this connection. For runaway processes in WinSW there is an open pull-request: https://github.com/kohsuke/winsw/pull/133 But I agree with Matthew Mitchell that it's is not the only cause. We also have some cases with channel hanging due to deadlocks, which have been likely fixed by https://github.com/jenkinsci/remoting/pull/100 Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
Title: Message Title Matthew Mitchell commented on JENKINS-28492 Re: The server rejected the connection: *** is already connected to this master. Rejecting this connection. Some more info. While I think the "already connected to this master" is a real bug, I think the way that it's triggered might be considered expected. I see this when the number of threads in the system suddenly spikes. This seems to cause some cascading failures (maybe "Unable to allocate new native thread")? In our instance, this might be caused by the Build Failure Analyzer, which can use a lot of threads. I do know some other things: 1) Raising the java heap size reduces the amount of memory for native threads and potentially seems to reduce the time to repro. 2) Allocating a ton of new threads very fast seems to cause this. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
Title: Message Title Matthew Mitchell commented on JENKINS-28492 Re: The server rejected the connection: *** is already connected to this master. Rejecting this connection. So I think the "already connected" may be caused by a cycle in the closing logic. Though I don't know why the error keeps propping up. It looks like the channel pinger could cause a channel close to happen. Under the jnlp remoting, we set up a channel with an onClose listener, which itself tries to close the channel under certain circumstances. These listeners are run in the channel termination finally logic, so it's possible that we could try to close while we are already under the close operations. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
Title: Message Title Matthew Mitchell commented on JENKINS-28492 Re: The server rejected the connection: *** is already connected to this master. Rejecting this connection. hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:578) at hudson.remoting.Request.callAsync(Request.java:205) at hudson.remoting.Channel.callAsync(Channel.java:807) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:75) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.io.IOException at hudson.remoting.Channel.close(Channel.java:1163) at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:123) at hudson.remoting.PingThread.ping(PingThread.java:126) at hudson.remoting.PingThread.run(PingThread.java:85) Caused by: java.util.concurrent.TimeoutException: Ping started at 1479165919577 hasn't completed by 1479166819577 ... 2 more Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
Title: Message Title Oleg Nenashev edited a comment on JENKINS-28492 Re: The server rejected the connection: *** is already connected to this master. Rejecting this connection. Added to my analysis queue (no time commitment).Issue with runaway slaves processes in Windows services should be fixed by JENKINS-39231 Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
Title: Message Title Oleg Nenashev commented on JENKINS-28492 Re: The server rejected the connection: *** is already connected to this master. Rejecting this connection. Added to my analysis queue (no time commitment). Issue with runaway slaves should be fixed by JENKINS-39231 Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-28492) The server rejected the connection: *** is already connected to this master. Rejecting this connection.
Title: Message Title Oleg Nenashev assigned an issue to Oleg Nenashev Jenkins / JENKINS-28492 The server rejected the connection: *** is already connected to this master. Rejecting this connection. Change By: Oleg Nenashev Assignee: Oleg Nenashev Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
