[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Federico Naum commented on JENKINS-53397 Re: ChannelClosedException when using kafka nodes Hey Pham Vu Tuan, We have 2 different instances of Jenkins in production. Currently, I'm using it on the most contained, lower risk instance, only 5 nodes and 2 dozens of jobs. I'm planning to move the other instance that has 12 nodes and 400 jobs soon. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Jeff Thompson commented on JENKINS-53397 Re: ChannelClosedException when using kafka nodes Pham Vu Tuan I have no idea why you would see that stack trace. There shouldn't be anything in 3.26 that would cause it. But, there was another long-standing issue that showed up for someone in 3.26 so something may have tweaked some timing in Remoting. Looks like you must have gotten it figured out well enough to proceed, though. Great! Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Pham Vu Tuan updated JENKINS-53397 Released version 1.1.1 with remoting 3.26, this should not happen again. Jenkins / JENKINS-53397 ChannelClosedException when using kafka nodes Change By: Pham Vu Tuan Status: In Review Resolved Resolution: Fixed Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Pham Vu Tuan updated JENKINS-53397 Jenkins / JENKINS-53397 ChannelClosedException when using kafka nodes Change By: Pham Vu Tuan Status: In Progress Review Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Pham Vu Tuan commented on JENKINS-53397 Re: ChannelClosedException when using kafka nodes Btw, are you using this remoting-kafka-plugin in production? Federico Naum Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Pham Vu Tuan commented on JENKINS-53397 Re: ChannelClosedException when using kafka nodes Hi Jeff Thompson, Im getting this error log when connecting master with kafka agent in remoting 3.26, which doesn't happen when I use 3.25. java.util.concurrent.CancellationExceptionjava.util.concurrent.CancellationException at java.util.concurrent.FutureTask.report(FutureTask.java:121) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at io.jenkins.plugins.remotingkafka.KafkaComputerLauncher.launch(KafkaComputerLauncher.java:119) at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Oct 02, 2018 3:28:58 PM hudson.slaves.ChannelPinger installSEVERE: Failed to set up a ping for testjava.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:177) at hudson.remoting.Channel.call(Channel.java:954) at hudson.slaves.ChannelPinger.install(ChannelPinger.java:112) at hudson.slaves.ChannelPinger.preOnline(ChannelPinger.java:95) at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:660) at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:465) at io.jenkins.plugins.remotingkafka.KafkaComputerLauncher$1.call(KafkaComputerLauncher.java:88) at io.jenkins.plugins.remotingkafka.KafkaComputerLauncher$1.call(KafkaComputerLauncher.java:74) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Is there any special changed thing in 3.26 related to this? And cc Oleg Nenashev, I still can connect Kafka agent to master and do the build when having this exception, seem like another plugin issue Add Comment
[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Jeff Thompson commented on JENKINS-53397 Re: ChannelClosedException when using kafka nodes That's good news, Federico Naum. Pham Vu Tuan, I would be surprised at Remoting 3.26 causing a new exception. Let me know if you continue to see issues there. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Pham Vu Tuan commented on JENKINS-53397 Re: ChannelClosedException when using kafka nodes Thanks for trying it Federico Naum, last time I upgraded remoting 3.26 in my machine it has a strange exception, I will debug it and spin a release to fix these problems soon. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Federico Naum commented on JENKINS-53397 Re: ChannelClosedException when using kafka nodes Just wanted to mention that since I build that against remoting 3.26 I haven't seen this issue again Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Federico Naum commented on JENKINS-53397 Re: ChannelClosedException when using kafka nodes Disregard my last comment, I just saw Oleg Nenashev comments on the PR Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Federico Naum commented on JENKINS-53397 Re: ChannelClosedException when using kafka nodes Hi Pham Vu Tuan, thanks for the quick turn around. I do not know much about jenkins/remoting dependencies, but I noticed in your PR that you have jenkins.version=2.129 , and the remoting version 3.26 is only bundled on Jenkins 2.141 (https://jenkins.io/changelog/) Shouldn't this need to be updated as well? F Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Pham Vu Tuan commented on JENKINS-53397 Re: ChannelClosedException when using kafka nodes yes, merged the PR https://github.com/jenkinsci/remoting-kafka-plugin/pull/43, I will try to release soon by the end of this week. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Jeff Thompson commented on JENKINS-53397 Re: ChannelClosedException when using kafka nodes As Oleg mentions, upgrading to Remoting 3.26 will probably clear up that log message, "Trying to unexport an object that's already unexported". That's a good first step. There are still probably other issues after that. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Pham Vu Tuan started work on JENKINS-53397 Change By: Pham Vu Tuan Status: Open In Progress Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Pham Vu Tuan commented on JENKINS-53397 Re: ChannelClosedException when using kafka nodes Thanks for reporting Federico Naum! As Oleg Nenashev suggested, I will try to update Jenkins and remoting version for the plugin first to see if you are still facing any similar issue. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Federico Naum commented on JENKINS-53397 Re: ChannelClosedException when using kafka nodes I'll try building the kafka plugin against remoting 3.26 and updating Jenkins to 2.141 Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Oleg Nenashev commented on JENKINS-53397 Re: ChannelClosedException when using kafka nodes The unexport pattern looks similar to https://issues.jenkins-ci.org/browse/JENKINS-42533?focusedCommentId=294094=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-294094 . It should be fixed by upgrading Remoting to 3.26. CC Jeff Thompson just in case. Pham Vu Tuan my recommendation would be to update Remoting and spin another release. I am not sure it fixes the root cause of this issue, but at least it will prevent known issues. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Federico Naum updated an issue Jenkins / JENKINS-53397 ChannelClosedException when using kafka nodes Change By: Federico Naum Attachment: AGENT_LOG_after_channel_closed_down.log Attachment: AGENT_LOG_initial.log Attachment: all_2018-08-31_06.00.21.log Attachment: all_2018-09-03_22.59.06.log Attachment: Jenkins_job.log Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Federico Naum commented on JENKINS-53397 Re: ChannelClosedException when using kafka nodes having problems uploading the logs. Will upload them soon Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53397) ChannelClosedException when using kafka nodes
Title: Message Title Federico Naum created an issue Jenkins / JENKINS-53397 ChannelClosedException when using kafka nodes Issue Type: Bug Assignee: Pham Vu Tuan Components: remoting-kafka-plugin Created: 2018-09-04 01:36 Environment: Jenkins Instance (2.135) with 5 agents all connected using Kafka (no security) remoting Kafka plugin 1.1, built with remoting 3.23 Priority: Minor Reporter: Federico Naum Using Remoting Kafka Plugin to version 1.1 (changed the pom file to use the remoting.version=3.23 to match the version on my set up) I've set a Jenkins instance with 5 nodes all connected with kafka. It did run jobs for about 3 days with a considerable load without failures. But today I had the following failure (full log attached): Error when executing always post condition:hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on daybreakk failed. The channel is closing down or has closed down The process that I started in the kafka agent with: > java -jar remoting-kafka-agent.jar -name daybreakk -master http://jenkins:8081/ -secret 6e6c09c25550bd721cef2efba02bfbd54d6306fc711c2482450abb5ee454cd7f -kafkaURL kafka.al.com.au:9093 -noauth exited
