Had a look at the hudson git repo. Could see that there is something wrong in my assumption.
In fact, the connection failure is happening between the Jenkins and the Slave. And it is triggering the destroy. Here destroy also causes error on latest Jenkins in my setup, after the below mentioned ways. I arrived at this conclusion after verifying long pings. I could see ICMP redirect is happening a few times (e.g. ~100 times in 3hrs) for a continuous ping. So to make the long story short, if I can ensure good connectivity between Jenkins and Slave, there won't be any issue. I can see pings giving "From <slave IP minus 1>: icmp_seq=7094 Redirect Host(New nexthop: <slave IP>)", when pinging from Jenkins server. Now seeing how I can solve this problem. Both these servers are VMs on same VMware host, using the same Physical network. On Sunday, 11 February 2018 18:21:52 UTC+5:30, Girish Adat wrote: > > Hello, > > *Sorry for a long post!* > > We recently upgraded to the latest Jenkins - 2.89.3. But we started > getting inconsistent failures from Jenkins builds on our main build server, > which is a remote slave for Jenkins server. These looks to be something > similar to some "postponed" jdk9 bugs ( > https://issues.jenkins-ci.org/browse/JENKINS-46523) > > I tried experimenting the below. > > 1. Upgraded to latest JDK, same version on Jenkins and Slave. > Initially JDK 1.8u152, and later to 1.8u144, where the latest Jenkins is > built. > 2. Tried downgrading to 1.7 on slave. > 3. Using JNLP with latest agent.jar, as well as Jenkins to start > slave.jar via SSH. > 4. Clearing the -jar-cache location. I think after clearing, the > stacktrace is slightly different. Earlier it was complaining > hudson.util.ProcessTree$UnixReflection, similar to > https://issues.jenkins-ci.org/browse/JENKINS-21341 (a bug fixed in > 2015, in 1.5xx version). > 5. Checked the hudson.util.ProcessTree from slave's Groovy console. > Yet to get into slave/agent jar code and see how the class loading works > there. > 1. Below works. (thanks to > https://issues.jenkins-ci.org/browse/JENKINS-6068) > - import hudson.uril.ProcessTree > - println(ProcessTree.class) > 2. Below does NOT work. > - ClassLoader.systemClassLoader.loadClass > "hudson.uril.ProcessTree" > 3. Checked and made sure that the hudson core jar is fetched to > jar cache, and it has the required class. The jar is shown listed in > the > slave.jar's java process, in *lsof* command. > > Some more TODOs I have, though I am assuming I am getting errors in a very > straight forward case. So I should solve than going back to older versions. > > 1. Revert back to our older 2.46.1 version. > 2. Experiment with 2.73 versions. > > From Slave logs could see that it is showing below error. Note that this > is not happening for all the builds, but for at least a two third of the > builds. And the builds are not failing at a common place, as you can > imagine. > > > Feb 10, 2018 8:43:44 PM hudson.remoting.UserRequest perform > > WARNING: LinkageError while performing UserRequest:UserRPCRequest(4,join) > > java.lang.NoClassDefFoundError: hudson/util/ProcessTree > > at hudson.Proc$LocalProc.destroy(Proc.java:384) > > at hudson.Proc$LocalProc.join(Proc.java:357) > > at hudson.Launcher$RemoteLaunchCallable$1.join(Launcher.java:1304) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:922) > > at > hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:896) > > at > hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:853) > > at hudson.remoting.UserRequest.perform(UserRequest.java:207) > > at hudson.remoting.UserRequest.perform(UserRequest.java:53) > > at hudson.remoting.Request$2.run(Request.java:358) > > at > hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > Caused by: java.lang.ClassNotFoundException: hudson.util.ProcessTree > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:159) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > ... 18 more > > I could not understand if I am doing something wrong in the > configurations. Please share if you came across similar problem and managed > to solve it. > > > Thanks and Regards, > > Girish Adat > > > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/98ac5576-192a-4d4e-bf47-9dc94721e1f2%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
