Hello all, I was able to solve my problem. If you are interested find the full explanation on the ticket itself: https://issues.jenkins-ci.org/browse/JENKINS-59137?focusedCommentId=374533&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-374533
Cheers On Thursday, August 29, 2019 at 1:05:20 PM UTC+2, Toni Van de Voorde wrote: > > Hey everyone, > > I suddenly have issues with my master slave setup on jenkins. It seems > there is a connectivity issue each time a job wants to run on it. I already > opened a ticket on Jira for ec2-plugin ( > https://issues.jenkins-ci.org/browse/JENKINS-59137) but not knowing if > the problem is really in that plugin, or if they uberhaubt are going to > answer quickly, I'm trying my luck here, in the hope someone already > encountered this. > > As previously mentioned, AFAIK the problem suddenly happened ... I did > upgrade to latest versions when the problem started, but the it didn't > solve it. > > So what's really happening is that whenever a job needs to a run, the > master starts a slave and executes the job ... but the job fails in 2 use > cases: > > *1. The job fails immediately* > > Branch event > Obtained Jenkinsfile from 140cfe60aa7963ef13cb23c18f07bf70ee97d061 > Running in Durability level: MAX_SURVIVABILITY > [Pipeline] Start of Pipeline > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] stage > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] { (php 5.6 and php 7) > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] parallel > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] { (Branch: php 5.6) > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] { (Branch: php 7) > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] stage > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] { (php 5.6) > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] stage > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] { (php 7) > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] node > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] node > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > Running on EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxx) > <https://ci.adsdaq.eu/computer/EC2%20(jenkins-slave)%20-%20Adsdaq%20Jenkins%20Slave%20(i-0c2875a63156eab4c)/> > in /home/ec2-user/workspace/adsdaq_portal_master > [Pipeline] { > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] checkout > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] } > [Pipeline] // node > [Pipeline] } > [Pipeline] // stage > Running on EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxx) > <https://ci.adsdaq.eu/computer/EC2%20(jenkins-slave)%20-%20Adsdaq%20Jenkins%20Slave%20(i-0c2875a63156eab4c)/> > in /home/ec2-user/workspace/adsdaq_portal_master > [Pipeline] { > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] } > Failed in branch php 5.6 > [Pipeline] checkout > <https://ci.adsdaq.eu/job/adsdaq/job/portal/job/master/856/console#> > [Pipeline] } > [Pipeline] // node > [Pipeline] } > [Pipeline] // stage > [Pipeline] } > Failed in branch php 7 > [Pipeline] // parallel > [Pipeline] } > > > This job above runs in parallel and even executes 2 slaves. It happens > that both fail (like here), but sometimes one passes. > > The additional stack trace we get in this use case is: > > Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 > (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxx) > at > hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) > at > hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356) > at hudson.remoting.Channel.call(Channel.java:955) > at hudson.FilePath.act(FilePath.java:1072) > at hudson.FilePath.act(FilePath.java:1061) > at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144) > at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822) > at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813) > at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) > at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124) > at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) > at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) > at > org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > java.lang.NoClassDefFoundError: Could not initialize class > com.sun.proxy.$Proxy8 > at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown > Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739) > at > hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:167) > at hudson.remoting.Channel.export(Channel.java:768) > at hudson.remoting.Channel.export(Channel.java:731) > at > org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:204) > at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1230) > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1136) > at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) > at hudson.remoting.UserRequest._serialize(UserRequest.java:263) > at hudson.remoting.UserRequest.serialize(UserRequest.java:272) > at hudson.remoting.UserRequest.perform(UserRequest.java:222) > at hudson.remoting.UserRequest.perform(UserRequest.java:54) > at hudson.remoting.Request$2.run(Request.java:369) > at > hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) > Also: Also: hudson.remoting.Channel$CallSiteStackTrace: Remote > call to EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxx) > at > hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) > at > hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356) > at hudson.remoting.Channel.call(Channel.java:955) > at hudson.FilePath.act(FilePath.java:1072) > at hudson.FilePath.act(FilePath.java:1061) > at > org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144) > at > hudson.plugins.git.GitSCM.createClient(GitSCM.java:822) > at > hudson.plugins.git.GitSCM.createClient(GitSCM.java:813) > at > hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) > at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124) > at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) > at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) > at > org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > java.lang.NoClassDefFoundError: Could not initialize class > com.sun.proxy.$Proxy8 > at > sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at > java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739) > at > hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:167) > at hudson.remoting.Channel.export(Channel.java:768) > at hudson.remoting.Channel.export(Channel.java:731) > at > org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:204) > at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown > Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1230) > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1136) > at > java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) > at > hudson.remoting.UserRequest._serialize(UserRequest.java:263) > at hudson.remoting.UserRequest.serialize(UserRequest.java:272) > at hudson.remoting.UserRequest.perform(UserRequest.java:222) > at hudson.remoting.UserRequest.perform(UserRequest.java:54) > at hudson.remoting.Request$2.run(Request.java:369) > at > hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) > Caused: java.io.IOException: Remote call on EC2 (jenkins-slave) - > Adsdaq Jenkins Slave (i-xxxx) failed > Caused: java.io.IOException: Remote call on EC2 (jenkins-slave) - Adsdaq > Jenkins Slave (i-xxxx) failed > at hudson.remoting.Channel.call(Channel.java:961) > at hudson.FilePath.act(FilePath.java:1072) > at hudson.FilePath.act(FilePath.java:1061) > at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144) > at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822) > at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813) > at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) > at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124) > at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) > at > org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) > at > org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Finished: FAILURE > > > *2. The job executes but fails when recording the junits* > > Another scenario which happens (randomly) is that the execution starts > correctly, but when all steps were performed, the job fails from the moment > the junits need to be recorded: > > [30;42mOK (715 tests, 2105 assertions) [0m > > Generating code coverage report in Clover XML format ... done > > Generating code coverage report in HTML format ... done > > Remaining indirect deprecation notices (17) > > Other deprecation notices (7) > [Pipeline] step > <https://ci.adsdaq.eu/job/adsdaq/job/print/job/engine/job/PR-258/8/console#> > Publishing Clover coverage report... > Publishing Clover HTML report... > Publishing Clover XML report... > Processing Clover XML report ... > Publishing Clover coverage results... > [Pipeline] junit > <https://ci.adsdaq.eu/job/adsdaq/job/print/job/engine/job/PR-258/8/console#> > Recording test results > Remote call on EC2 (jenkins-slave) - Adsdaq Jenkins Slave > (i-0617d5ac254d28fce) failed > > > The stack trace provide after is: > > java.io.FileNotFoundException: > /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-0.amzn2.x86_64/jre/lib/resources.jar > at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:938) > at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:801) > at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:886) > at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:879) > at java.security.AccessController.doPrivileged(Native Method) > at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:878) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1050) > Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 > (jenkins-slave) - Adsdaq Jenkins Slave (i-xxx) > at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) > at > hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356) > at hudson.remoting.Channel.call(Channel.java:955) > at hudson.FilePath.act(FilePath.java:1072) > at hudson.FilePath.act(FilePath.java:1061) > at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:114) > at > hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:137) > at > hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:167) > at > hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:52) > at > hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:25) > at > org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > Caused: java.lang.InternalError > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1052) > at sun.misc.URLClassPath.getResource(URLClassPath.java:249) > at sun.misc.URLClassPath.getResource(URLClassPath.java:302) > at java.lang.ClassLoader.getBootstrapResource(ClassLoader.java:1264) > at java.lang.ClassLoader.getResource(ClassLoader.java:1093) > at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:233) > at > java.util.ResourceBundle$RBClassLoader.getResourceAsStream(ResourceBundle.java:524) > at java.util.ResourceBundle$Control$1.run(ResourceBundle.java:2686) > at java.util.ResourceBundle$Control$1.run(ResourceBundle.java:2671) > at java.security.AccessController.doPrivileged(Native Method) > at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2670) > at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1510) > at java.util.ResourceBundle.findBundle(ResourceBundle.java:1474) > at java.util.ResourceBundle.findBundle(ResourceBundle.java:1428) > at java.util.ResourceBundle.findBundle(ResourceBundle.java:1428) > at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1370) > at java.util.ResourceBundle.getBundle(ResourceBundle.java:854) > at > com.sun.org.apache.xerces.internal.utils.SecuritySupport$7.run(SecuritySupport.java:169) > at > com.sun.org.apache.xerces.internal.utils.SecuritySupport$7.run(SecuritySupport.java:166) > at java.security.AccessController.doPrivileged(Native Method) > at > com.sun.org.apache.xerces.internal.utils.SecuritySupport.getResourceBundle(SecuritySupport.java:166) > at > com.sun.org.apache.xerces.internal.util.SAXMessageFormatter.formatMessage(SAXMessageFormatter.java:58) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.getProperty(AbstractSAXParser.java:2063) > at > com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.setProperty(SAXParserImpl.java:574) > at org.dom4j.io.SAXHelper.setParserProperty(SAXHelper.java:34) > at org.dom4j.io.SAXReader.configureReader(SAXReader.java:911) > at org.dom4j.io.SAXReader.read(SAXReader.java:463) > at org.dom4j.io.SAXReader.read(SAXReader.java:343) > at hudson.tasks.junit.SuiteResult.parse(SuiteResult.java:178) > at hudson.tasks.junit.TestResult.parse(TestResult.java:348) > at hudson.tasks.junit.TestResult.parsePossiblyEmpty(TestResult.java:281) > at hudson.tasks.junit.TestResult.parse(TestResult.java:206) > at hudson.tasks.junit.TestResult.parse(TestResult.java:178) > at hudson.tasks.junit.TestResult.<init>(TestResult.java:143) > at > hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:146) > at > hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:118) > at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3052) > at hudson.remoting.UserRequest.perform(UserRequest.java:211) > at hudson.remoting.UserRequest.perform(UserRequest.java:54) > at hudson.remoting.Request$2.run(Request.java:369) > at > hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) > Caused: java.io.IOException: Remote call on EC2 (jenkins-slave) - Adsdaq > Jenkins Slave (i-xxx) failed > at hudson.remoting.Channel.call(Channel.java:961) > at hudson.FilePath.act(FilePath.java:1072) > at hudson.FilePath.act(FilePath.java:1061) > at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:114) > at > hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:137) > at > hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:167) > at > hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:52) > at > hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:25) > at > org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Finished: FAILURE > > > As you can see these are very strange errors. I repeat that we didn't > change anything in the structure of the configuration. Not sure if it's a > problem with ec2-plugin, or a change somewhere in amazon. > > If anybody has or had this issue and knows how to solve it, I'm all ears, > as this is blocking our CD flow :/ > > Thanks in advance > > Cheers > > Toni > > > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2d48a635-54eb-4b31-8a05-10cad3446ea0%40googlegroups.com.
