Re: Issue with master-agent communication over ssh
On Monday, March 1, 2021 at 4:03:24 PM UTC-5 Jigar R wrote: > I configured my jenkins agent with over 16GB of memory now. I use Apache > ivy & Apache Ant to build the project. I still get resource temporarily > unavailable. > > On the other hand, It works like a charm with Java Web start agent that I > have. I haven't configured any JVM options for it and it just works > smoothly. This is nerve wrecking. > On Saturday, February 13, 2021 at 8:35:07 AM UTC-5 kuisat...@gmail.com > wrote: > >> the memory for the remoting process is not the issue, it is the free >> memory you left for the builds and the systems. I dunno what memory have >> your agents, but let's said they have 4GB if you give 2GB to the remoting >> process you only have 2GB for your builds and the system tasks so probably >> if you run a maven build that generates reports, it will need more than 4GB >> but only have 2GB so the maven process will die and because it grabs all >> the memory probably tear down another system process so the agent will >> disconnect the SSH connection. >> The remoting process usually not need more than 512MB (-Xms512m >> -Xms512m), I use to keep 1-2 GB for the system (ssh, Docker, and >> other services) so if you maven build need 4GB the agent will need at least >> 6-8GB. But all this is a guess because I dunno how much memory your build >> needs (maven or whatever) and I dunno which services you have running on >> your agents, and so on. >> >> Java web start agent uses agent.jar whereas SSH agent uses remoting.jar. Is there a difference between them? > El vie, 12 feb 2021 a las 19:39, Jigar Rathod () >> escribió: >> >>> >>> >>> Sent from my iPhone >>> >>> On Feb 12, 2021, at 1:09 PM, Iván Fernández Calvo >>> wrote: >>> >>> Looks like an abrupt disconnect, this point to the resources you give >>> to the remote process and the resources you have in the agent. The resource >>> management on a JVM is tricky, the JDK gives a 75% of the memory resources >>> if you do not pass Xmx and Xms settings. So I wonder how much memory you >>> have on those agents and if you set those limits. Based on my experience >>> less than 4GB for sunfire tasks use to be tight. >>> >>> Un Saludo >>> Ivan Fernandez Calvo >>> >>> I am using -Xmx2048m -Xms2048m. I will increase it to 4096 and see what >>> happens. >>> >>> Thanks, >>> Jigar R >>> >>> >>> El 12 feb 2021, a las 17:32, Jigar R escribió: >>> >>> >>> >>> On Wednesday, February 10, 2021 at 9:30:28 AM UTC-5 kuisat...@gmail.com >>> wrote: >>> >>>> you attached the Jenkins build log and the Agent log, should be also an >>>> exception in the Jenkins log, Is the same you posted before `invalid >>>> type code: 6D`? >>>> >>>> This is all I see in jenkins.err log >>> >>> 2021-02-11 21:44:00.121+ [id=67650] INFO >>> h.r.SynchronousCommandTransport$ReaderThread#run: I/O error in channel >>> ssh_agent >>> java.io.EOFException >>> at >>> java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:3106) >>> at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1956) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567) >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) >>> at >>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) >>> at >>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) >>> at >>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) >>> at >>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) >>> at hudson.remoting.Command.readFromObjectStream(Command.java:155) >>> at hudson.remoting.Command.readFrom(Command.java:142) >>> at hudson.remoting.Command.readFrom(Command.java:128) >>> at >>> hudson.remoting.AbstractSynchronousByte
Re: Issue with master-agent communication over ssh
I configured my jenkins agent with over 16GB of memory now. I use Apache ivy & Apache Ant to build the project. I still get resource temporarily unavailable. On the other hand, It works like a charm with Java Web start agent that I have. I haven't configured any JVM options for it and it just works smoothly. This is nerve wrecking. On Saturday, February 13, 2021 at 8:35:07 AM UTC-5 kuisat...@gmail.com wrote: > the memory for the remoting process is not the issue, it is the free > memory you left for the builds and the systems. I dunno what memory have > your agents, but let's said they have 4GB if you give 2GB to the remoting > process you only have 2GB for your builds and the system tasks so probably > if you run a maven build that generates reports, it will need more than 4GB > but only have 2GB so the maven process will die and because it grabs all > the memory probably tear down another system process so the agent will > disconnect the SSH connection. > The remoting process usually not need more than 512MB (-Xms512m -Xms512m), > I use to keep 1-2 GB for the system (ssh, Docker, and other services) so if > you maven build need 4GB the agent will need at least 6-8GB. But all this > is a guess because I dunno how much memory your build needs (maven or > whatever) and I dunno which services you have running on your agents, and > so on. > > El vie, 12 feb 2021 a las 19:39, Jigar Rathod () > escribió: > >> >> >> Sent from my iPhone >> >> On Feb 12, 2021, at 1:09 PM, Iván Fernández Calvo >> wrote: >> >> Looks like an abrupt disconnect, this point to the resources you give to >> the remote process and the resources you have in the agent. The resource >> management on a JVM is tricky, the JDK gives a 75% of the memory resources >> if you do not pass Xmx and Xms settings. So I wonder how much memory you >> have on those agents and if you set those limits. Based on my experience >> less than 4GB for sunfire tasks use to be tight. >> >> Un Saludo >> Ivan Fernandez Calvo >> >> I am using -Xmx2048m -Xms2048m. I will increase it to 4096 and see what >> happens. >> >> Thanks, >> Jigar R >> >> >> El 12 feb 2021, a las 17:32, Jigar R escribió: >> >> >> >> On Wednesday, February 10, 2021 at 9:30:28 AM UTC-5 kuisat...@gmail.com >> wrote: >> >>> you attached the Jenkins build log and the Agent log, should be also an >>> exception in the Jenkins log, Is the same you posted before `invalid >>> type code: 6D`? >>> >>> This is all I see in jenkins.err log >> >> 2021-02-11 21:44:00.121+ [id=67650] INFO >> h.r.SynchronousCommandTransport$ReaderThread#run: I/O error in channel >> ssh_agent >> java.io.EOFException >> at >> java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:3106) >> at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1956) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567) >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) >> at >> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) >> at >> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) >> at hudson.remoting.Command.readFromObjectStream(Command.java:155) >> at hudson.remoting.Command.readFrom(Command.java:142) >> at hudson.remoting.Command.readFrom(Command.java:128) >> at >> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) >> at >> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) >> Caused: java.io.IOException: Unexpected termination of the channel >> at >> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77) >> 2021-02-11 21:44:00.122+ [id=67569] WARNING >> h.m.AbstractBuild$AbstractBuildExecution#reportError: Step ‘Record compiler >> warnings and static anal
Re: Issue with master-agent communication over ssh
On Wednesday, February 10, 2021 at 9:30:28 AM UTC-5 kuisat...@gmail.com wrote: > you attached the Jenkins build log and the Agent log, should be also an > exception in the Jenkins log, Is the same you posted before `invalid type > code: 6D`? > > This is all I see in jenkins.err log 2021-02-11 21:44:00.121+ [id=67650] INFO h.r.SynchronousCommandTransport$ReaderThread#run: I/O error in channel ssh_agent java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:3106) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1956) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) at hudson.remoting.Command.readFromObjectStream(Command.java:155) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) Caused: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77) 2021-02-11 21:44:00.122+ [id=67569] WARNING h.m.AbstractBuild$AbstractBuildExecution#reportError: Step ‘Record compiler warnings and static analysis results’ aborted due to exception: java.io.IOException: No workspace found for JOB_NAME #116 at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.perform(IssuesRecorder.java:577) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690) at hudson.model.Build$BuildExecution.post2(Build.java:186) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635) at hudson.model.Run.execute(Run.java:1919) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:428) 2021-02-11 21:44:00.283+ [id=67569] WARNING hudson.ivy.IvyBuildTrigger#recomputeModuleDescriptor: Cannot read ivy file backup...removing ModuleDescriptor 2021-02-11 21:44:00.287+ [id=67569] WARNING hudson.ivy.IvyBuildTrigger#getModuleDescriptor: Node is offline for JOB_NAME, using project to get Module Descriptor > El mar, 9 feb 2021 a las 21:23, Jigar R () escribió: > >> Any recommendations on how should I go about this new error? >> >> On Thursday, February 4, 2021 at 5:26:13 PM UTC-5 Jigar R wrote: >> >>> On Thursday, February 4, 2021 at 12:20:21 PM UTC-5 Jigar R wrote: >>> >>>> On Thu, Feb 4, 2021 at 9:29 AM kuisathaverat >>>> wrote: >>>> >>>>> I see some serialization fails and this breaks the channel, the plugin >>>>> that causes the exception seems >>>>> https://github.com/jenkinsci/tasks-plugin, and the `[Deprecated] Scan >>>>> workspace for open tasks` I think matters, this plugins has been >>>>> integrated >>>>> into https://github.com/jenkinsci/warnings-ng-plugin and >>>>> https://github.com/jenkinsci/analysis-model >>>>> >>>>> ERROR: Step ‘[Deprecated] Scan workspace for open tasks’ aborted due to >>>>> exception: >>>>> java.io.StreamCorruptedException: invalid type code: 6D >>>>> ... >>>>> at >>>>> hudson.plugins.tasks.TasksPublisher.perform(TasksPublisher.java:182) >>>>> at >>>>> hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:69) >>>>> at >>>>> hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:298) >>>>> at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:112) >>>>
Re: Issue with master-agent communication over ssh
Any recommendations on how should I go about this new error? On Thursday, February 4, 2021 at 5:26:13 PM UTC-5 Jigar R wrote: > On Thursday, February 4, 2021 at 12:20:21 PM UTC-5 Jigar R wrote: > >> On Thu, Feb 4, 2021 at 9:29 AM kuisathaverat wrote: >> >>> I see some serialization fails and this breaks the channel, the plugin >>> that causes the exception seems >>> https://github.com/jenkinsci/tasks-plugin, and the `[Deprecated] Scan >>> workspace for open tasks` I think matters, this plugins has been integrated >>> into https://github.com/jenkinsci/warnings-ng-plugin and >>> https://github.com/jenkinsci/analysis-model >>> >>> ERROR: Step ‘[Deprecated] Scan workspace for open tasks’ aborted due to >>> exception: >>> java.io.StreamCorruptedException: invalid type code: 6D >>> ... >>> at hudson.plugins.tasks.TasksPublisher.perform(TasksPublisher.java:182) >>> at >>> hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:69) >>> at >>> hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:298) >>> at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:112) >>> at >>> hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78) >>> at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) >>> at >>> hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741) >>> at >>> hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690) >>> at hudson.model.Build$BuildExecution.post2(Build.java:186) >>> at >>> hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635) >>> at hudson.model.Run.execute(Run.java:1919) >>> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) >>> at hudson.model.ResourceController.execute(ResourceController.java:97) >>> at hudson.model.Executor.run(Executor.java:428) >>> >>> >>> Thanks for this information. I will move to use warning-ng plugin & see >> if it breaks this or not. >> > > I updated jenkins job to use warnings NG instead of deprecated plugins. > After lots of trial and error, I found that jenkins SSH agent would throw > EOFException while running JaCoCo plugin v3.0.7 ( > https://plugins.jenkins.io/jacoco/). Attached logs. > > > > >> >> El jue, 4 feb 2021 a las 14:58, Jigar R () >>> escribió: >>> >>>> Hello Ivan, >>>> >>>> Attached the logs in the email >>>> >>>>- jenkins.log - jenkins build output >>>>- Jenkins-agent.log - output from jenkins ssh agent >>>> >>>> Jenkins SSH agent was created with following: >>>> >>>>- >>>>- launch method : launch agents via ssh >>>>- JavaPath: $JAVA_HOME >>>>- JVM options: -Xmx2048m -Xms2048m >>>>- Use TCP_NODELAY flag on the ssh connection - enabled >>>> >>>> Environment information >>>> >>>> >>>>- Jenkins v2.249.1 >>>>- RH6 >>>>- SSH agent plugin 1.20 >>>>- SSH slaves plugin 1.30.4 >>>> >>>> >>>> On Tuesday, February 2, 2021 at 5:27:07 PM UTC-5 kuisat...@gmail.com >>>> wrote: >>>> >>>>> Weird, Could you share a screencapture of what you configurate? Also >>>>> the whole exception those lines alone mean nothing. Know the version of >>>>> Jenkins and the version of the SSH build agents plugins you use can help, >>>>> the memory you have in your agents, if they are bare metal or cloud. In >>>>> overall if you want help please provide more context. >>>>> >>>>> El mar, 2 feb 2021 a las 22:52, Jigar R () >>>>> escribió: >>>>> >>>>>> >>>>>> >>>>>> On Tuesday, February 2, 2021 at 12:20:19 PM UTC-5 kuisat...@gmail.com >>>>>> wrote: >>>>>> >>>>>>> >I have 2 different kind of jenkins agents. >>>>>>> >1. java web start >>>>>>> >2. ssh >>>>>>> >If the memory was issue then wouldn't it fail on both cases? >>>>>>> >>>>>>> Not necessarily, starting by the
Re: Issue with master-agent communication over ssh
On Thursday, February 4, 2021 at 12:20:21 PM UTC-5 Jigar R wrote: > On Thu, Feb 4, 2021 at 9:29 AM kuisathaverat wrote: > >> I see some serialization fails and this breaks the channel, the plugin >> that causes the exception seems https://github.com/jenkinsci/tasks-plugin, >> and the `[Deprecated] Scan workspace for open tasks` I think matters, this >> plugins has been integrated into >> https://github.com/jenkinsci/warnings-ng-plugin and >> https://github.com/jenkinsci/analysis-model >> >> ERROR: Step ‘[Deprecated] Scan workspace for open tasks’ aborted due to >> exception: >> java.io.StreamCorruptedException: invalid type code: 6D >> ... >> at hudson.plugins.tasks.TasksPublisher.perform(TasksPublisher.java:182) >> at >> hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:69) >> at >> hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:298) >> at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:112) >> at >> hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78) >> at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) >> at >> hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741) >> at >> hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690) >> at hudson.model.Build$BuildExecution.post2(Build.java:186) >> at >> hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635) >> at hudson.model.Run.execute(Run.java:1919) >> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) >> at hudson.model.ResourceController.execute(ResourceController.java:97) >> at hudson.model.Executor.run(Executor.java:428) >> >> >> Thanks for this information. I will move to use warning-ng plugin & see > if it breaks this or not. > I updated jenkins job to use warnings NG instead of deprecated plugins. After lots of trial and error, I found that jenkins SSH agent would throw EOFException while running JaCoCo plugin v3.0.7 (https://plugins.jenkins.io/jacoco/). Attached logs. > > El jue, 4 feb 2021 a las 14:58, Jigar R () escribió: >> >>> Hello Ivan, >>> >>> Attached the logs in the email >>> >>>- jenkins.log - jenkins build output >>>- Jenkins-agent.log - output from jenkins ssh agent >>> >>> Jenkins SSH agent was created with following: >>> >>>- >>>- launch method : launch agents via ssh >>>- JavaPath: $JAVA_HOME >>>- JVM options: -Xmx2048m -Xms2048m >>>- Use TCP_NODELAY flag on the ssh connection - enabled >>> >>> Environment information >>> >>> >>>- Jenkins v2.249.1 >>>- RH6 >>>- SSH agent plugin 1.20 >>>- SSH slaves plugin 1.30.4 >>> >>> >>> On Tuesday, February 2, 2021 at 5:27:07 PM UTC-5 kuisat...@gmail.com >>> wrote: >>> >>>> Weird, Could you share a screencapture of what you configurate? Also >>>> the whole exception those lines alone mean nothing. Know the version of >>>> Jenkins and the version of the SSH build agents plugins you use can help, >>>> the memory you have in your agents, if they are bare metal or cloud. In >>>> overall if you want help please provide more context. >>>> >>>> El mar, 2 feb 2021 a las 22:52, Jigar R () >>>> escribió: >>>> >>>>> >>>>> >>>>> On Tuesday, February 2, 2021 at 12:20:19 PM UTC-5 kuisat...@gmail.com >>>>> wrote: >>>>> >>>>>> >I have 2 different kind of jenkins agents. >>>>>> >1. java web start >>>>>> >2. ssh >>>>>> >If the memory was issue then wouldn't it fail on both cases? >>>>>> >>>>>> Not necessarily, starting by the point that are different ways to >>>>>> establish the connection, the JNLP agents could not update the remoting >>>>>> jar >>>>>> file (depends on your configurations), so you can be running different >>>>>> versions of remoting. I agree with Jeff looks like an OOM issue, review >>>>>> my >>>>>> comments at >>>>>> https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ >>>>>> my re
Re: Issue with master-agent communication over ssh
On Thu, Feb 4, 2021 at 9:29 AM kuisathaverat wrote: > I see some serialization fails and this breaks the channel, the plugin > that causes the exception seems https://github.com/jenkinsci/tasks-plugin, > and the `[Deprecated] Scan workspace for open tasks` I think matters, this > plugins has been integrated into > https://github.com/jenkinsci/warnings-ng-plugin and > https://github.com/jenkinsci/analysis-model > > ERROR: Step ‘[Deprecated] Scan workspace for open tasks’ aborted due to > exception: > java.io.StreamCorruptedException: invalid type code: 6D > ... > at hudson.plugins.tasks.TasksPublisher.perform(TasksPublisher.java:182) > at > hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:69) > at > hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:298) > at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:112) > at > hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78) > at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) > at > hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741) > at > hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690) > at hudson.model.Build$BuildExecution.post2(Build.java:186) > at > hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635) > at hudson.model.Run.execute(Run.java:1919) > at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) > at hudson.model.ResourceController.execute(ResourceController.java:97) > at hudson.model.Executor.run(Executor.java:428) > > > Thanks for this information. I will move to use warning-ng plugin & see if it breaks this or not. El jue, 4 feb 2021 a las 14:58, Jigar R () > escribió: > >> Hello Ivan, >> >> Attached the logs in the email >> >>- jenkins.log - jenkins build output >>- Jenkins-agent.log - output from jenkins ssh agent >> >> Jenkins SSH agent was created with following: >> >>- >>- launch method : launch agents via ssh >>- JavaPath: $JAVA_HOME >>- JVM options: -Xmx2048m -Xms2048m >>- Use TCP_NODELAY flag on the ssh connection - enabled >> >> Environment information >> >> >>- Jenkins v2.249.1 >>- RH6 >>- SSH agent plugin 1.20 >>- SSH slaves plugin 1.30.4 >> >> >> On Tuesday, February 2, 2021 at 5:27:07 PM UTC-5 kuisat...@gmail.com >> wrote: >> >>> Weird, Could you share a screencapture of what you configurate? Also the >>> whole exception those lines alone mean nothing. Know the version of Jenkins >>> and the version of the SSH build agents plugins you use can help, the >>> memory you have in your agents, if they are bare metal or cloud. In overall >>> if you want help please provide more context. >>> >>> El mar, 2 feb 2021 a las 22:52, Jigar R () >>> escribió: >>> >>>> >>>> >>>> On Tuesday, February 2, 2021 at 12:20:19 PM UTC-5 kuisat...@gmail.com >>>> wrote: >>>> >>>>> >I have 2 different kind of jenkins agents. >>>>> >1. java web start >>>>> >2. ssh >>>>> >If the memory was issue then wouldn't it fail on both cases? >>>>> >>>>> Not necessarily, starting by the point that are different ways to >>>>> establish the connection, the JNLP agents could not update the remoting >>>>> jar >>>>> file (depends on your configurations), so you can be running different >>>>> versions of remoting. I agree with Jeff looks like an OOM issue, review my >>>>> comments at >>>>> https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ >>>>> my recommendation is to fix the mem for the remoting process to 1024M >>>>> (-Xmx1024m -Xms1024m) see if the issue disappear or change, if disappear, >>>>> you would have to adjust the remoting process memory to the right one >>>>> between 256M-1024M, to use 512M usually is safe a not too much (but >>>>> depends >>>>> on your agents' memory we do not know how much they have) >>>>> >>>> >>>> I created SSH agent with -Xmx1024m -Xms1024m. >>>> I do see bunch of Warnings about "Ping failed. Terminating the channel >>>> channel". >>>> I got following error: >>>> " >>>>
Re: Issue with master-agent communication over ssh
Hello Ivan, Attached the logs in the email - jenkins.log - jenkins build output - Jenkins-agent.log - output from jenkins ssh agent Jenkins SSH agent was created with following: - - launch method : launch agents via ssh - JavaPath: $JAVA_HOME - JVM options: -Xmx2048m -Xms2048m - Use TCP_NODELAY flag on the ssh connection - enabled Environment information - Jenkins v2.249.1 - RH6 - SSH agent plugin 1.20 - SSH slaves plugin 1.30.4 On Tuesday, February 2, 2021 at 5:27:07 PM UTC-5 kuisat...@gmail.com wrote: > Weird, Could you share a screencapture of what you configurate? Also the > whole exception those lines alone mean nothing. Know the version of Jenkins > and the version of the SSH build agents plugins you use can help, the > memory you have in your agents, if they are bare metal or cloud. In overall > if you want help please provide more context. > > El mar, 2 feb 2021 a las 22:52, Jigar R () escribió: > >> >> >> On Tuesday, February 2, 2021 at 12:20:19 PM UTC-5 kuisat...@gmail.com >> wrote: >> >>> >I have 2 different kind of jenkins agents. >>> >1. java web start >>> >2. ssh >>> >If the memory was issue then wouldn't it fail on both cases? >>> >>> Not necessarily, starting by the point that are different ways to >>> establish the connection, the JNLP agents could not update the remoting jar >>> file (depends on your configurations), so you can be running different >>> versions of remoting. I agree with Jeff looks like an OOM issue, review my >>> comments at >>> https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ >>> my recommendation is to fix the mem for the remoting process to 1024M >>> (-Xmx1024m -Xms1024m) see if the issue disappear or change, if disappear, >>> you would have to adjust the remoting process memory to the right one >>> between 256M-1024M, to use 512M usually is safe a not too much (but depends >>> on your agents' memory we do not know how much they have) >>> >> >> I created SSH agent with -Xmx1024m -Xms1024m. >> I do see bunch of Warnings about "Ping failed. Terminating the channel >> channel". >> I got following error: >> " >> WARNING: Failed to send back a reply to te rquest >> hudson.remoting.Request$ >> java.io.IOException: Resource temporarily unavailable >> at java.io.FileOutputStream.writeBytes(Native method) >> . >> " >> >> >>> >>> El lunes, 1 de febrero de 2021 a las 23:52:19 UTC+1, >>> jigarra...@gmail.com escribió: >>> >>>> On Monday, January 4, 2021 at 12:37:52 PM UTC-5 jtho...@cloudbees.com >>>> wrote: >>>> >>>>> When I saw something like this in the past, it was because the process >>>>> was running out of resources. Specifically when running SpotBugs, I got >>>>> out-of-memory errors. I had to modify the pom to allocate more memory. >>>>> >>>> I have 2 different kind of jenkins agents. >>>> 1. java web start >>>> 2. ssh >>>> If the memory was issue then wouldn't it fail on both cases? >>>> >>>> I do see "Agent went offline during build Connection was broker: >>>> java.io.StreamCorruptedException: invalid stream header:". >>>> >>>>> I recommend ensuring your build runs normally on the agent without the >>>>> additional complexities of the Jenkins environment. This might show where >>>>> additional resources are needed. If that all passes, then continue on to >>>>> the additional troubleshooting steps involving the Jenkins controller and >>>>> agent. >>>>> >>>>> Jeff Thompson >>>>> On 12/30/20 3:40 PM, jiga...@gmail.com wrote: >>>>> >>>>> Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, >>>>> it runs into " Resource temporarily unavailable". >>>>> I have >>>>> >>>>>- verified TCP parameters >>>>>i.e. tcp_keepalive_time, tcp_keepalive_intvl, tcp_keepalive_probes >>>>>& tcp_fin_timeout ; All of them have higher values then what cloudbees >>>>>recommended at >>>>> >>>>> https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected >>>>> >>>>>- set
Re: Issue with master-agent communication over ssh
adPoolExecutor.java:1149) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>> at java.lang.Thread.run(Thread.java:748) >>>>> >>>>> Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run >>>>> WARNING: Failed to send back a reply to the request >>>>> hudson.remoting.Request$2@2511e2d4 >>>>> java.io.IOException: Resource temporarily unavailable >>>>> at java.io.FileOutputStream.writeBytes(Native Method) >>>>> at java.io.FileOutputStream.write(FileOutputStream.java:326) >>>>> at >>>>> hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88) >>>>> at >>>>> hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90) >>>>> at >>>>> hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85) >>>>> at >>>>> hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54) >>>>> at java.io.OutputStream.write(OutputStream.java:75) >>>>> at >>>>> hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45) >>>>> at >>>>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46) >>>>> at hudson.remoting.Channel.send(Channel.java:766) >>>>> at hudson.remoting.Request$2.run(Request.java:388) >>>>> at >>>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>> at java.lang.Thread.run(Thread.java:748) >>>>> >>>>> ERROR: Connection terminated >>>>> java.io.StreamCorruptedException: invalid stream header: 00025B42 >>>>> at >>>>> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866) >>>>> at java.io.ObjectInputStream.(ObjectInputStream.java:358) >>>>> at >>>>> hudson.remoting.ObjectInputStreamEx.(ObjectInputStreamEx.java:49) >>>>> at hudson.remoting.Command.readFrom(Command.java:142) >>>>> at hudson.remoting.Command.readFrom(Command.java:128) >>>>> at >>>>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) >>>>> at >>>>> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) >>>>> Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead >>>>> INFO: Ping failed. Terminating the channel channel. >>>>> java.util.concurrent.TimeoutException: Ping started at 1607120910589 >>>>> hasn't completed by 1607121150590 >>>>> at hudson.remoting.PingThread.ping(PingThread.java:134) >>>>> at hudson.remoting.PingThread.run(PingThread.java:90) >>>>> >>>>> Thanks, >>>>> Jigar R >>>>> >>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to jenkinsci-use...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com?utm_medium=email_source=footer> >>> . >>> >>> -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/615a4333-587c-4ea2-a11f-fea45fe38a56n%40googlegroups.com.
Re: Issue with master-agent communication over ssh
FS discoverGitSystemConfig >>> WARNING: Exception caught during execution of command '[git, config, >>> --system, --edit]' in '$GIT_PATH/bin', return code '128', error message >>> 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory >>> ' >>> Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes >>> saveToConfig >>> WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] >>> failed after 5 retries >>> Dec 04, 2020 3:33:59 PM >>> org.jenkinsci.remoting.util.AnonymousClassWarnings warn >>> WARNING: Attempt to (de-)serialize anonymous class >>> com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; >>> >>> see: https://jenkins.io/redirect/serialization-of-anonymous-classes/ >>> Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run >>> WARNING: Failed to send back a reply to the request >>> hudson.remoting.Request$2@493c5a4e >>> java.io.IOException: Resource temporarily unavailable >>> at java.io.FileOutputStream.writeBytes(Native Method) >>> at java.io.FileOutputStream.write(FileOutputStream.java:313) >>> at >>> hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83) >>> at >>> hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89) >>> at >>> hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62) >>> at >>> hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46) >>> at >>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46) >>> at hudson.remoting.Channel.send(Channel.java:766) >>> at hudson.remoting.Request$2.run(Request.java:388) >>> at >>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> at java.lang.Thread.run(Thread.java:748) >>> >>> Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run >>> WARNING: Failed to send back a reply to the request >>> hudson.remoting.Request$2@2511e2d4 >>> java.io.IOException: Resource temporarily unavailable >>> at java.io.FileOutputStream.writeBytes(Native Method) >>> at java.io.FileOutputStream.write(FileOutputStream.java:326) >>> at >>> hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88) >>> at >>> hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90) >>> at >>> hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85) >>> at >>> hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54) >>> at java.io.OutputStream.write(OutputStream.java:75) >>> at >>> hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45) >>> at >>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46) >>> at hudson.remoting.Channel.send(Channel.java:766) >>> at hudson.remoting.Request$2.run(Request.java:388) >>> at >>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> at java.lang.Thread.run(Thread.java:748) >>> >>> ERROR: Connection terminated >>> java.io.StreamCorruptedException: invalid stream header: 00025B42 >>> at >>> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866) >>> at java.io.ObjectInputStream.(ObjectInputStream.java:358) >>> at >>> hudson.remoting.ObjectInputStreamEx.(ObjectInputStreamEx.java:49) >>> at hudson.remoting.Command.readFrom(Command.java:142) >>> at hudson.remoting.Command.readFrom(Command.java:128) >>> at >>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) >>> at >>> hudso
Re: Confusion with JNLP
On Monday, January 25, 2021 at 7:07:37 PM UTC-5 jtho...@cloudbees.com wrote: > On 1/25/21 1:23 PM, Jigar R wrote: > > Hello Jenkins community, > > I have created jenkins (v2.249.1) agent with launch method "Lauch agent by > connecting it to the master". This prints a command on Jenkins UI i.e. > "java -jar agent.jar -jnlpUrl -secret ". Does this method use java > web start / JNLP4? Is it going away? > > > I meant to send the following explanation to the list a few weeks back but > apparently it didn't work out that way. > > > Hello Jeff, Thanks for your response. I am not well-versed with JNLPs. My only question is that, am I using java web start? Should I be concerned? If yes then Do you recommend me to move to something more stabled? > On 12/31/20 9:32 AM, jiga...@gmail.com wrote: > > I have been running on JNLP for a while. Is it going to be deprecated? > Should I prepare to move to SSH? > > Unfortunately, that's a more confusing question than it should be. There > are three things related to Jenkins agents that are named "JNLP". Only one > of them actually has anything to do with JNLP. > > First, the one that actually has anything to do with JNLP. If you use the > "Launch" button on the controller to start your agent, that uses Java > WebStart and JNLP is the launch protocol / mechanism. WebStart is going > away. Or not. It has been removed from later versions of Java. Though, the > OpenWebStart project has reimplemented these capabilities, which you could > add onto later Java installations. However, I don't recommend using > WebStart in production. It has no restart capabilities. It has become > increasingly difficult to get it to work on current operating systems, etc. > > Second, the one in Jenkins that has been renamed. When you configure an > agent in Jenkins, there used to be an option to have a JNLP agent. This was > renamed to "Inbound TCP agent" a while back as it has nothing to do with > JNLP. This one is not going away. > > Third, the one(s) in Jenkins that are still misnamed. Jenkins Remoting has > had a series of network protocols named "JNLP". Currently, the only > available one is "JNLP4-connect". This name is incorrect, as it has nothing > to do with JNLP. The misleading name is retained for backwards > compatibility purposes. The earlier versions of these protocols had a > variety of problems and have gone away. The current one is not going away. > Jeff Thompson > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e7ab64a4-4f61-4d41-b9da-f4af655bf7b4n%40googlegroups.com.
Confusion with JNLP
Hello Jenkins community, I have created jenkins (v2.249.1) agent with launch method "Lauch agent by connecting it to the master". This prints a command on Jenkins UI i.e. "java -jar agent.jar -jnlpUrl -secret ". Does this method use java web start / JNLP4? Is it going away? -- Jigar R -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/7679139e-e3bd-436d-b50f-23c50b0a5a8dn%40googlegroups.com.
