ok Gavin Thanks for your feedback It's what I wanted to be sure, I didn't want to break something you are maintaining
On Wed, Jul 21, 2021 at 6:47 PM Gavin McDonald <gmcdon...@apache.org> wrote: > Hi Arnaud > > Surem feel free to tweak the node configs directly, these configs are not > in version control or puppet. > > On Wed, Jul 21, 2021 at 6:38 PM Arnaud Héritier <aherit...@gmail.com> > wrote: > >> I am looking at our builds and I try to understand why our agents are >> often disconnected during the builds. >> We have in general a stacktrace like >> >> maven6 was marked offline: Connection was broken: java.io.IOException: Pipe >> closed after 0 cycles >> at >> org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:118) >> at >> org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:101) >> at >> hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:92) >> at >> hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:73) >> at >> hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103) >> at >> hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39) >> at >> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) >> at >> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) >> >> >> >> As far I can see we are using 16Gb "hosts" for linux agents >> >> Something very strange is that the jenkins agent (this small component >> doing the link between the build host and the controller) is configured >> with `-Xms8g -Xmx8g` thus we are reserving for it 50% of the server mem >> (even more because of the non-heap) >> This one in general should require in general really less. 1Gb is already >> a lot from my exp. >> Due to this, the OS can see it has the biggest process on the host and >> decide to kill it when the rest of the memory is used by the build. >> I think we should decrease this value. >> (I can do it but I don't know how was configured the ci.apache.org >> agents and I would like to not add more issue if this setting was here in >> the past >> >> I don't think it is the root cause of our instabilities (at least all) >> and there is something else I have to find but it's a cheap fix to try >> >> FYI our agents VMs are ~like this today: >> >> - Java >> + Home: `/usr/local/asfpackages/java/oraclejdk-1.8.0-291/jre` >> + Vendor: Oracle Corporation >> + Version: 1.8.0_291 >> + Maximum memory: 7.67 GB (8232370176) >> + Allocated memory: 7.67 GB (8232370176) >> + Free memory: 6.03 GB (6470953760) >> + In-use memory: 1.64 GB (1761416416) >> + GC strategy: ParallelGC >> + Available CPUs: 4 >> >> 8Gb is reserved, 1Gb is used (because the GC does nothing as the Free mem >> is high) >> >> I would be in favor to try to launch them with -Xms128m >> -Xmx1g -XX:+UseG1GC -XX:+UseStringDeduplication >> >> I think it's enough customization to start with >> >> Cheers >> >> On Wed, Jul 21, 2021 at 1:28 PM Arnaud Héritier <aherit...@gmail.com> >> wrote: >> >>> I am not sure about the setup >>> AFAICS we don't use any JDK installer ( >>> https://ci-maven.apache.org/configureTools/ ) thus I suppose that the >>> different JDKs are supposed to be installed directly on the agent ? >>> I am not sure how it was done on the previous environment >>> >>> On Sun, Jul 18, 2021 at 5:30 PM Tibor Digana <tibordig...@apache.org> >>> wrote: >>> >>>> The new CI system has the following issue: >>>> >>>> /home/jenkins/tools/java/latest1.7/bin/java: not found >>>> >>>> >>>> https://ci-maven.apache.org/job/Maven/job/maven-box/job/maven-surefire/job/master/104/execution/node/183/log/ >>>> >>>> >>>> >>>> On Wed, Jun 30, 2021 at 8:03 PM Gavin McDonald <gmcdon...@apache.org> >>>> wrote: >>>> >>>> > Hi Maven folks. >>>> > >>>> > Infra has decided to separate off the Maven build jobs from >>>> > ci-builds.apache.org over to its very own Jenkins Controller and >>>> Agents. >>>> > >>>> > This means that Maven now has a dedicated Jenkins environment for >>>> itself. >>>> > It >>>> > also means that no other projects build jobs can build on the Maven >>>> nodes; >>>> > and >>>> > then Maven jobs will no longer be able to build on the ci-builds >>>> jobs. >>>> > >>>> > Your new Controller is set up as https://ci-maven.apache.org and all >>>> Maven >>>> > Committers >>>> > can login via LDAP and create jobs. >>>> > >>>> > At the time of writing, there is one node/agent attached but I am >>>> building >>>> > 4 more - all >>>> > Ubuntu 20.04 and based in our Azure account. >>>> > >>>> > We can automagically move all your jobs over from ci-builds to >>>> ci-maven - I >>>> > just need someone to tell me go ahead and do it. >>>> > >>>> > In the meantime, feel free to have a test. The remaining 4 agents >>>> will be >>>> > online >>>> > by tomorrow. We will review after a month if 5 is enough nodes. >>>> > >>>> > As with other projects having their own dedicated controller, who have >>>> > taken advantage >>>> > of this isolation by having some nodes/agents given to the project as >>>> a >>>> > 'targeted donation' >>>> > so someone here may know of a Company will to donate 5 - 10 or more >>>> nodes >>>> > specifically >>>> > for Maven Jenkins environment. Infra can afford to hand you over 5 >>>> right >>>> > now. >>>> > >>>> > Let me know if you have any questions, otherwise let me know when I >>>> can >>>> > make the >>>> > transfer of your jobs. >>>> > >>>> > Thanks >>>> > >>>> > -- >>>> > >>>> > *Gavin McDonald* >>>> > Systems Administrator >>>> > ASF Infrastructure Team >>>> > >>>> >>> >>> >>> -- >>> Arnaud Héritier >>> Twitter/Skype : aheritier >>> >> >> >> -- >> Arnaud Héritier >> Twitter/Skype : aheritier >> > > > -- > > *Gavin McDonald* > Systems Administrator > ASF Infrastructure Team > -- Arnaud Héritier Twitter/Skype : aheritier
