Hi Arnaud Surem feel free to tweak the node configs directly, these configs are not in version control or puppet.
On Wed, Jul 21, 2021 at 6:38 PM Arnaud Héritier <aherit...@gmail.com> wrote: > I am looking at our builds and I try to understand why our agents are > often disconnected during the builds. > We have in general a stacktrace like > > maven6 was marked offline: Connection was broken: java.io.IOException: Pipe > closed after 0 cycles > at > org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:118) > at > org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:101) > at > hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:92) > at > hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:73) > at > hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103) > at > hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39) > at > hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) > at > hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) > > > > As far I can see we are using 16Gb "hosts" for linux agents > > Something very strange is that the jenkins agent (this small component > doing the link between the build host and the controller) is configured > with `-Xms8g -Xmx8g` thus we are reserving for it 50% of the server mem > (even more because of the non-heap) > This one in general should require in general really less. 1Gb is already > a lot from my exp. > Due to this, the OS can see it has the biggest process on the host and > decide to kill it when the rest of the memory is used by the build. > I think we should decrease this value. > (I can do it but I don't know how was configured the ci.apache.org agents > and I would like to not add more issue if this setting was here in the past > > I don't think it is the root cause of our instabilities (at least all) and > there is something else I have to find but it's a cheap fix to try > > FYI our agents VMs are ~like this today: > > - Java > + Home: `/usr/local/asfpackages/java/oraclejdk-1.8.0-291/jre` > + Vendor: Oracle Corporation > + Version: 1.8.0_291 > + Maximum memory: 7.67 GB (8232370176) > + Allocated memory: 7.67 GB (8232370176) > + Free memory: 6.03 GB (6470953760) > + In-use memory: 1.64 GB (1761416416) > + GC strategy: ParallelGC > + Available CPUs: 4 > > 8Gb is reserved, 1Gb is used (because the GC does nothing as the Free mem > is high) > > I would be in favor to try to launch them with -Xms128m > -Xmx1g -XX:+UseG1GC -XX:+UseStringDeduplication > > I think it's enough customization to start with > > Cheers > > On Wed, Jul 21, 2021 at 1:28 PM Arnaud Héritier <aherit...@gmail.com> > wrote: > >> I am not sure about the setup >> AFAICS we don't use any JDK installer ( >> https://ci-maven.apache.org/configureTools/ ) thus I suppose that the >> different JDKs are supposed to be installed directly on the agent ? >> I am not sure how it was done on the previous environment >> >> On Sun, Jul 18, 2021 at 5:30 PM Tibor Digana <tibordig...@apache.org> >> wrote: >> >>> The new CI system has the following issue: >>> >>> /home/jenkins/tools/java/latest1.7/bin/java: not found >>> >>> >>> https://ci-maven.apache.org/job/Maven/job/maven-box/job/maven-surefire/job/master/104/execution/node/183/log/ >>> >>> >>> >>> On Wed, Jun 30, 2021 at 8:03 PM Gavin McDonald <gmcdon...@apache.org> >>> wrote: >>> >>> > Hi Maven folks. >>> > >>> > Infra has decided to separate off the Maven build jobs from >>> > ci-builds.apache.org over to its very own Jenkins Controller and >>> Agents. >>> > >>> > This means that Maven now has a dedicated Jenkins environment for >>> itself. >>> > It >>> > also means that no other projects build jobs can build on the Maven >>> nodes; >>> > and >>> > then Maven jobs will no longer be able to build on the ci-builds jobs. >>> > >>> > Your new Controller is set up as https://ci-maven.apache.org and all >>> Maven >>> > Committers >>> > can login via LDAP and create jobs. >>> > >>> > At the time of writing, there is one node/agent attached but I am >>> building >>> > 4 more - all >>> > Ubuntu 20.04 and based in our Azure account. >>> > >>> > We can automagically move all your jobs over from ci-builds to >>> ci-maven - I >>> > just need someone to tell me go ahead and do it. >>> > >>> > In the meantime, feel free to have a test. The remaining 4 agents will >>> be >>> > online >>> > by tomorrow. We will review after a month if 5 is enough nodes. >>> > >>> > As with other projects having their own dedicated controller, who have >>> > taken advantage >>> > of this isolation by having some nodes/agents given to the project as a >>> > 'targeted donation' >>> > so someone here may know of a Company will to donate 5 - 10 or more >>> nodes >>> > specifically >>> > for Maven Jenkins environment. Infra can afford to hand you over 5 >>> right >>> > now. >>> > >>> > Let me know if you have any questions, otherwise let me know when I can >>> > make the >>> > transfer of your jobs. >>> > >>> > Thanks >>> > >>> > -- >>> > >>> > *Gavin McDonald* >>> > Systems Administrator >>> > ASF Infrastructure Team >>> > >>> >> >> >> -- >> Arnaud Héritier >> Twitter/Skype : aheritier >> > > > -- > Arnaud Héritier > Twitter/Skype : aheritier > -- *Gavin McDonald* Systems Administrator ASF Infrastructure Team
