Hi Jean, Thank you so much for delivering this awesome fight against the red CI.
In the past, such unstable builds have always been linked to some resources leaks during tests. At this point, we usually stopped implementing new things for a while and focus in pluging leaks to bring back build stability. I guess it's this time of the year again and you are the one paying the price for now. The first solution is to recycle JVMs less to mitigate leaks effects (with surefire reusefork option). If it's not enough, I'll have a look at pluging some leaks myself, if nobody else care enough. Keep me posted about the outcome of this change, I do care about having a CI for James very much. Cheers, -- Matthieu Baechler On Wed, 2021-01-13 at 12:50 +0100, Jean Helou wrote: > Happy new year fellow jamers ! > > In this thrilling new episode you might learn if 2021 will be the > year the > james project gets a public ci rolling again ! > > CI wars > Episode 49e^55 > The Memory errors strike back > The CI Resistance succeeded in configuring jenkins, fixed some tests, > exposed some bugs and tagged a lot of unstable tests as being > unstable. > After such a striking defeat the empire of bugs reacted in the most > vicious > way ever, it deployed "Direct buffer memory" errors throughout the > galaxy > to find contributors to the CI effort and tear down their hope and > motivation. They found the apache jenkins and it will need help from > all > the CI resistance members to fight them off ! > > On a bit more serious note, > I am at a loss as to how to fix this issue. My last four builds have > failed > because a `java.lang.OutOfMemoryError: Direct buffer memory` caused > the > forked jvm to crash, crashing the surefire plugin and the build with > it. > and that has been a build failure cause for a lot of the 63 builds on > the > apache CI. until now I updated the pom files of the corresponding > projects > to increase heap to 2G but the last failure occured in a project > where the > heap was already increased. > > Looking at a specific log > https://builds.apache.org/blue/organizations/jenkins/james%2FApacheJames/detail/PR-264/12/pipeline > I get a more classical `java.lang.OutOfMemoryError: Java heap space` > a bit > before (12:14:09.563 vs 12:45:57.988). > The last non error line before the fatal Direct buffer memory error > is > [INFO] Running > org.apache.james.webadmin.integration.rabbitmq.RabbitMQReindexingWith > EventDeadLettersTest > The last non error line before the nonfatal heap memory error is > [INFO] Running > org.apache.james.jmap.memory.cucumber.MemoryDownloadCucumberTest > > I will try to increase surefire's heap for the > memory-jmap-draft-integration-testing project too in case the inital > heap > space OOM triggered the other one. > stackoverflow is not very helpful either > https://stackoverflow.com/search?q=java.lang.OutOfMemoryError%3A+Direct+buffer+memory > or I have not been able to comprehend how the solutions there could > help > > I have gone through the files in /dockerfiles without finding > anything that > looked related to memory configuration of maven itself, If people who > run > the build locally with success or on their own CI could check the > MVN_OPTS > and let me know if they override maven's Xmx itself I would > appreciate it. > > thanks for your help > jean > > > > On Tue, Dec 29, 2020 at 9:10 AM Jean Helou <jean.he...@gmail.com> > wrote: > > > Hi Benoit, > > > > As someone operating another CI, I want to play even unstable test > > on > > > every runs. Is there some adaptation needed to do this? > > > > > > > Yes you will have to change your CI, > > > mvn -B -e -fae test > > now only runs stable tests, to run unstable tests you need an > > additional > > step > > > mvn -B -e -fae test -Punstable-tests > > > > I believe your CI is also based on jenkins (because of the stress > > test > > jenkinsfile at the root of the project) in which case you could > > configure > > your jenkins to pick up the jenkinsfile and use the same pipeline > > as we use > > on apache CI > > > > cheers, > > jean > > --------------------------------------------------------------------- To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
