Just chiming in on my experience using the redis locks with maven 3.8.x and resolver 1.6. I've been running it on my Jenkins nodes for about a year now with great success. Before it, we had corrupted local maven repos (zip file empty kind of exceptions) a couple of times a week. Then, we configured Jenkins to properly handle multiple concurrent builds by not sharing the maven local repo and run each build in docker which is wasteful since you have to redownload all your dependencies on each build and build time is obviously very long.
Then I bumped into the lock feature which is available on resolver 1.6 (instructions here https://svn.apache.org/repos/asf/maven/website/components/resolver-archives/resolver-1.6.3/maven-resolver-synccontext-redisson/index.html). It is a bit cumbersome to setup but after that, we started sharing the same local repos and our build time dramatically improved (something like from 4 minutes to ~30 seconds) and we had a corrupted maven repo once or twice throughout the year (and it could have been some job that a dev created without the proper config, I didn't investigate much). Redis is really easy to setup and is available via docker or apt so it should be a breeze to add it to your init script like we did. My nodes had docker already installed so it was just adding this to the "Init script" in the nodes configuration : #!/bin/bash -ue docker run -d --network host --name mvn_redis redis:6.2-alpine I'd be interested to keep up with all the latest work that have been done on this in resolver 1.7, that's why I created another issue to request a java 8 maven 3.9 release. On 2021/10/04 20:13:45, Francois Marot <francois.ma...@gmail.com> wrote: > "Salut" Michael, > > thanks for the detailled answer. > Regarding my Jenkins 'hack' you can totally forget about it: it just > ensures no more than one Maven execution uses a given repo at once by > locating the local repositories in a folder reflecting the executor (which > are kinda Jenkins "threads"). > I plan not to use your work right now mostly because it would involve > setting up a Redis Database and my 'hack' still works (despite using a lot > of disk space and being a bit slow). > > If I understand correctly after Tams' message, your work is mostly geared > toward running multiple Maven build in parallel and on multiple hosts, all > sharing one big "local" repository through the network. Am I correct ? In > my case, I have multiple Jenkins workers but never thought about making > them collaborate on a shared "local" repo. I'll have to think about it ! > > Thanks ! > > François > > > > Le lun. 4 oct. 2021 à 21:22, Michael Osipov <micha...@apache.org> a écrit : > > > Am 2021-10-04 um 14:31 schrieb Francois Marot: > > > Hello all, > > > > > > I would like clarifications on MNG-2802[*] that seems to be solved. If I > > > understand correctly it solves the problem where multiple simultaneous > > > Maven executions shared the same local repository. > > > I have been keeping for years a bit of code in my Jenkinsfiles ensuring > > the > > > local repos were accessed only by a single jenkins executor at the same > > > time, so it seems like I can get rid of this doesn't it ? > > > I'd like your input on this because reproducing the bug is difficult so > > I'd > > > like to be sure before simplifying my Jenkinsfile. > > > > > > Moreover, does anyone know how this problem is solved technically ? Using > > > files lock ? Can anybody explain ? > > > > > > Thanks you and thanks the Maven team for keeping up the good work at such > > > pace ! > > > > Salut François, > > > > I don't know what you exactly fiddled in your Jenkinsfile, but there is > > no way make it right just with Jenkinsfile foo when you want access > > *one* local repo with *multiple* processes you *must* use an external > > coordinator process. > > Forget also about file locks, they are all advisory, at least on > > POSIX-like filesystems. There is no way, or a very hard way to make them > > right. We tried and removed them. > > > > Now, let me give you bit of history: I have integrated/implemented a > > fully working multi-process Redisson-based synchronization lock factory > > for Maven Resolver 1.6.x which ships with Maven 3.8.x. (It has already > > been battle tested for several months by a user on a busy CI server). > > Tamás took it to the next level in Resolver 1.7.x and generalized it to > > a named lock approach where the Redisson Lock Factory plugs in nicely > > with other lock implementations. Almost nine months of work and it works > > very good now. > > Now here is the problem: Maven Resolver 1.7.x comes only with Maven 4.x. > > I have created a branch which reconciles best of breed: Maven 3.8.x and > > Maven Resolver 1.7.x. It just works (also verified by this user). I also > > ran more than a thousand builds with 8 parallel processes and 4 threads. > > It will easily handle thousands of locks in parralel with an one digit > > percent overhead. > > > > I don't know for what documentation you are waiting for, but everything > > is there, you can start *now* to leverage it: [2], [3]. With Resolver > > 2.0.0 and Maven 4.0.0 it will likely be even easier to integrate. > > > > Hazelcast will also work, but Tamás is the special here. > > > > Again: You *must* use an external process to synchronize access. No > > excuses. > > > > Michael > > > > [1] https://github.com/apache/maven/tree/maven-3.8.x-resolver-1.7.x > > [2] > > https://maven.apache.org/resolver/maven-resolver-named-locks/index.html > > [3] > > > > https://maven.apache.org/resolver/maven-resolver-named-locks-redisson/index.html > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > > For additional commands, e-mail: users-h...@maven.apache.org > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
