"Salut" Michael, thanks for the detailled answer. Regarding my Jenkins 'hack' you can totally forget about it: it just ensures no more than one Maven execution uses a given repo at once by locating the local repositories in a folder reflecting the executor (which are kinda Jenkins "threads"). I plan not to use your work right now mostly because it would involve setting up a Redis Database and my 'hack' still works (despite using a lot of disk space and being a bit slow).
If I understand correctly after Tams' message, your work is mostly geared toward running multiple Maven build in parallel and on multiple hosts, all sharing one big "local" repository through the network. Am I correct ? In my case, I have multiple Jenkins workers but never thought about making them collaborate on a shared "local" repo. I'll have to think about it ! Thanks ! François Le lun. 4 oct. 2021 à 21:22, Michael Osipov <micha...@apache.org> a écrit : > Am 2021-10-04 um 14:31 schrieb Francois Marot: > > Hello all, > > > > I would like clarifications on MNG-2802[*] that seems to be solved. If I > > understand correctly it solves the problem where multiple simultaneous > > Maven executions shared the same local repository. > > I have been keeping for years a bit of code in my Jenkinsfiles ensuring > the > > local repos were accessed only by a single jenkins executor at the same > > time, so it seems like I can get rid of this doesn't it ? > > I'd like your input on this because reproducing the bug is difficult so > I'd > > like to be sure before simplifying my Jenkinsfile. > > > > Moreover, does anyone know how this problem is solved technically ? Using > > files lock ? Can anybody explain ? > > > > Thanks you and thanks the Maven team for keeping up the good work at such > > pace ! > > Salut François, > > I don't know what you exactly fiddled in your Jenkinsfile, but there is > no way make it right just with Jenkinsfile foo when you want access > *one* local repo with *multiple* processes you *must* use an external > coordinator process. > Forget also about file locks, they are all advisory, at least on > POSIX-like filesystems. There is no way, or a very hard way to make them > right. We tried and removed them. > > Now, let me give you bit of history: I have integrated/implemented a > fully working multi-process Redisson-based synchronization lock factory > for Maven Resolver 1.6.x which ships with Maven 3.8.x. (It has already > been battle tested for several months by a user on a busy CI server). > Tamás took it to the next level in Resolver 1.7.x and generalized it to > a named lock approach where the Redisson Lock Factory plugs in nicely > with other lock implementations. Almost nine months of work and it works > very good now. > Now here is the problem: Maven Resolver 1.7.x comes only with Maven 4.x. > I have created a branch which reconciles best of breed: Maven 3.8.x and > Maven Resolver 1.7.x. It just works (also verified by this user). I also > ran more than a thousand builds with 8 parallel processes and 4 threads. > It will easily handle thousands of locks in parralel with an one digit > percent overhead. > > I don't know for what documentation you are waiting for, but everything > is there, you can start *now* to leverage it: [2], [3]. With Resolver > 2.0.0 and Maven 4.0.0 it will likely be even easier to integrate. > > Hazelcast will also work, but Tamás is the special here. > > Again: You *must* use an external process to synchronize access. No > excuses. > > Michael > > [1] https://github.com/apache/maven/tree/maven-3.8.x-resolver-1.7.x > [2] > https://maven.apache.org/resolver/maven-resolver-named-locks/index.html > [3] > > https://maven.apache.org/resolver/maven-resolver-named-locks-redisson/index.html > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > For additional commands, e-mail: users-h...@maven.apache.org > >
