You should avoid concurrent builds on a local Repo. Either use a “per worker” 
Repo (all good CI servers should support that, but you can also do it by hand 
command line option) or use a Container build process.

There is also a locking scheme for Maven now, but I would start with Seperate 
repos.

And also clean them up at least once a week or more often, that also helps 
against other corruptions and reproducibility issues.

You always should have a caching mirror to make this less impactful.

Gruss
Bernd


--
http://bernd.eckenfels.net
________________________________
Von: David Karr <davidmichaelk...@gmail.com>
Gesendet: Wednesday, March 23, 2022 11:11:19 PM
An: Maven Users List <users@maven.apache.org>
Betreff: Strategies for automatically fixing local maven repo corruption?

Our enterprise has builds for a couple of hundred services running many
times a day, on a set of Jenkins build nodes, most of which are Maven
builds.The builds use a settings.xml that doesn't override the
"localRepository", so it will use a local repository on the build node that
is used by all builds.

We often run into weird build errors like "Could not find artifact", for
artifacts that I can clearly show are available in our intranet repository.
In every one of those cases, when we inspect the directory in the local
maven repo on that build node, we see that it's "corrupted", in that
perhaps the pom file is there, but the jar file is missing.

We fix these errors by simply removing the directory where the key files
are missing, then rerun the build, which correctly downloads the artifact,
and the build succeeds.  We often have to do this on multiple build nodes,
although I don't have visibility to that (not on ops team), so I don't
really know if the same artifact gets corrupted on multiple build nodes at
the same time. I doubt that.

I wish there was some way that we could either prevent some of these
incidents, or somehow automatically fix them (by removing the corrupted
directory). The latter seems like it's in the realm of possibilities, but
it would have to be constantly scanning the entire repository, looking at
leaf directories that haven't changed in several minutes, and somehow
verifying the "sanity" of the contents from a very limited perspective.

Is there anything we can do to improve this situation?

Reply via email to