Yes I did it for Gradle too, sorry. The `/efs-maven-artifacts` is the guilty mount point.
I don't know any quick solutions for the various concerns you all raised, so I'll roll this back tonight. It's good to know that it's not too hard to have a shared FS between these machines; needs better planning though. Thanks, Sanne On 16 January 2018 at 19:41, Steve Ebersole <st...@hibernate.org> wrote: > Did you happen to do the same for Gradle caches? > > Some jobs are failing: > > > * What went wrong: > Could not resolve all dependencies for configuration > ':buildSrc:runtimeClasspath'. >> Timeout waiting to lock artifact cache >> (/efs-maven-artifacts/.gradle/caches/modules-2). It is currently in use by >> another Gradle instance. > Owner PID: 1423 > Our PID: 10249 > Owner Operation: resolve configuration ':classpath' > Our operation: > Lock file: /efs-maven-artifacts/.gradle/caches/modules-2/modules-2.lock > > > > On Mon, Jan 15, 2018 at 5:06 AM Yoann Rodiere <yo...@hibernate.org> wrote: >> >> > We should reconfigure those to not "install" - that's actually a bad >> > habit, legacy from Maven 2 times - people nowadays recommend using >> > "mvn clean verify", especially on CI environments. >> >> I could not agree more, that would be cleaner, but that's not possible. >> And >> believe me, I tried hard. Last time I checked, some of the plugins we use >> with dynamic dependency resolution would ignore the artifacts being built, >> and would always fetch the artifacts from the Maven repos (for SNAPSHOTs, >> they would end up using nightlies). >> I'm not talking about when we use standard maven markup to declare >> dependencies, but when the plugin itself has to fetch dependencies >> "dynamically", which happens when we setup a WildFly server with our own >> modules in particular. See maven-dependency-plugin's "artifactItems" >> configuration. >> >> >> >> On Mon, 15 Jan 2018 at 11:29 Sanne Grinovero <sa...@hibernate.org> wrote: >> >> > On 15 January 2018 at 08:42, Yoann Rodiere <yo...@hibernate.org> wrote: >> > > Thanks Sanne ! >> > > >> > > I have one question... >> > > >> > >> Please never rely on this as "storage": it's just meant as cache and >> > >> we reserve the right to wipe it all out at any time. >> > > >> > > I gather you say that so that we don't try to "release" artifacts into >> > this >> > > cache? But temporary storage for the duration of one build will still >> > > be >> > > safe? >> > > >> > > Because our builds obviously rely on the local repository for >> > > short-term >> > > storage (for the duration of the build). For example the dependencies >> > > are >> > > only checked and downloaded if necessary at the beginning of the >> > > build, >> > and >> > > then are expected to exist in the local repository until the build >> > > stops. >> > > Another example: our WildFly modules are first built and installed in >> > > the >> > > "modules" subproject, and later "fetched" from the local repository in >> > the >> > > "integrationtest/wildfly" subproject. >> > > >> > > If we were to clear the cache during a build, things would probably go >> > > wrong. Worse, if two parallel builds were to install the same >> > > artifacts >> > > (e.g. hibernate-search-engine version 5.9.0-SNAPSHOT), we would run >> > > the >> > risk >> > > of testing the wrong "version" of this artifact in one of the >> > > builds... >> > >> > SNAPSHOT being installed are indeed a problem, e.g the PR testing jobs >> > could conflict with the regular master jobs. >> > We should reconfigure those to not "install" - that's actually a bad >> > habit, legacy from Maven 2 times - people nowadays recommend using >> > "mvn clean verify", especially on CI environments. >> > >> > I agree about the perils of clearing the cache during in-progress builds >> > too. >> > >> > I just meant to warn that we don't have any backup plan in place, and >> > I do plan to just wipe the whole thing occasionally: >> > - when we have any direct need, e.g. currupted downloads >> > - when it gets too large >> > - if it gets too expensive >> > - regularly, just to "practice" that everything works with an empty >> > cache >> > >> > Also our "disaster recovery" plan to rebuild all infrastructure will >> > always assume it's ok to reboot with having this file system empty. >> > >> > Thanks, >> > Sanne >> > >> > > >> > > >> > > On Sun, 14 Jan 2018 at 01:18 Sanne Grinovero <sa...@hibernate.org> >> > wrote: >> > >> >> > >> Hi all, >> > >> >> > >> while the new build machines are fast, some of you pointed out we're >> > >> now spending a relative high amount of time downloading maven >> > >> dependencies, this problem being compounded by the fact we "nuke" >> > >> idle >> > >> slaves shortly after they become idle. >> > >> >> > >> I just spent the day testing a distributed file system, and it's now >> > >> running in "production". >> > >> It's used exclusively to store the Gradle and Maven caches. This is >> > >> stateful and independent from the lifecycle of individual slave >> > >> nodes. >> > >> >> > >> Unfortunately this solution is not viable for Docker images, so while >> > >> I experimented with the idea I backed off from moving the docker >> > >> storage graph to a similar device. Please don't waste time trying >> > >> that >> > >> w/o carefully reading the Docker documentation or talking with me :) >> > >> Also, beyond correctness of storage semantics, it's likely far less >> > >> efficient for Docker. >> > >> >> > >> To learn more about our new cache: >> > >> - >> > >> >> > >> > https://github.com/hibernate/ci.hibernate.org/commit/dc6e0a4bd09fb3ae6347081243b4fb796a219f90 >> > >> - https://docs.aws.amazon.com/efs/latest/ug/how-it-works.html >> > >> >> > >> I'd add that - because of other IO tuning in place - writes might >> > >> appear out of order to other nodes, and conflicts are not handled. >> > >> Shouldn't be a problem since snapshots now have timestamps, but this >> > >> might be something to keep in mind. >> > >> >> > >> N.B. >> > >> Please never rely on this as "storage": it's just meant as cache and >> > >> we reserve the right to wipe it all out at any time. >> > >> >> > >> Thanks, >> > >> Sanne >> > >> _______________________________________________ >> > >> hibernate-dev mailing list >> > >> hibernate-dev@lists.jboss.org >> > >> https://lists.jboss.org/mailman/listinfo/hibernate-dev >> > > >> > > >> > > >> > > -- >> > > Yoann Rodiere >> > > yo...@hibernate.org / yrodi...@redhat.com >> > > Software Engineer >> > > Hibernate NoORM team >> > >> >> >> -- >> Yoann Rodiere >> yo...@hibernate.org / yrodi...@redhat.com >> Software Engineer >> Hibernate NoORM team >> _______________________________________________ >> hibernate-dev mailing list >> hibernate-dev@lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/hibernate-dev _______________________________________________ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev