Re: [hibernate-dev] ci.hibernate.org : announcing distributed cache for maven artifacts

2018-01-15 Thread Yoann Rodiere 
> We should reconfigure those to not "install" - that's actually a bad
> habit, legacy from Maven 2 times - people nowadays recommend using
> "mvn clean verify", especially on CI environments.

I could not agree more, that would be cleaner, but that's not possible. And
believe me, I tried hard. Last time I checked, some of the plugins we use
with dynamic dependency resolution would ignore the artifacts being built,
and would always fetch the artifacts from the Maven repos (for SNAPSHOTs,
they would end up using nightlies).
I'm not talking about when we use standard maven markup to declare
dependencies, but when the plugin itself has to fetch dependencies
"dynamically", which happens when we setup a WildFly server with our own
modules in particular. See maven-dependency-plugin's "artifactItems"
configuration.



On Mon, 15 Jan 2018 at 11:29 Sanne Grinovero  wrote:

> On 15 January 2018 at 08:42, Yoann Rodiere  wrote:
> > Thanks Sanne !
> >
> > I have one question...
> >
> >> Please never rely on this as "storage": it's just meant as cache and
> >> we reserve the right to wipe it all out at any time.
> >
> > I gather you say that so that we don't try to "release" artifacts into
> this
> > cache? But temporary storage for the duration of one build will still be
> > safe?
> >
> > Because our builds obviously rely on the local repository for short-term
> > storage (for the duration of the build). For example the dependencies are
> > only checked and downloaded if necessary at the beginning of the build,
> and
> > then are expected to exist in the local repository until the build stops.
> > Another example: our WildFly modules are first built and installed in the
> > "modules" subproject, and later "fetched" from the local repository in
> the
> > "integrationtest/wildfly" subproject.
> >
> > If we were to clear the cache during a build, things would probably go
> > wrong. Worse, if two parallel builds were to install the same artifacts
> > (e.g. hibernate-search-engine version 5.9.0-SNAPSHOT), we would run the
> risk
> > of testing the wrong "version" of this artifact in one of the builds...
>
> SNAPSHOT being installed are indeed a problem, e.g the PR testing jobs
> could conflict with the regular master jobs.
> We should reconfigure those to not "install" - that's actually a bad
> habit, legacy from Maven 2 times - people nowadays recommend using
> "mvn clean verify", especially on CI environments.
>
> I agree about the perils of clearing the cache during in-progress builds
> too.
>
> I just meant to warn that we don't have any backup plan in place, and
> I do plan to just wipe the whole thing occasionally:
>  - when we have any direct need, e.g. currupted downloads
>  - when it gets too large
>  - if it gets too expensive
>  - regularly, just to "practice" that everything works with an empty cache
>
> Also our "disaster recovery" plan to rebuild all infrastructure will
> always assume it's ok to reboot with having this file system empty.
>
> Thanks,
> Sanne
>
> >
> >
> > On Sun, 14 Jan 2018 at 01:18 Sanne Grinovero 
> wrote:
> >>
> >> Hi all,
> >>
> >> while the new build machines are fast, some of you pointed out we're
> >> now spending a relative high amount of time downloading maven
> >> dependencies, this problem being compounded by the fact we "nuke" idle
> >> slaves shortly after they become idle.
> >>
> >> I just spent the day testing a distributed file system, and it's now
> >> running in "production".
> >> It's used exclusively to store the Gradle and Maven caches. This is
> >> stateful and independent from the lifecycle of individual slave nodes.
> >>
> >> Unfortunately this solution is not viable for Docker images, so while
> >> I experimented with the idea I backed off from moving the docker
> >> storage graph to a similar device. Please don't waste time trying that
> >> w/o carefully reading the Docker documentation or talking with me :)
> >> Also, beyond correctness of storage semantics, it's likely far less
> >> efficient for Docker.
> >>
> >> To learn more about our new cache:
> >>  -
> >>
> https://github.com/hibernate/ci.hibernate.org/commit/dc6e0a4bd09fb3ae6347081243b4fb796a219f90
> >>  - https://docs.aws.amazon.com/efs/latest/ug/how-it-works.html
> >>
> >> I'd add that - because of other IO tuning in place - writes might
> >> appear out of order to other nodes, and conflicts are not handled.
> >> Shouldn't be a problem since snapshots now have timestamps, but this
> >> might be something to keep in mind.
> >>
> >> N.B.
> >> Please never rely on this as "storage": it's just meant as cache and
> >> we reserve the right to wipe it all out at any time.
> >>
> >> Thanks,
> >> Sanne
> >> ___
> >> hibernate-dev mailing list
> >> hibernate-dev@lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/hibernate-dev
> >
> >
> >
> > --
> > Yoann Rodiere
> > yo...@hibernate.org /

Re: [hibernate-dev] ci.hibernate.org : announcing distributed cache for maven artifacts

2018-01-15 Thread Sanne Grinovero 
On 15 January 2018 at 08:42, Yoann Rodiere  wrote:
> Thanks Sanne !
>
> I have one question...
>
>> Please never rely on this as "storage": it's just meant as cache and
>> we reserve the right to wipe it all out at any time.
>
> I gather you say that so that we don't try to "release" artifacts into this
> cache? But temporary storage for the duration of one build will still be
> safe?
>
> Because our builds obviously rely on the local repository for short-term
> storage (for the duration of the build). For example the dependencies are
> only checked and downloaded if necessary at the beginning of the build, and
> then are expected to exist in the local repository until the build stops.
> Another example: our WildFly modules are first built and installed in the
> "modules" subproject, and later "fetched" from the local repository in the
> "integrationtest/wildfly" subproject.
>
> If we were to clear the cache during a build, things would probably go
> wrong. Worse, if two parallel builds were to install the same artifacts
> (e.g. hibernate-search-engine version 5.9.0-SNAPSHOT), we would run the risk
> of testing the wrong "version" of this artifact in one of the builds...

SNAPSHOT being installed are indeed a problem, e.g the PR testing jobs
could conflict with the regular master jobs.
We should reconfigure those to not "install" - that's actually a bad
habit, legacy from Maven 2 times - people nowadays recommend using
"mvn clean verify", especially on CI environments.

I agree about the perils of clearing the cache during in-progress builds too.

I just meant to warn that we don't have any backup plan in place, and
I do plan to just wipe the whole thing occasionally:
 - when we have any direct need, e.g. currupted downloads
 - when it gets too large
 - if it gets too expensive
 - regularly, just to "practice" that everything works with an empty cache

Also our "disaster recovery" plan to rebuild all infrastructure will
always assume it's ok to reboot with having this file system empty.

Thanks,
Sanne

>
>
> On Sun, 14 Jan 2018 at 01:18 Sanne Grinovero  wrote:
>>
>> Hi all,
>>
>> while the new build machines are fast, some of you pointed out we're
>> now spending a relative high amount of time downloading maven
>> dependencies, this problem being compounded by the fact we "nuke" idle
>> slaves shortly after they become idle.
>>
>> I just spent the day testing a distributed file system, and it's now
>> running in "production".
>> It's used exclusively to store the Gradle and Maven caches. This is
>> stateful and independent from the lifecycle of individual slave nodes.
>>
>> Unfortunately this solution is not viable for Docker images, so while
>> I experimented with the idea I backed off from moving the docker
>> storage graph to a similar device. Please don't waste time trying that
>> w/o carefully reading the Docker documentation or talking with me :)
>> Also, beyond correctness of storage semantics, it's likely far less
>> efficient for Docker.
>>
>> To learn more about our new cache:
>>  -
>> https://github.com/hibernate/ci.hibernate.org/commit/dc6e0a4bd09fb3ae6347081243b4fb796a219f90
>>  - https://docs.aws.amazon.com/efs/latest/ug/how-it-works.html
>>
>> I'd add that - because of other IO tuning in place - writes might
>> appear out of order to other nodes, and conflicts are not handled.
>> Shouldn't be a problem since snapshots now have timestamps, but this
>> might be something to keep in mind.
>>
>> N.B.
>> Please never rely on this as "storage": it's just meant as cache and
>> we reserve the right to wipe it all out at any time.
>>
>> Thanks,
>> Sanne
>> ___
>> hibernate-dev mailing list
>> hibernate-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>
>
>
> --
> Yoann Rodiere
> yo...@hibernate.org / yrodi...@redhat.com
> Software Engineer
> Hibernate NoORM team
___
hibernate-dev mailing list
hibernate-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev

Re: [hibernate-dev] ci.hibernate.org : announcing distributed cache for maven artifacts

2018-01-15 Thread Yoann Rodiere 
Thanks Sanne !

I have one question...

> Please never rely on this as "storage": it's just meant as cache and
> we reserve the right to wipe it all out at any time.

I gather you say that so that we don't try to "release" artifacts into this
cache? But temporary storage for the duration of one build will still be
safe?

Because our builds obviously rely on the local repository for short-term
storage (for the duration of the build). For example the dependencies are
only checked and downloaded if necessary at the beginning of the build, and
then are expected to exist in the local repository until the build stops.
Another example: our WildFly modules are first built and installed in the
"modules" subproject, and later "fetched" from the local repository in the
"integrationtest/wildfly" subproject.

If we were to clear the cache during a build, things would probably go
wrong. Worse, if two parallel builds were to install the same artifacts
(e.g. hibernate-search-engine version 5.9.0-SNAPSHOT), we would run the
risk of testing the wrong "version" of this artifact in one of the builds...


On Sun, 14 Jan 2018 at 01:18 Sanne Grinovero  wrote:

> Hi all,
>
> while the new build machines are fast, some of you pointed out we're
> now spending a relative high amount of time downloading maven
> dependencies, this problem being compounded by the fact we "nuke" idle
> slaves shortly after they become idle.
>
> I just spent the day testing a distributed file system, and it's now
> running in "production".
> It's used exclusively to store the Gradle and Maven caches. This is
> stateful and independent from the lifecycle of individual slave nodes.
>
> Unfortunately this solution is not viable for Docker images, so while
> I experimented with the idea I backed off from moving the docker
> storage graph to a similar device. Please don't waste time trying that
> w/o carefully reading the Docker documentation or talking with me :)
> Also, beyond correctness of storage semantics, it's likely far less
> efficient for Docker.
>
> To learn more about our new cache:
>  -
> https://github.com/hibernate/ci.hibernate.org/commit/dc6e0a4bd09fb3ae6347081243b4fb796a219f90
>  - https://docs.aws.amazon.com/efs/latest/ug/how-it-works.html
>
> I'd add that - because of other IO tuning in place - writes might
> appear out of order to other nodes, and conflicts are not handled.
> Shouldn't be a problem since snapshots now have timestamps, but this
> might be something to keep in mind.
>
> N.B.
> Please never rely on this as "storage": it's just meant as cache and
> we reserve the right to wipe it all out at any time.
>
> Thanks,
> Sanne
> ___
> hibernate-dev mailing list
> hibernate-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>


-- 
Yoann Rodiere
yo...@hibernate.org / yrodi...@redhat.com
Software Engineer
Hibernate NoORM team
___
hibernate-dev mailing list
hibernate-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev