Re: Continuous Delivery and Maven

[jhumble]

Hi Brian

It just seems like the rev ID is really useful here for identifying
> reproducible builds without creating releases every time, does it fit with
> your ideas?  If so, a hypothetical repository manager plugin could be
> maintaining information about snapshot dependencies based on SCM rev ID,
> thus allowing for reproducibility without modifying Maven or existing
> snapshot mechanics.  Such a plugin might be able to generate a POM that has
> the extra rev ID metadata that the repo manager would recognize, allowing
> for existing SNAPSHOT-style identifiers to keep working for developer
> desktops (avoiding SCM thrash), but also providing reproducibility through
> synthetic POMs.


I think this is a great idea. If the pom for each snapshot contains enough
metadata about each of its upstream snapshots to be able to reproduce an
identical binary, we're good. Since I believe Maven already stores the md5
for each snapshot, this is verifiable.

Perhaps including in the POM for a snapshot the SCM URIs that were used to
create it and the version id (for SCMs that support atomic commits - I can't
see a happy way to do this for CVS, for example, without using tags) and the
same information nested for any of their upstream snapshots?

And then for extra points a command that could look at this pom, regenerate
the whole thing from scratch, and verify it against the md5 for the original
binary.

Jez.

-- 
Jez Humble
Co-author, *Continuous Delivery *
http://continuousdelivery.com/
http://jezhumble.net/

-- 
View this message in context: 
http://maven.40175.n5.nabble.com/Continuous-Delivery-and-Maven-tp3245370p3254548.html
Sent from the Maven - Users mailing list archive at Nabble.com.

Re: Continuous Delivery and Maven

[jhumble]

Hi Curtis

I'm the first to admit I'm no Maven expert.

So please let me just confirm. Let's assume I am working on project A which
depends on projects B and C. For the sake of argument, let's say that the
source code for A, B and C have separate roots in SVN, and are usually
checked out independently. The CI system builds A at version 50 using
snapshot dependencies on B and C, which are fetched from a central snapshot
repository.

Later, there are multiple updates to projects B and C which result in new
versions of them becoming available.

Say I now check out the source code to project A to version 50, because I
want to try and debug some issue that cropped up in a performance testing
environment, and I run a maven build. Will that use the latest versions of
the snapshots from the repo, or the versions that were originally fetched
when it was run on the CI system?

Can I even find out exactly which versions from svn the snapshots of B and C
came from that were used by the CI system to generate the original build of
A?

Thanks,

Jez.

On 7 November 2010 20:10, Yanko, Curtis [via Maven] <
ml-node+3254520-1036569885-143...@n5.nabble.com
> wrote:

> Very interesting discussion. With all due respect to Mr. Humble, and I
> am a big fan of CD, I am going to venture to say that you don't
> understand Maven very well. As a thought experiment, you are correct in
> saying that a build based on snapshots is not reproducible. As a more
> practical matter however, I feel it is.
>
> Dependencies come in two flavors, our and theirs (internal and 3rd
> party). If, all of *our* dependencies are SNAPSHOT (we're doing the
> developing) and all of *theirs* are 'versioned' then the build is in
> fact reproducible assuming you build everything from a particular repo
> version even with the default auto-update setting (in fact, it's
> required).
> 
>
> Curt Yanko | Continuous Integration Services | UnitedHealth Group IT
> Making IT Happen, one build at a time, 600 times a day
>
> -Original Message-
> From: jhumble [mailto:[hidden 
> email]]
>
> Sent: Sunday, November 07, 2010 11:15 AM
> To: [hidden email] 
> Subject: RE: Continuous Delivery and Maven
>
>
> Hey Todd
>
> The whole point of continuous delivery is that every check-in creates a
> potential release candidate.
>
> When you're doing continuous deployment, you could be releasing multiple
> times a day, so you don't bother cutting branches or tagging or any of
> that stuff because of the overhead. I'd rather not get into the
> justification for this process on this thread - but I wrote a book on it
> if you're interested:
> http://www.amazon.com/gp/product/0321601912 and many other people have
> blogged about it.
>
> You're right that creating a concrete release for each commit could
> potentially use up a lot of space - but that's fine, you can just delete
> the older ones. What this *does* mean in turn though is that it is
> essential to be able to recreate any given build given the version in
> source control it came from, and this is where Maven falls down.
> Snapshots just aren't suitable because they aren't reproducible: what
> the snapshot looks like depends not only on what versions of the
> dependencies are available at the time the snapshot is created, but also
> what Maven's configuration and plug-ins happen to be at the time you run
> it (assuming Maven is configured to auto-update - the default). I can't
> revert back to a particular revision in version control, run maven, and
> be sure that the artifact it generates is identical to the one it
> created when the commit was initially triggered.
>
> Ideally what I'd like is for Maven to explicitly support the continuous
> delivery model and provide snapshots that are reproducible. Failing
> that, a guide to configuring Maven so that its binaries are reproducible
> (for example by switching off auto-update, and having sufficient
> metadata stored in pom files and Maven's artifacts repository to know
> what the state of each of the dependencies was at any given time.
> --
> View this message in context:
> http://maven.40175.n5.nabble.com/Continuous-Delivery-and-Maven-tp3245370
> p3254090.html
> Sent from the Maven - Users mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: [hidden 
> email]
> For additional commands, e-mail: [hidden 
> email]
>
>
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or entity
> to which it is addressed. If the reader of this e-mail is not the intended
> recipient or his or her authoriz

RE: Continuous Delivery and Maven

[Yanko, Curtis]

Very interesting discussion. With all due respect to Mr. Humble, and I
am a big fan of CD, I am going to venture to say that you don't
understand Maven very well. As a thought experiment, you are correct in
saying that a build based on snapshots is not reproducible. As a more
practical matter however, I feel it is.

Dependencies come in two flavors, our and theirs (internal and 3rd
party). If, all of *our* dependencies are SNAPSHOT (we're doing the
developing) and all of *theirs* are 'versioned' then the build is in
fact reproducible assuming you build everything from a particular repo
version even with the default auto-update setting (in fact, it's
required).


Curt Yanko | Continuous Integration Services | UnitedHealth Group IT 
Making IT Happen, one build at a time, 600 times a day

-Original Message-
From: jhumble [mailto:j...@jezhumble.net] 
Sent: Sunday, November 07, 2010 11:15 AM
To: users@maven.apache.org
Subject: RE: Continuous Delivery and Maven


Hey Todd

The whole point of continuous delivery is that every check-in creates a
potential release candidate.

When you're doing continuous deployment, you could be releasing multiple
times a day, so you don't bother cutting branches or tagging or any of
that stuff because of the overhead. I'd rather not get into the
justification for this process on this thread - but I wrote a book on it
if you're interested:
http://www.amazon.com/gp/product/0321601912 and many other people have
blogged about it.

You're right that creating a concrete release for each commit could
potentially use up a lot of space - but that's fine, you can just delete
the older ones. What this *does* mean in turn though is that it is
essential to be able to recreate any given build given the version in
source control it came from, and this is where Maven falls down.
Snapshots just aren't suitable because they aren't reproducible: what
the snapshot looks like depends not only on what versions of the
dependencies are available at the time the snapshot is created, but also
what Maven's configuration and plug-ins happen to be at the time you run
it (assuming Maven is configured to auto-update - the default). I can't
revert back to a particular revision in version control, run maven, and
be sure that the artifact it generates is identical to the one it
created when the commit was initially triggered.

Ideally what I'd like is for Maven to explicitly support the continuous
delivery model and provide snapshots that are reproducible. Failing
that, a guide to configuring Maven so that its binaries are reproducible
(for example by switching off auto-update, and having sufficient
metadata stored in pom files and Maven's artifacts repository to know
what the state of each of the dependencies was at any given time.
--
View this message in context:
http://maven.40175.n5.nabble.com/Continuous-Delivery-and-Maven-tp3245370
p3254090.html
Sent from the Maven - Users mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org


This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.


-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: Continuous Delivery and Maven

[Brian Topping]

On Nov 7, 2010, at 1:29 PM, jhumble wrote:

> One possibility to get repeatable builds without filling up an artifacts
> repository too fast could be to make Maven store the fully qualified pom
> files in the artifacts repo and an md5 of the binary but not necessarily the
> actual binary. I know artifacts repos already store some of this
> information.
> 
> That way you could make sure sufficient metadata is publicly available such
> that they can be reproduced, without using up loads of disk space. You could
> also happily delete older binaries, safe in the knowledge that people could
> reproduce them from the metadata in the artifacts repo.

One of the things I like about snapshots is it just simply means "latest".  
Though the thing about timestamped snapshots is that they aren't guaranteed to 
exist (the repository is not typically assumed to be reliable), and they aren't 
100% reproducible (the timestamp offset includes the time it took to build the 
artifact and all the artifacts before it, meaning there's no way to know 
exactly what point in time the build came from).  Even if one could find the 
correct timestamp to check out from to get the same binary, whatever subsystem 
creates the timestamp on upload (wagon?) probably doesn't like being told what 
to call the snapshot.  

It follows the only way to get a reproducible build is either to tag the 
original sources or to know the SCM revision id.  The revision id is a natural 
tag that is automatically generated, and does not clutter the named tag space 
with thousands of tags that have no organizational meaning.  On my CI builds, 
the first thing that happens is grabbing the revision ID from SVN, and that's 
put in a properties file that can be used when the UI is generated.  Where the 
version number helps users identify the general features to expect of the 
current software, the revision ID is great for filing issues so devs don't have 
to guess at what sources have the issue.  

When the sources all come from the same SCM repository tree, the rev ID makes 
it a cinch to reproduce the build.  Of course, a better solution can span 
multiple trees and is reproducible.

It just seems like the rev ID is really useful here for identifying 
reproducible builds without creating releases every time, does it fit with your 
ideas?  If so, a hypothetical repository manager plugin could be maintaining 
information about snapshot dependencies based on SCM rev ID, thus allowing for 
reproducibility without modifying Maven or existing snapshot mechanics.  Such a 
plugin might be able to generate a POM that has the extra rev ID metadata that 
the repo manager would recognize, allowing for existing SNAPSHOT-style 
identifiers to keep working for developer desktops (avoiding SCM thrash), but 
also providing reproducibility through synthetic POMs.



-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: Continuous Delivery and Maven

[jhumble]

On 7 November 2010 10:02, Brian Topping wrote:


> Does your book discuss ways to get around these issues?


No, it's a patterns / practices book so we don't go into a lot of detail on
the tools because that information tends to go out of date fast. We discuss
Maven a bit at the end of Chapter 13 ("Managing components and
dependencies"), pp375-378

The advice we give is this: "have your CI server produce canonical versions
of each dependency, using the build label as part of the artifact’s version
number, and store these in your organization’s central artifact repoistory.
You can then use Maven’s version quantifiers in your pom file to specify a
range of acceptable versions to use. If you really need to do some
exploratory work on your local machine, you can always edit your pom
definition to temporarily enable snapshots"

This isn't really ideal, partly for the reason you specify: this leads to a
lot of thrashing with the POM file. So I'm interested to see what could be
done to make Maven work better with the CD paradigm, for instance, stug23's
idea:

we have a so-called base-pom, which all projects inherit from, that locks
> down all of the Maven plugin versions so that the build is repeatable at a
> later time.


I think you identify the problem exactly right:

If I understand the paradigm, it's not that developers would want to reject
> the latest version of any dependencies, just that the snapshot builds should
> be reproducible


In fact, I'd go further and say we want to encourage people taking the
latest version of dependencies - otherwise you're not doing continuous
integration. But of course we want all of those dependencies in the artifact
repo, not built from scratch on the developer machines, because then it
means firstly wasted time and secondly that everybody is working with
potentially different binaries.

One possibility to get repeatable builds without filling up an artifacts
repository too fast could be to make Maven store the fully qualified pom
files in the artifacts repo and an md5 of the binary but not necessarily the
actual binary. I know artifacts repos already store some of this
information.

That way you could make sure sufficient metadata is publicly available such
that they can be reproduced, without using up loads of disk space. You could
also happily delete older binaries, safe in the knowledge that people could
reproduce them from the metadata in the artifacts repo.

As you can probably tell I'm no Maven maven, but I do want to help in
whatever way I can to make it work well with a continuous delivery process.

Jez.

-- 
Jez Humble
Co-author, *Continuous Delivery *
http://continuousdelivery.com/
http://jezhumble.net/

-- 
View this message in context: 
http://maven.40175.n5.nabble.com/Continuous-Delivery-and-Maven-tp3245370p3254183.html
Sent from the Maven - Users mailing list archive at Nabble.com.

Re: Problems using maven-dependencies-plugin

[Wayne Fay]

> I really need to have maven copy a set of "external" scoped dependencies,
> and I´d hate to have to make a "company branch" of a maven plugin

Perhaps the Flexmojos project would be willing to take ownership of
the altered Maven plugin... since they are "causing" the problem in
the first place.

Wayne

-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: Continuous Delivery and Maven

[Brian Topping]

On Nov 7, 2010, at 11:14 AM, jhumble wrote:

> Ideally what I'd like is for Maven to explicitly support the continuous
> delivery model and provide snapshots that are reproducible.

Snapshots can be reproducible if developers set dependencies on the timestamped 
name of the deployed version.  I remember there being reasons that these 
strategies are discouraged (should be in the list archives), but those reasons 
may not apply here.

If I understand the paradigm, it's not that developers would want to reject the 
latest version of any dependencies, just that the snapshot builds should be 
reproducible.  Since the POM is the source of dependency resolution for any 
Maven build, it seems like the release plugin in use would have to update the 
project POMs to the currently resolved name of the actual dependency and check 
them in before the tag.  

But if the release checks in a POM and this is happening over and over for 
every checkin (a one line change in a heavily connected project could easily 
cause several POMs to be updated), suddenly dependent versions need to have 
their POMs changed, checked in, and new version propagated iteratively.  On 
developer desktops, all these new POMs would need to be kept up to date 
constantly to avoid text conflicts.  Maybe a VFS SCM could get around this, 
but ick!  The tooling is suddenly very heavyweight.

Does your book discuss ways to get around these issues?
-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

RE: Continuous Delivery and Maven

[Michael Hüttermann]

Hello fellows,

perhaps it is interesting for you that the book "Agile ALM" [1],
illustrates strategies and tools (especially Maven). I'm also covering
aspects that are discussed here in this thread.

[1] http://huettermann.net/alm


Cheers
Michael

--
-Java Champion-
mich...@huettermann.net
http://huettermann.net
http://twitter.com/huettermann

-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

RE: Continuous Delivery and Maven

[jhumble]

Hey Todd

The whole point of continuous delivery is that every check-in creates a
potential release candidate.

When you're doing continuous deployment, you could be releasing multiple
times a day, so you don't bother cutting branches or tagging or any of that
stuff because of the overhead. I'd rather not get into the justification for
this process on this thread - but I wrote a book on it if you're interested:
http://www.amazon.com/gp/product/0321601912 and many other people have
blogged about it.

You're right that creating a concrete release for each commit could
potentially use up a lot of space - but that's fine, you can just delete the
older ones. What this *does* mean in turn though is that it is essential to
be able to recreate any given build given the version in source control it
came from, and this is where Maven falls down. Snapshots just aren't
suitable because they aren't reproducible: what the snapshot looks like
depends not only on what versions of the dependencies are available at the
time the snapshot is created, but also what Maven's configuration and
plug-ins happen to be at the time you run it (assuming Maven is configured
to auto-update - the default). I can't revert back to a particular revision
in version control, run maven, and be sure that the artifact it generates is
identical to the one it created when the commit was initially triggered.

Ideally what I'd like is for Maven to explicitly support the continuous
delivery model and provide snapshots that are reproducible. Failing that, a
guide to configuring Maven so that its binaries are reproducible (for
example by switching off auto-update, and having sufficient metadata stored
in pom files and Maven's artifacts repository to know what the state of each
of the dependencies was at any given time.
-- 
View this message in context: 
http://maven.40175.n5.nabble.com/Continuous-Delivery-and-Maven-tp3245370p3254090.html
Sent from the Maven - Users mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: Maven-versions-plugin - How to ignore a particular "newer" version?

[Stephen Connolly]

http://mojo.codehaus.org/versions-maven-plugin/update-properties-mojo.html#properties

On 6 Nov 2010 23:22, "Hilco Wijbenga"  wrote:

On 6 November 2010 15:25, Stephen Connolly

 wrote:
> You can define a range of updates to consider... On secon...
As in version ranges? Something like [1.0b3, 10]? Where and how would
I do that? I don't see anything about it at
http://mojo.codehaus.org/versions-maven-plugin (property related or
otherwise).

I suppose these version ranges would help but what if I wanted to
exclude a particular newer version that doesn't work for me or is
broken? I think it would be useful to be able to (also) exclude
particular versions.


-
To unsubscribe, e-mail: users...