RE: Who should use SNAPSHOT when? RE: The Future of the Release Process.
Wendy Smoak wrote: On 12/22/06, Dan Fabulich [EMAIL PROTECTED] wrote: From: Kenney Westerhof [mailto:[EMAIL PROTECTED] If I understand correctly, the problem is that a 'staged' release still contains a SNAPSHOT keyword in the metadata/filename? Yes, that's the problem. I would agree, but that's not how I understand staging to work at all. The Apache projects I work on like to vote on the *exact* artifacts that will be released to the public, so the staged release must not have a -SNAPSHOT qualifier. I realize now that I agreed with Kenney too soon. The problem is, really, the existence of a -SNAPSHOT qualifier at all, which, in turn, forces us to jump through hoops at the last second in the release process in order to remove that qualifier. (-SNAPSHOT has some advantages, but for many organizations, the disadvantages are considerably more significant.) The real point here is that the staging process (which creates non-SNAPSHOT binaries that have not yet been released) should be something that we could use throughout the development cycle, not some special last-minute thing that provides special last-minute binaries. The way to do that is to provide build numbers on non-SNAPSHOT releases, allowing us to bless a release binary without modifying it (for example, by moving it from one repository to another). -Dan ___ Notice: This email message, together with any attachments, may contain information of BEA Systems, Inc., its subsidiaries and affiliated entities, that may be confidential, proprietary, copyrighted and/or legally privileged, and is intended solely for the use of the individual or entity named in this message. If you are not the intended recipient, and have received this message in error, please immediately return this by email and then delete it. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Who should use SNAPSHOT when? RE: The Future of the Release Process.
On 1/2/07, Dan Fabulich [EMAIL PROTECTED] wrote: Wendy Smoak wrote: On 12/22/06, Dan Fabulich [EMAIL PROTECTED] wrote: From: Kenney Westerhof [mailto:[EMAIL PROTECTED] If I understand correctly, the problem is that a 'staged' release still contains a SNAPSHOT keyword in the metadata/filename? Yes, that's the problem. I would agree, but that's not how I understand staging to work at all. The Apache projects I work on like to vote on the *exact* artifacts that will be released to the public, so the staged release must not have a -SNAPSHOT qualifier. I realize now that I agreed with Kenney too soon. The problem is, really, the existence of a -SNAPSHOT qualifier at all, which, in turn, forces us to jump through hoops at the last second in the release process in order to remove that qualifier. (-SNAPSHOT has some advantages, but for many organizations, the disadvantages are considerably more significant.) The real point here is that the staging process (which creates non-SNAPSHOT binaries that have not yet been released) should be something that we could use throughout the development cycle, not some special last-minute thing that provides special last-minute binaries. The way to do that is to provide build numbers on non-SNAPSHOT releases, allowing us to bless a release binary without modifying it (for example, by moving it from one repository to another). That's essentially what Eclipse does... While developing to a 3.3 release, each bundle in every build gets a version qualifier (e.g. 3.3.0.v20061116) which is incremented if the bundle changes. The eventual 3.3 release will then just be a collection of bundles with the latest qualifier for each. As an example, these are some off the bundles in the 3.2.1 release. Note that some bundles that never changed still have the 3.2.0.v version. org.eclipse.core.boot_3.1.100.v20060603.jar org.eclipse.core.commands_3.2.0.I20060605-1400.jar org.eclipse.core.contenttype_3.2.0.v20060603.jar org.eclipse.core.expressions_3.2.1.r321_v20060721.jar org.eclipse.core.filebuffers_3.2.1.r321_v20060721.jar org.eclipse.core.filesystem.win32.x86_1.0.0.v20060603.jar org.eclipse.core.filesystem_1.0.0.v20060603.jar org.eclipse.core.jobs_3.2.0.v20060603.jar org.eclipse.core.resources.compatibility_3.2.0.v20060603.jar org.eclipse.core.resources.win32_3.2.0.v20060603.jar org.eclipse.core.resources_3.2.1.R32x_v20060914.jar org.eclipse.core.runtime.compatibility.auth_3.2.0.v20060601.jar org.eclipse.core.runtime.compatibility.registry_3.2.1.R32x_v20060907 org.eclipse.core.runtime.compatibility_3.1.100.v20060603.jar org.eclipse.core.runtime_3.2.0.v20060603.jar org.eclipse.core.variables_3.1.100.v20060605.jar More info at http://wiki.eclipse.org/index.php/Version_Numbering Tom -Dan ___ Notice: This email message, together with any attachments, may contain information of BEA Systems, Inc., its subsidiaries and affiliated entities, that may be confidential, proprietary, copyrighted and/or legally privileged, and is intended solely for the use of the individual or entity named in this message. If you are not the intended recipient, and have received this message in error, please immediately return this by email and then delete it. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Who should use SNAPSHOT when? RE: The Future of the Release Process.
On 12/23/06, Craig McClanahan [EMAIL PROTECTED] wrote: On 12/22/06, Wendy Smoak [EMAIL PROTECTED] wrote: On 12/22/06, Dan Fabulich [EMAIL PROTECTED] wrote: From: Kenney Westerhof [mailto:[EMAIL PROTECTED] If I understand correctly, the problem is that a 'staged' release still contains a SNAPSHOT keyword in the metadata/filename? Yes, that's the problem. I would agree, but that's not how I understand staging to work at all. The Apache projects I work on like to vote on the *exact* artifacts that will be released to the public, so the staged release must not have a -SNAPSHOT qualifier. Agreed. When I vote on a release, I'm always saying show me the bits. In this scenario, it would be nice for the release plugin to have an option to copy approved artifacts from the staging area to the release area (along with corresponding signatures and metadata updates) *without* any modification to the artifacts themselves. I wrote the repositorytools plugin in mojo-sandbox for this reason. It has goals to copy a specific artifact (including signatures) or an entire remote repository to another repository, while merging the necessary repository metadata. This is still work-in-progress although Jason already tested it (I think it was on a Geronimo release ). Tom -- Wendy Craig - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Who should use SNAPSHOT when? RE: The Future of the Release Process.
On 22 Dec 06, at 9:45 PM 22 Dec 06, Wendy Smoak wrote: On 12/22/06, Dan Fabulich [EMAIL PROTECTED] wrote: From: Kenney Westerhof [mailto:[EMAIL PROTECTED] If I understand correctly, the problem is that a 'staged' release still contains a SNAPSHOT keyword in the metadata/filename? Yes, that's the problem. I would agree, but that's not how I understand staging to work at all. The staging directory would container artifacts as they would be released in the wild. No SNAPSHOTs, and in the form they would be merged into the central repository. Jason. The Apache projects I work on like to vote on the *exact* artifacts that will be released to the public, so the staged release must not have a -SNAPSHOT qualifier. -- Wendy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Who should use SNAPSHOT when? RE: The Future of the Release Process.
On 22 Dec 06, at 10:03 PM 22 Dec 06, Craig McClanahan wrote: On 12/22/06, Wendy Smoak [EMAIL PROTECTED] wrote: On 12/22/06, Dan Fabulich [EMAIL PROTECTED] wrote: From: Kenney Westerhof [mailto:[EMAIL PROTECTED] If I understand correctly, the problem is that a 'staged' release still contains a SNAPSHOT keyword in the metadata/filename? Yes, that's the problem. I would agree, but that's not how I understand staging to work at all. The Apache projects I work on like to vote on the *exact* artifacts that will be released to the public, so the staged release must not have a -SNAPSHOT qualifier. Agreed. When I vote on a release, I'm always saying show me the bits. In this scenario, it would be nice for the release plugin to have an option to copy approved artifacts from the staging area to the release area (along with corresponding signatures and metadata updates) *without* any modification to the artifacts themselves. Yes, that's what we have just tried with a Geronimo release where the actual release was staged, Geronimo folks votes and then I merged the artifacts into the central syncing directory on Apache using Tom's new repository copier which takes into account merging the necessary metadata. Jason. -- Wendy Craig - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Who should use SNAPSHOT when? RE: The Future of the Release Process.
OK, I've finally written up the Compass way of doing things. I've got it as a blog article for easier reading (and HTML formatting), but I'm also including the text below. http://darkforge.blogspot.com/2006/12/compass-as-compared-with-mavens.ht ml In this article, I'll describe some of the differences between Maven 2.x and the Compass internal home-grown system we use at work. I'll first describe our repository layout, then describe our component descriptor file, and finally I'll summarize some of the advantages and disadvantages of using the different systems and suggest future work. The Compass system was designed with Maven 1.x in mind. The original developers had said, roughly: You know, Maven's got the right idea, but this really hasn't been implemented the way we'd want it. We should rewrite it ourselves from scratch. REPOSITORY LAYOUT Like Maven, Compass has one or more remote repositories containing official built artifacts, (or components, as we call them,) as well as a local repo on each developer's machine which caches artifacts from the remote repo and contains locally built artifacts. Where Maven and Compass substantially diverge is in how artifacts are stored in the repository. While Compass doesn't have a notion of groupId, our remote repository is divided up into sections, like this: thirdparty/ log4j/ junit/ firstparty/ RECENT/ foo/ bar/ INTERNAL/ foo/ bar/ RELEASE/ foo/ bar/ [NOTE: This isn't exactly how it looks, but it's close enough.] Within a given section, you find a flat list of components. In this example, foo and bar are buildable components that we've created; log4j and junit are, naturally, components built by other people. RECENT contains only freshly built components. INTERNAL contains components that have been blessed by some human being, and are intended for internal consumption. RELEASE contains released components and products. In practice, there are 914 components in RECENT and 671 components in INTERNAL. Within a given component directory, you'll find a number of subdirectories, which define the version of the component. Thirdparty versions may have any arbitrary strings for their names (e.g. 3.8.1 1.0beta3 deerpark). However, firstparty versions are strictly defined: they are simply the P4 Changelist number of the product at the time it was built. (A quick note about changelist numbers as opposed to revision numbers. Most people are familiar with the distinction between CVS revision numbers and SVN revision numbers: CVS revision numbers are per file whereas SVN revision numbers are global to the repository. P4 changelist numbers are like SVN revision numbers. [Also note that you can calculate something like an SVN revision number in CVS, simply by noting the timestamp of the most recent check-in.]) So, within the foo directory in RECENT, you'll see this: foo/ 123456/ foo.jar 123457/ foo.jar 123458/ foo.jar @LATEST - 123458 That's three numbered directories with a LATEST symlink, pointing to the most recent build in that directory. The first thing to note about this system is that if you build 123458 and then rebuild 123458, it will replace the old 123458 directory. The second thing to note is that if you change foo at all, it will get a new changelist/revision number, and so it will get a new subdirectory under foo once automatically built. The three sections within the firstparty directory (RECENT, INTERNAL, RELEASE) are called release levels, and we have a process about how components move into each release level. foo and bar are automatically built every night and deployed into RECENT; if there are more than three builds in RECENT, we automatically delete the oldest build. If somebody thinks that a build of foo is good enough to keep around, they promote that build into INTERNAL by simply copying the numbered changelist directory into INTERNAL. Once we think it's good enough to release, we can promote that INTERNAL build into RELEASE by copying it there. There is no tool, nor any need for a tool, to rebuild for release or make even the slightest changes to the released binaries. Especially note that we don't put any of this information in the filename of the jar. It's called foo.jar whether it's in RECENT, INTERNAL, or RELEASE. We do burn the changelist number of foo.jar into its META-INF/manifest.mf at build-time... that information remains constant whether foo.jar is copied to INTERNAL or RELEASE. COMPONENT DESCRIPTOR FILE Compass has a file that looks a lot like the Maven POM XML file... our file is called component.xml. component.xml defines a list of dependency elements. Here's an example component.xml file: component namefoo/name release6.1.0/release branchmain/branch depends dependency type='internal'
RE: Who should use SNAPSHOT when? RE: The Future of the Release Process.
Comments inline... -Original Message- From: Kenney Westerhof [mailto:[EMAIL PROTECTED] Sent: Thursday, December 14, 2006 5:07 AM To: Maven Developers List Subject: Re: Who should use SNAPSHOT when? RE: The Future of the Release Process. Hi, If I understand correctly, the problem is that a 'staged' release still contains a SNAPSHOT keyword in the metadata/filename? Yes, that's the problem. [Even the filename is a problem when you know the files will be installed in the context of a larger assembly installed on end-users' desktops. More generally, I think, most professional closed source ISVs have it as a requirement that you can't go around renaming libraries right before you release your software; if they don't make this a requirement, IMO they should.] Also, how would you see snapshot 'releases' without a snapshot keyword? If there's no indicator (timestamp?) in the filename, you'll overwrite the previous deployed versions, which is bad. You keep them in separate directories. http://darkforge.blogspot.com/2006/12/compass-as-compared-with-mavens.ht ml The most important change I suggest here is to modify the way we deploy builds to include a build number directory in the repository layout; every deployed artifact (even released artifact) would be in its own build number directory, so they would never clobber each other. The official release vote would be to vote on promoting a particular build number to a release. Personally I'm pro release-candidate marking of artifacts. I don't have an opinion about release-candidate marking in *general*, but it should be at least possible to have a release process in Maven that doesn't mark release candidates at all (even in the filename). What if maven understood the difference between the version in the pom and the version in the filename? Whatever comes of this discussion, I hope it emerges that the point you raise here is an essential part of the answer: some part of what you might conventionally call the version number (whether that's a timestamp, build number, or the release status) needs to not appear in the POM, even (especially) for released binaries. -Dan ___ Notice: This email message, together with any attachments, may contain information of BEA Systems, Inc., its subsidiaries and affiliated entities, that may be confidential, proprietary, copyrighted and/or legally privileged, and is intended solely for the use of the individual or entity named in this message. If you are not the intended recipient, and have received this message in error, please immediately return this by email and then delete it. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Who should use SNAPSHOT when? RE: The Future of the Release Process.
Comments inline... Brett Porter wrote: Does this cover the whole topic? http://docs.codehaus.org/display/MAVENUSER/BEA+Maven+Requirement+Documen ts I'd say that highlights a lot of the most important requirements; it's probably best read together with the blog article as well. http://darkforge.blogspot.com/2006/12/compass-as-compared-with-mavens.ht ml I think it's all good - as long as it is all built on top of the stuff we already have (including what Joakim is proposing). It will probably require changes to many aspects of Maven (packaging, dependency resolution, deployment and release). It will also touch on Continuum and Archiva, I imagine. Is this right? I would say so. -Dan ___ Notice: This email message, together with any attachments, may contain information of BEA Systems, Inc., its subsidiaries and affiliated entities, that may be confidential, proprietary, copyrighted and/or legally privileged, and is intended solely for the use of the individual or entity named in this message. If you are not the intended recipient, and have received this message in error, please immediately return this by email and then delete it. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Who should use SNAPSHOT when? RE: The Future of the Release Process.
On 12/22/06, Dan Fabulich [EMAIL PROTECTED] wrote: From: Kenney Westerhof [mailto:[EMAIL PROTECTED] If I understand correctly, the problem is that a 'staged' release still contains a SNAPSHOT keyword in the metadata/filename? Yes, that's the problem. I would agree, but that's not how I understand staging to work at all. The Apache projects I work on like to vote on the *exact* artifacts that will be released to the public, so the staged release must not have a -SNAPSHOT qualifier. -- Wendy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Who should use SNAPSHOT when? RE: The Future of the Release Process.
On 12/22/06, Wendy Smoak [EMAIL PROTECTED] wrote: On 12/22/06, Dan Fabulich [EMAIL PROTECTED] wrote: From: Kenney Westerhof [mailto:[EMAIL PROTECTED] If I understand correctly, the problem is that a 'staged' release still contains a SNAPSHOT keyword in the metadata/filename? Yes, that's the problem. I would agree, but that's not how I understand staging to work at all. The Apache projects I work on like to vote on the *exact* artifacts that will be released to the public, so the staged release must not have a -SNAPSHOT qualifier. Agreed. When I vote on a release, I'm always saying show me the bits. In this scenario, it would be nice for the release plugin to have an option to copy approved artifacts from the staging area to the release area (along with corresponding signatures and metadata updates) *without* any modification to the artifacts themselves. -- Wendy Craig
Re: The Future of the Release Process.
Carlos Sanchez wrote: just some holiday fun ahead... On 12/20/06, Steve Loughran [EMAIL PROTECTED] wrote: We are 1. planning an apache con EU talk, fear the repository police. in suitable costumes, i hope ;-) nothing planned, but if we have someone in the audience cued up to say I didnt expect the spanish inquisition we could have somebody burst in monty-python style with nobody expects the spanish inquisition I volunteer to play as Spanish inquisitor ;) Oh, now that would be good. You could wear a t-shirt at the conference. As far as the repository concerned, I am spanish inquisition hehe I like it, A repo man spends his life getting into tense situations. Yes indeed. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: The Future of the Release Process.
robert burrell donkin wrote: On 12/20/06, Steve Loughran [EMAIL PROTECTED] wrote: robert burrell donkin wrote: On 12/19/06, Steve Loughran [EMAIL PROTECTED] wrote: robert burrell donkin wrote: On 12/19/06, Steve Loughran [EMAIL PROTECTED] wrote: snip 2. looking to get management approval to hand the code to the MIT Simile project (e.g. Stefano) cool (there seems to a lot of interest developing around RDF ATM amongst apache members) stefano's keen to see discussions related to apache and RDFs in the labs http://mail-archives.apache.org/mod_mbox/labs-labs/200612.mbox/[EMAIL PROTECTED] (which makes sense to me.) In theory, RDF is a better way of gluing together metadata in a way that is tool neutral. For little tools, it should be effective. I know Stefano is a fan of the semantic web, but to me the JAR repository is an interesting analysis of how well it will work. We know that today there's a lot of inconsistency out there, even though there are some dedicated people (esp. Carlos) who put a lot of effort in to keeping things under control. If we have problems chaining together metadata from a single repository, what are the long term implications for the semantic web going to be? +1 be interesting to see the level of practical benefit from application neutral meta-data yes indeed. 3. Thinking of something to audit the metadata. Maybe in prolog (my choice), maybe something else. i plan to persuade projects (by including this in RAT rather than a conventional configuration file) to start recording auditing meta data about documents which didn't originate at apache in RDF. i've also been toying with the idea of using RDF to record relationships between licenses and policy about licenses. these sound like related problems to me yes, one of my proposed 'enhancements' for both pom and ivy.xml files is to include an audit trail in there, such as who created the pom. Supporting a metadata element that took RDF-as-XML triples would be a very extensible way to do this. Wagon and Ivy could ignore the data, but other tools could extract it. +1 probably want more than just creator. for a repository, the creator of the pom may be different from the distributor who uploaded the descriptor and the original author of the artifact described. all of these would be useful in determining the audit trail. for example, i'd be more inclined to trust an artifact with signatures from representatives of each group than any alone. you're into a pool of complexity there. Ideally, each artifact should have an extensible set of metadata, including stuff that could be added later, even by different people. Instead of having RDF triples, each fact should be marked as who issued it, and when. We'd require that each entity/agent is consistent at a point in time, but they are allowed to change their mind later, and other people are allowed to be utterly inconsistent. the hard part is reconciling all that data, as you cannot just treat them as facts to chain off. you need to look at later stuff first, and apply trust rules to the entities. Which is why the current set of RDF tools don't go there yet. How would you use RDF to differentiate the mysql interpreation of GPL from everyone elses? i've been trying to work that out for a while it seems to me that there are several different layers of meta-data for a particular resource (an element of an open source project), i'm interested in license, license family, origin (url where the copy came from) and rights holder. (maybe other stuff such as whether it's been modified) i'm also interested in questions about aggregates: whether a particular collection of elements can be distributed together given some rules and meta-data about the other URLs referenced in the resource meta-data. together, these rules and meta-data would be policy. in the case of mysql, the family would be GPL but the license would be mysql-GPL. a policy might decide to trust their promise and describe this in meta-data. (i haven't really had time to explore this yet but i have hopes that this combination might be enough to make things work.) I see, so the relaxed LGPL policy would declare that they consider themselves compatible with ASF2, and apache could consider themselves compatible with GPL, but the FSF could say that they dont (or that they havent decided, which is a separate fact) [error] artifact rejected. [error] It happens sometimes. People just explode. Natural causes. +1 humour injection is a powerful design pattern repo man powers the satirical web Maybe we need a commons-imdb-quotes package that retrieves quote bundles from IMDB for reporting. When you install an app you can choose whether you want quotes fro repo-man, life of brian or even the al pacino version of scarface. -steve - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands,
Re: The Future of the Release Process.
On 12/21/06, Steve Loughran [EMAIL PROTECTED] wrote: robert burrell donkin wrote: On 12/20/06, Steve Loughran [EMAIL PROTECTED] wrote: robert burrell donkin wrote: On 12/19/06, Steve Loughran [EMAIL PROTECTED] wrote: robert burrell donkin wrote: On 12/19/06, Steve Loughran [EMAIL PROTECTED] wrote: snip 2. looking to get management approval to hand the code to the MIT Simile project (e.g. Stefano) cool (there seems to a lot of interest developing around RDF ATM amongst apache members) stefano's keen to see discussions related to apache and RDFs in the labs http://mail-archives.apache.org/mod_mbox/labs-labs/200612.mbox/[EMAIL PROTECTED] (which makes sense to me.) In theory, RDF is a better way of gluing together metadata in a way that is tool neutral. For little tools, it should be effective. I know Stefano is a fan of the semantic web, but to me the JAR repository is an interesting analysis of how well it will work. We know that today there's a lot of inconsistency out there, even though there are some dedicated people (esp. Carlos) who put a lot of effort in to keeping things under control. If we have problems chaining together metadata from a single repository, what are the long term implications for the semantic web going to be? +1 be interesting to see the level of practical benefit from application neutral meta-data yes indeed. 3. Thinking of something to audit the metadata. Maybe in prolog (my choice), maybe something else. i plan to persuade projects (by including this in RAT rather than a conventional configuration file) to start recording auditing meta data about documents which didn't originate at apache in RDF. i've also been toying with the idea of using RDF to record relationships between licenses and policy about licenses. these sound like related problems to me yes, one of my proposed 'enhancements' for both pom and ivy.xml files is to include an audit trail in there, such as who created the pom. Supporting a metadata element that took RDF-as-XML triples would be a very extensible way to do this. Wagon and Ivy could ignore the data, but other tools could extract it. +1 probably want more than just creator. for a repository, the creator of the pom may be different from the distributor who uploaded the descriptor and the original author of the artifact described. all of these would be useful in determining the audit trail. for example, i'd be more inclined to trust an artifact with signatures from representatives of each group than any alone. you're into a pool of complexity there. Ideally, each artifact should have an extensible set of metadata, including stuff that could be added later, even by different people. Instead of having RDF triples, each fact should be marked as who issued it, and when. We'd require that each entity/agent is consistent at a point in time, but they are allowed to change their mind later, and other people are allowed to be utterly inconsistent. the hard part is reconciling all that data, as you cannot just treat them as facts to chain off. you need to look at later stuff first, and apply trust rules to the entities. Which is why the current set of RDF tools don't go there yet. the problem of relatavism in the semantic web sounds tough How would you use RDF to differentiate the mysql interpreation of GPL from everyone elses? i've been trying to work that out for a while it seems to me that there are several different layers of meta-data for a particular resource (an element of an open source project), i'm interested in license, license family, origin (url where the copy came from) and rights holder. (maybe other stuff such as whether it's been modified) i'm also interested in questions about aggregates: whether a particular collection of elements can be distributed together given some rules and meta-data about the other URLs referenced in the resource meta-data. together, these rules and meta-data would be policy. in the case of mysql, the family would be GPL but the license would be mysql-GPL. a policy might decide to trust their promise and describe this in meta-data. (i haven't really had time to explore this yet but i have hopes that this combination might be enough to make things work.) I see, so the relaxed LGPL policy would declare that they consider themselves compatible with ASF2, and apache could consider themselves compatible with GPL, but the FSF could say that they dont (or that they havent decided, which is a separate fact) +1 [error] artifact rejected. [error] It happens sometimes. People just explode. Natural causes. +1 humour injection is a powerful design pattern repo man powers the satirical web Maybe we need a commons-imdb-quotes package that retrieves quote bundles from IMDB for reporting. When you install an app you can choose whether you want quotes fro repo-man, life of brian or even the al pacino
Re: The Future of the Release Process.
robert burrell donkin wrote: On 12/19/06, Steve Loughran [EMAIL PROTECTED] wrote: robert burrell donkin wrote: On 12/19/06, Steve Loughran [EMAIL PROTECTED] wrote: snip I'd like to see POM auditing in there somewhere. +1 henri's been talking about adding this to RAT. IMHO this should be implemented as a separate component so that it could be used by a variety of applications: for example, it might be interesting to be able to apply policy rules as part of subversion commits or staging updates. I have a colleague who has been converting all the POMs and .class files in the repository to RDF, ready for auditing. cool :-) We are 1. planning an apache con EU talk, fear the repository police. in suitable costumes, i hope ;-) nothing planned, but if we have someone in the audience cued up to say I didnt expect the spanish inquisition we could have somebody burst in monty-python style with nobody expects the spanish inquisition 2. looking to get management approval to hand the code to the MIT Simile project (e.g. Stefano) cool (there seems to a lot of interest developing around RDF ATM amongst apache members) stefano's keen to see discussions related to apache and RDFs in the labs http://mail-archives.apache.org/mod_mbox/labs-labs/200612.mbox/[EMAIL PROTECTED] (which makes sense to me.) In theory, RDF is a better way of gluing together metadata in a way that is tool neutral. For little tools, it should be effective. I know Stefano is a fan of the semantic web, but to me the JAR repository is an interesting analysis of how well it will work. We know that today there's a lot of inconsistency out there, even though there are some dedicated people (esp. Carlos) who put a lot of effort in to keeping things under control. If we have problems chaining together metadata from a single repository, what are the long term implications for the semantic web going to be? 3. Thinking of something to audit the metadata. Maybe in prolog (my choice), maybe something else. i plan to persuade projects (by including this in RAT rather than a conventional configuration file) to start recording auditing meta data about documents which didn't originate at apache in RDF. i've also been toying with the idea of using RDF to record relationships between licenses and policy about licenses. these sound like related problems to me yes, one of my proposed 'enhancements' for both pom and ivy.xml files is to include an audit trail in there, such as who created the pom. Supporting a metadata element that took RDF-as-XML triples would be a very extensible way to do this. Wagon and Ivy could ignore the data, but other tools could extract it. How would you use RDF to differentiate the mysql interpreation of GPL from everyone elses? Working title repo-man. This not only lets us have a good name, it gives us good quotes repo man rules :-) Best film Alex cox ever made. it's amazing at the cinema... It's 4 A.M., do you know where your jar is? You don't even know what's in your own trunk! And you know what? I think you're afraid to find out! +1 Ok. then, we have a name! I wonder if we could include some of the quotes as error messages ( http://www.imdb.com/title/tt0087995/quotes) . Like when an artifact fails the audit [error] artifact rejected. [error] It happens sometimes. People just explode. Natural causes. -Steve - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: The Future of the Release Process.
just some holiday fun ahead... On 12/20/06, Steve Loughran [EMAIL PROTECTED] wrote: We are 1. planning an apache con EU talk, fear the repository police. in suitable costumes, i hope ;-) nothing planned, but if we have someone in the audience cued up to say I didnt expect the spanish inquisition we could have somebody burst in monty-python style with nobody expects the spanish inquisition I volunteer to play as Spanish inquisitor ;) Working title repo-man. This not only lets us have a good name, it gives us good quotes repo man rules :-) Best film Alex cox ever made. it's amazing at the cinema... It's 4 A.M., do you know where your jar is? You don't even know what's in your own trunk! And you know what? I think you're afraid to find out! +1 Ok. then, we have a name! I wonder if we could include some of the quotes as error messages ( http://www.imdb.com/title/tt0087995/quotes) . Like when an artifact fails the audit [error] artifact rejected. [error] It happens sometimes. People just explode. Natural causes. hehe I like it, A repo man spends his life getting into tense situations. -Steve - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- I could give you my word as a Spaniard. No good. I've known too many Spaniards. -- The Princess Bride - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: The Future of the Release Process.
On 12/20/06, Steve Loughran [EMAIL PROTECTED] wrote: robert burrell donkin wrote: On 12/19/06, Steve Loughran [EMAIL PROTECTED] wrote: robert burrell donkin wrote: On 12/19/06, Steve Loughran [EMAIL PROTECTED] wrote: snip 2. looking to get management approval to hand the code to the MIT Simile project (e.g. Stefano) cool (there seems to a lot of interest developing around RDF ATM amongst apache members) stefano's keen to see discussions related to apache and RDFs in the labs http://mail-archives.apache.org/mod_mbox/labs-labs/200612.mbox/[EMAIL PROTECTED] (which makes sense to me.) In theory, RDF is a better way of gluing together metadata in a way that is tool neutral. For little tools, it should be effective. I know Stefano is a fan of the semantic web, but to me the JAR repository is an interesting analysis of how well it will work. We know that today there's a lot of inconsistency out there, even though there are some dedicated people (esp. Carlos) who put a lot of effort in to keeping things under control. If we have problems chaining together metadata from a single repository, what are the long term implications for the semantic web going to be? +1 be interesting to see the level of practical benefit from application neutral meta-data 3. Thinking of something to audit the metadata. Maybe in prolog (my choice), maybe something else. i plan to persuade projects (by including this in RAT rather than a conventional configuration file) to start recording auditing meta data about documents which didn't originate at apache in RDF. i've also been toying with the idea of using RDF to record relationships between licenses and policy about licenses. these sound like related problems to me yes, one of my proposed 'enhancements' for both pom and ivy.xml files is to include an audit trail in there, such as who created the pom. Supporting a metadata element that took RDF-as-XML triples would be a very extensible way to do this. Wagon and Ivy could ignore the data, but other tools could extract it. +1 probably want more than just creator. for a repository, the creator of the pom may be different from the distributor who uploaded the descriptor and the original author of the artifact described. all of these would be useful in determining the audit trail. for example, i'd be more inclined to trust an artifact with signatures from representatives of each group than any alone. How would you use RDF to differentiate the mysql interpreation of GPL from everyone elses? i've been trying to work that out for a while it seems to me that there are several different layers of meta-data for a particular resource (an element of an open source project), i'm interested in license, license family, origin (url where the copy came from) and rights holder. (maybe other stuff such as whether it's been modified) i'm also interested in questions about aggregates: whether a particular collection of elements can be distributed together given some rules and meta-data about the other URLs referenced in the resource meta-data. together, these rules and meta-data would be policy. in the case of mysql, the family would be GPL but the license would be mysql-GPL. a policy might decide to trust their promise and describe this in meta-data. (i haven't really had time to explore this yet but i have hopes that this combination might be enough to make things work.) but this needs more work Working title repo-man. This not only lets us have a good name, it gives us good quotes repo man rules :-) Best film Alex cox ever made. it's amazing at the cinema... It's 4 A.M., do you know where your jar is? You don't even know what's in your own trunk! And you know what? I think you're afraid to find out! +1 Ok. then, we have a name! I wonder if we could include some of the quotes as error messages ( http://www.imdb.com/title/tt0087995/quotes) . Like when an artifact fails the audit [error] artifact rejected. [error] It happens sometimes. People just explode. Natural causes. +1 humour injection is a powerful design pattern repo man powers the satirical web - robert - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: The Future of the Release Process.
On 12/20/06, Carlos Sanchez [EMAIL PROTECTED] wrote: just some holiday fun ahead... On 12/20/06, Steve Loughran [EMAIL PROTECTED] wrote: We are 1. planning an apache con EU talk, fear the repository police. in suitable costumes, i hope ;-) nothing planned, but if we have someone in the audience cued up to say I didnt expect the spanish inquisition we could have somebody burst in monty-python style with nobody expects the spanish inquisition I volunteer to play as Spanish inquisitor ;) +1 maybe the chair should give the cue - robert - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: The Future of the Release Process.
Joakim Erdfelt wrote: This is just a synopsis email about the future of the release process. Disclaimer: I am not attempting to set policy, just to get discussion going, document it, and work towards the ideal toolchain that make the future of apache releases smooth, consistent, and resilient This is good, but some people out there are very sensitive to their build tools dictating policy, so get a broad consensus on this before shipping . One issue is that I believe apache requires the PMC to vote on a built up redistributable artifact. You dont vote before you release, you build, stick it in staging, download it, test it and then finally say we ship this one. I'd like to see POM auditing in there somewhere. Desired End Goal: This discussion will revolve around the apache process for official releases of projects. 1) Releases are non-SNAPSHOT. 2) All releases shall be voted on. * Call a vote on the appropriate dev@ mailing list. * Wait 72 hours. * If unanimous +1 vote by 7 or more PMC members, then release can occur before 72 hour window is up. * Document the following in the vote: a) Project Name b) Project Version in Subversion. c) Desired Release Version ID. d) Expected Next Development Version ID. e) Age of development version (days since last non-snapshot release) f) Downstream snapshot dependencies that might cause problems. g) Jiras that have been closed/resolved for this release. f) Tasks that have been completed in SCM but do not appear in the Jira completion list. g) URL to jira completion report for this version. h) SCM revision being voted on. * If a change occurs to the project, the vote should be updated with the change and a new vote started. (this resets the 72 hours) This only consitutes changes that occur in the project itself, not downstream dependencies. 2) A Released project will contain the following files. Example directory contents of a project artifact 'foo', version '1.0' * foo-1.0.pom(pom / metadata for artifact) * foo-1.0.jar(actual binary artifact) * foo-1.0-sources.jar(source code for artifact) * foo-1.0-javadoc.jar(javadoc for artifact) 3) Each released file will be signed using the (apache.org) gpg signature of the commitor doing the release. 4) Each released file and gpg signature will have a hashcode generated for the file (managed by wagon) in SHA1 and MD5 format. 5) All jar files produced (binary, sources, and javadoc) will contain the following mandatory files: * META-INF/LICENSE.txt * META-INF/NOTICE.txt 6) All source files in the sources.jar files will contain the following header. ---(snip)--- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the License); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ---(snip)--- Techniques: 1) Original release plugin process. This is the release:prepare and release:perform technique. This technique does the following... a) Gather information about release IDs. * Release Version ID * Tag ID * Next Development Version ID a) Updates the poms in trunk to the provided Release Version ID. b) Tags these updated poms in subversion using the provided Tag ID. c) Updates the poms in trunk to the provided Next Development Version ID. d) Builds the release using the Subversion Tag. * clean * integration-test * install * site * deploy (binary and site) 2) Staged release process This process uses a staging workflow, there is no plugin for this process (yet). The benefits of this process is that a real binary is being blessed. This process also provides a way to resolve problems that occur during the release process. It goes like this... a) Call vote b) Wait on approval. c) Collect release information. * Release Version ID * Tag/Branch ID * Next Development Version ID c) 'prepare
Re: The Future of the Release Process.
Hello Joakim, I like this discussion, I am a big fan of automating the release process. I also have some hands on experience being RM for Ant 1.6 and 1.7. For Ant 1.7, we are using what you call the staged release process. When I prepare a build, I have to assume that the vote on the binaries will be positive and prepare everything with a release version in mind. Tagging the release should to my opinion happen between preparing and staging the release. To make sure everything is clean, the SVN sandbox could be recreated based on the tag. When I did my first build of ANT 1.7.0 last week, someone checked code in between the time when I created my SVN sandbox and the time when I created my tag. In the end, this change was included in the tag but not in my build. Bad ... See [1] the command used to create the tag Also, creating a branch for a release is optional. Right now we are releasing Ant 1.7.0 from the trunk, and we do not have a ANT_17_BRANCH. Suggested modifications for your staged release process : c) 'prepare' release (occurs once) * Create a branch using provided Branch ID for this release. * Update poms in branch to provided Release Version ID. +* Optionally update documentation in branch to provided Release Version ID. * Update poms in trunk to provided Next Development Version ID. + d) tag the release (occurs 1 or more times) + e) create a new SVN sandbox using the tag created in d) - d) stage release (occurs 1 or more times) + f) 'stage' release (occurs 1 or more times) It would be cool if you could persist your ideas concerning the release process to a document in SVN or to a Wiki page. Regards, Antoine [1] svn copy https://svn.apache.org/repos/asf/ant/core/trunk \ https://svn.apache.org/repos/asf/ant/core/tags/ANT_170 \ -m Tagging version 1.7.0 of Ant - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: The Future of the Release Process.
On 12/19/06, Antoine Levy-Lambert [EMAIL PROTECTED] wrote: Hello Joakim, I like this discussion, I am a big fan of automating the release process. I also have some hands on experience being RM for Ant 1.6 and 1.7. For Ant 1.7, we are using what you call the staged release process. When I prepare a build, I have to assume that the vote on the binaries will be positive and prepare everything with a release version in mind. Tagging the release should to my opinion happen between preparing and staging the release. To make sure everything is clean, the SVN sandbox could be recreated based on the tag. When I did my first build of ANT 1.7.0 last week, someone checked code in between the time when I created my SVN sandbox and the time when I created my tag. In the end, this change was included in the tag but not in my build. Bad ... See [1] the command used to create the tag This cannot happen if you make a tag based from your local working copy instead of trunk The release plugin already does this. http://maven.apache.org/plugins/maven-release-plugin/examples/prepare-release.html tom Also, creating a branch for a release is optional. Right now we are releasing Ant 1.7.0 from the trunk, and we do not have a ANT_17_BRANCH. Suggested modifications for your staged release process : c) 'prepare' release (occurs once) * Create a branch using provided Branch ID for this release. * Update poms in branch to provided Release Version ID. +* Optionally update documentation in branch to provided Release Version ID. * Update poms in trunk to provided Next Development Version ID. + d) tag the release (occurs 1 or more times) + e) create a new SVN sandbox using the tag created in d) - d) stage release (occurs 1 or more times) + f) 'stage' release (occurs 1 or more times) It would be cool if you could persist your ideas concerning the release process to a document in SVN or to a Wiki page. Regards, Antoine [1] svn copy https://svn.apache.org/repos/asf/ant/core/trunk \ https://svn.apache.org/repos/asf/ant/core/tags/ANT_170 \ -m Tagging version 1.7.0 of Ant - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: The Future of the Release Process.
On 12/19/06, Steve Loughran [EMAIL PROTECTED] wrote: snip I'd like to see POM auditing in there somewhere. +1 henri's been talking about adding this to RAT. IMHO this should be implemented as a separate component so that it could be used by a variety of applications: for example, it might be interesting to be able to apply policy rules as part of subversion commits or staging updates. - robert - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: The Future of the Release Process.
robert burrell donkin wrote: On 12/19/06, Steve Loughran [EMAIL PROTECTED] wrote: snip I'd like to see POM auditing in there somewhere. +1 henri's been talking about adding this to RAT. IMHO this should be implemented as a separate component so that it could be used by a variety of applications: for example, it might be interesting to be able to apply policy rules as part of subversion commits or staging updates. I have a colleague who has been converting all the POMs and .class files in the repository to RDF, ready for auditing. We are 1. planning an apache con EU talk, fear the repository police. 2. looking to get management approval to hand the code to the MIT Simile project (e.g. Stefano) 3. Thinking of something to audit the metadata. Maybe in prolog (my choice), maybe something else. Working title repo-man. This not only lets us have a good name, it gives us good quotes It's 4 A.M., do you know where your jar is? You don't even know what's in your own trunk! And you know what? I think you're afraid to find out! -steve - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: The Future of the Release Process.
On 12/19/06, Steve Loughran [EMAIL PROTECTED] wrote: robert burrell donkin wrote: On 12/19/06, Steve Loughran [EMAIL PROTECTED] wrote: snip I'd like to see POM auditing in there somewhere. +1 henri's been talking about adding this to RAT. IMHO this should be implemented as a separate component so that it could be used by a variety of applications: for example, it might be interesting to be able to apply policy rules as part of subversion commits or staging updates. I have a colleague who has been converting all the POMs and .class files in the repository to RDF, ready for auditing. cool :-) We are 1. planning an apache con EU talk, fear the repository police. in suitable costumes, i hope ;-) 2. looking to get management approval to hand the code to the MIT Simile project (e.g. Stefano) cool (there seems to a lot of interest developing around RDF ATM amongst apache members) stefano's keen to see discussions related to apache and RDFs in the labs http://mail-archives.apache.org/mod_mbox/labs-labs/200612.mbox/[EMAIL PROTECTED] (which makes sense to me.) 3. Thinking of something to audit the metadata. Maybe in prolog (my choice), maybe something else. i plan to persuade projects (by including this in RAT rather than a conventional configuration file) to start recording auditing meta data about documents which didn't originate at apache in RDF. i've also been toying with the idea of using RDF to record relationships between licenses and policy about licenses. these sound like related problems to me Working title repo-man. This not only lets us have a good name, it gives us good quotes repo man rules :-) it's amazing at the cinema... It's 4 A.M., do you know where your jar is? You don't even know what's in your own trunk! And you know what? I think you're afraid to find out! +1 - robert - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: The Future of the Release Process.
On 14/12/2006, at 10:46 AM, Joakim Erdfelt wrote: * If unanimous +1 vote by 7 or more PMC members, then release can occur before 72 hour window is up. Don't really understand the rush, or the arbitrary number 7 (yes, it is the number of perfection, but other than that, it has no significance in a growing PMC of 18 and is far more than the required 3 :). A flat 3 days is far simpler, and if we are executing better on our releases it should be no real barrier. It also gives the important time for people to change their votes if someone else highlights something wrong. I'm not adverse to emergency releases either - if they are designated as such they only require the 3 PMC members to vote (though more is better, and it must be exceptional circumstances - I'd generally prefer to just pull the previous one if there was something that wrong). e) Age of development version (days since last non-snapshot release) is this necessary? I think it's a generally useful thing to know, but probably more when one is neglected, not when it is about to be released f) Downstream snapshot dependencies that might cause problems. There must be none when it comes to be voted on. It should be ready to pull the trigger. f) Tasks that have been completed in SCM but do not appear in the Jira completion list. I don't think we should encourage this. Just put it in JIRA. g) URL to jira completion report for this version. Is this needed if they have been pasted into the email? * If a change occurs to the project, the vote should be updated with the change and a new vote started. (this resets the 72 hours) This only consitutes changes that occur in the project itself, not downstream dependencies. Agreed, but the downstream dependencies changing will either not be part of the release, or cause the project to update to a new snapshot that does constitute a change, so I don't see that the statement adds anything. [snip agreed on rules] For these rules, can we incorporate the use of RAT? http:// code.google.com/p/arat/ See also stuff Commons were shopping for last time I started discussing this: * http://wiki.apache.org/jakarta-commons/ReleaseChecking * http://wiki.apache.org/jakarta-commons/ReleaseShoppingList It goes like this... a) Call vote I think this is (e) since it requires steps from b, c, c, d. BTW, I think voting on binaries + source is a Good Thing (TM), but if we expect this to be adopted widely, we need to be flexible enough to allow production of just the source tag to vote on as well. I think all this can be done via phases in the release plugin (with increased support for resuming later), and then being able to turn some on/off (or doing binding of some sort, like the build lifecycle). * Generate 'need to bless', email to mailing list about staged artifacts and site. Agree, as a template, not an automated mail. * Send email announcing (to announce@ mailing lists) the release. - Include links to artifacts on real repository. - Include changelog I think again, this should be a template to help rather than be automated. maven-release-staging-plugin (not written) This could be useful to bundle up the various other plugins into an simple goal. possible goals :prepare, :stage, :bless. This plugin could also ensure the configuration parameters needed in the settings.xml file to perform a release. Does it need a separate plugin? I think this is just an extra workflow in the main release process. You'll want to think about this in terms of how it can be triggered inside Continuum as well... maven-apache-deploy-check-plugin (not written) I think this is where RAT could come in. Cheers, Brett - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Who should use SNAPSHOT when? RE: The Future of the Release Process.
Does this cover the whole topic? http://docs.codehaus.org/display/MAVENUSER/BEA+Maven+Requirement +Documents I think it's all good - as long as it is all built on top of the stuff we already have (including what Joakim is proposing). It will probably require changes to many aspects of Maven (packaging, dependency resolution, deployment and release). It will also touch on Continuum and Archiva, I imagine. Is this right? - Brett On 14/12/2006, at 2:46 PM, Jason van Zyl wrote: Hi Dan, I think what you are describing is what you do at your work place, and I think it might be good if you could give us a full description of your system for folks here to help them understand exactly what it is you're talking about. I probably know a little more about it then most here but I still don't entirely grok the workflow you're describing. Maybe some simple examples of how artifacts names would change through the workflow you're describing versus our current approach would be helpful. Anything that makes releases easier is a good thing. Thanks, Jason. On 13 Dec 06, at 10:11 PM 13 Dec 06, Dan Fabulich wrote: Thanks for a great e-mail Joakim. I wanted to chime in with my two cents... (I've been off the radar for a couple of months while waiting for permission to sign my ICLA; it's in now, and I'm now back to paying more careful attention to this process... forgive me if some of this has already been covered.) One of the goals I know we've expressed before (but not explicitly listed here) is that Maven's release process should lead by example: other projects (whether open-source or closed-source) who haven't put as much thought into release engineering as we have should look to us as an example of the right way to do it. IMO, the requirements around SNAPSHOT releases is an important difference between open-source requirements for the release process and closed-source requirements... In this e-mail I want to describe an alternative release process (overlapping with the one Joakim described) that never uses SNAPSHOT and which is more appropriate to some organizations, perhaps especially closed-source ones. I think we all agree that it's bad (at least a little bit) to change things at the last minute before release (whether it be source code, binaries, or even your build process); one goal of the updated release process is that we should make as few last-minute changes as possible, and, to the greatest extent possible, bless binaries. But so long as you have the word SNAPSHOT embedded into your JARs during development, you'll have to change *something* at the last second, if only to remove the word SNAPSHOT. There is another way, which is better for at least some groups some of the time. If you never used SNAPSHOT, but Maven enforced a requirement that all JARs would have build numbers embedded in them (not appearing in the file name, but appearing in JAR manifest.mf and in the deployed POM), then the release process could be as little as copying the JARs into the right place and updating some metadata to call them released. Here's another way of saying the same thing. The release process Joakim described goes like this: a) Call vote b) Wait on approval. c) Collect release information. c) 'prepare' release (occurs once) d) 'stage' release (occurs 1 or more times) e) Wait on consensus from PMC for blessed artifacts. f) 'bless' release (occurs once) As this is described, it sounds as if projects would normally spend extremely little time in step D, stage. But if Maven provided more complete build numbering support for non-SNAPSHOT builds, you could imagine the project spending their entire development life in step D. After step E a decision was made to release, in step F the blessing would occur, and development would immediately begin on 1.1 in step D... no period of time spent in SNAPSHOT, so you wouldn't need to modify your code (prepare) right before release. Although I've highlighted one big advantage of not marking code under development as SNAPSHOT, the most significant disadvantage of doing it this way is that end users might confuse SNAPSHOT releases with the real official thing. (Perhaps especially if users just copy the relevant jars out of the repository and then leave Maven behind.) This can result in unnecessary support questions from users as they (unwittingly) complain about bugs in unreleased code, and can complicated support diagnostics as the person providing support may believe that the end-user has version 1.4, when they really have a developer snapshot of 1.4, never intended for release. With that said, I think most closed-source software development organizations don't have anywhere near as much fear of end-users grabbing under-development code and calling for support, since those binaries are typically kept a secret; in that case, the advantage of
Re: The Future of the Release Process.
On Wednesday 13 December 2006 18:46, Joakim Erdfelt wrote: Techniques: 2) Staged release process This process uses a staging workflow, there is no plugin for this process (yet). The benefits of this process is that a real binary is being blessed. This process also provides a way to resolve problems that occur during the release process. It goes like this... a) Call vote b) Wait on approval. .. d) 'stage' release (occurs 1 or more times) . e) Wait on consensus from PMC for blessed artifacts. * if not-blessed, make changes required and re-do step d. f) 'bless' release (occurs once) I don't think this process is quite correct and is certainly not the process that is allowed to be used in the incubator. (a) and (b) basically boil down to: Start a discussion on the list about what needs to be in the release and make sure it's all documented in JIRA and nominate a release manager. When all items in JIRA are complete, the release manager will start producing release candidates. e) is the vote. You HAVE to vote on the staged binaries. You don't vote until the binaries are properly staged. If you have to restage the binaries (problem detected and fixed), you have to restart the vote. I know this is very different than the way the Maven team has been working. However, it's what we MUST do in the incubator. Tools: The following tools are to aide in this process. maven-remote-resource-plugin (released) This will copy the appropriate Apache process LICENSE.txt and NOTICE.txt files into place for the released artifacts. The NOTICE.txt file generation is not working correctly yet.It doesn't produced a NOTICE file that would actually pass Apache legal review. I think the velocity template is OK, but the plugin itself isn't passing the required information to velocity. Thus, the released flag above should change to alpha version available. Thanks! -- J. Daniel Kulp Principal Engineer IONA P: 781-902-8727C: 508-380-7194 [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: The Future of the Release Process.
On Thu, December 14, 2006 1:46 am, Joakim Erdfelt wrote: Disclaimer: I am not attempting to set policy, just to get discussion going, document it, and work towards the ideal toolchain that make the future of apache releases smooth, consistent, and resilient Desired End Goal: This discussion will revolve around the apache process for official releases of projects. [snip lots and lots of barriers to release] Although I can see the desire to enforce compliance on policy, I think adopting barriers to release like this will cause the projects to just never be released, making an already bad problem worse. There is another showstopper to this: One of the key provisions of releasing artifacts at the ASF is that a release cannot be vetoed. The thinking is that our code in SVN should always be policy compliant, meaning that a release at any time should never be a problem. What will probably work a lot better is for compliance to be enforced at the source level, and here a compliance plugin hooked into the test phase can check for all the various things like license headers on files etc as required by the project. So rather than chase people around months after code was committed to bring it into compliance, the developer of a plugin can get immediate feedback while they are developing to say that files X, Y and Z are non compliant and need to be fixed. Extra points if the plugin can auto-fix some of the issues. Regards, Graham -- - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: The Future of the Release Process.
On 12/14/06, Graham Leggett [EMAIL PROTECTED] wrote: On Thu, December 14, 2006 1:46 am, Joakim Erdfelt wrote: Disclaimer: I am not attempting to set policy, just to get discussion going, document it, and work towards the ideal toolchain that make the future of apache releases smooth, consistent, and resilient Desired End Goal: This discussion will revolve around the apache process for official releases of projects. [snip lots and lots of barriers to release] Although I can see the desire to enforce compliance on policy, I think adopting barriers to release like this will cause the projects to just never be released, making an already bad problem worse. There is another showstopper to this: One of the key provisions of releasing artifacts at the ASF is that a release cannot be vetoed. The thinking is that our code in SVN should always be policy compliant, meaning that a release at any time should never be a problem. What will probably work a lot better is for compliance to be enforced at the source level, and here a compliance plugin hooked into the test phase can check for all the various things like license headers on files etc as required by the project. So rather than chase people around months after code was committed to bring it into compliance, the developer of a plugin can get immediate feedback while they are developing to say that files X, Y and Z are non compliant and need to be fixed. Extra points if the plugin can auto-fix some of the issues. that sounds very reasonable to have running in Continuum. Some time ago i added CPD check to the Maven parent pom but had to be commented out because not all projects comply Regards, Graham -- - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- I could give you my word as a Spaniard. No good. I've known too many Spaniards. -- The Princess Bride - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Who should use SNAPSHOT when? RE: The Future of the Release Process.
Hi, If I understand correctly, the problem is that a 'staged' release still contains a SNAPSHOT keyword in the metadata/filename? Also, how would you see snapshot 'releases' without a snapshot keyword? If there's no indicator (timestamp?) in the filename, you'll overwrite the previous deployed versions, which is bad. Personally I'm pro release-candidate marking of artifacts. Either you have a '1.0-timestamp' release candidate version/file, or '1.0-rc1'. Doesn't really matter except that -rcX is more human readable. All snapshot deployments of artifacts could then be seen as release candidates/staged artifacts. What if maven understood the difference between the version in the pom and the version in the filename? You could deploy a release candidate with -rc1 (or timestamp) in the filename. This file will never change. The POM itself mentions the version without the -rc1. Then all you have to do when promoting a staged release to a real release is move the files to another directory (one without -SNAPSHOT or -rcX in the name), and rename the files in the process. No file contents have to be changed. You can never get access to that release candidate unless you specify '-rc1' in the version tag of dependencies on that artifact. Example of the repo structure to clarify things: groupId/ foo/ 1.0-rc1/ foo-1.0-rc1.jar foo-1.0-rc1.pom (version tag here says '1.0') 1.0/ ... bar/ 1.0-rc1/ bar-1.0-rc1.jar bar-1.0-rc1.pom (version tag here says '1.0') 1.0/ ... For projects apart from foo and bar that depend on this artifact, this works (comparable to SNAPSHOT artifacts). For dependencies of the rc artifact, it won't, unless you release per artifact: Assume both foo and bar are in the same release cycle, and foo depends on bar. The foo pom cannot be changed when released, so it's dependency on bar has to specify 1.0-rc1. This is a problem. But easily resolved: groupId/ foo/ 1.0/ maven-metadata.xml The metadata file mentiones that 1.0 is not released yet, and lists all release candidates. So when resolving bar-1.0 from foo, the metadata file is consulted and the 1.0-rc1 is used. The bad thing about this is that this works like SNAPSHOT resolution, except instead of specifying '1.0-SNAPSHOT' and getting '1.0-latest-timestamp' you specify '1.0' and get the latest '1.0-rcX'. This is not necessarily a bad thing: if you want to promote foo, you also have to release bar. Maven should check if 1.0 is resolved to 1.0 or an rc before releasing. Also something could be done in maven internally to mark projects as non-final, since it knows when resolving 1.0 that it got an RC. The scheme above is almost identical to SNAPSHOT resolution. Differences: - no SNAPSHOT tag in the artifact files themselves - wheter something is a SNAPSHOT is determined from the absense of the artifact in the correct version dir (1.0 is a snapshot since 1.0/ doesn't contain foo-1.0.pom and the metadata file points to another version). If you look at a pom file that has no SNAPSHOT in there, you won't know for sure if it's a snapshot unless you check where it's resolved from. Normally this is never a problem - all artifacts are resolved from the local repo. When creating wars or other compound artifacts, or assemblies, the filenames should match the version tag. For instance, assume foo is an ear project, embedding bar. If bar would be embedded as bar-1.0-rc1.jar, foo's contents would have to be changed on the final release because bar's filename would change, even though the version wouldn't. So when looking at a file bar-1.0.jar and the embedding pom you can only tell that it's a snapshot using the metadata file from groupId/bar/1.0/. Thoughts? Dan Fabulich wrote: Thanks for a great e-mail Joakim. I wanted to chime in with my two cents... (I've been off the radar for a couple of months while waiting for permission to sign my ICLA; it's in now, and I'm now back to paying more careful attention to this process... forgive me if some of this has already been covered.) One of the goals I know we've expressed before (but not explicitly listed here) is that Maven's release process should lead by example: other projects (whether open-source or closed-source) who haven't put as much thought into release engineering as we have should look to us as an example of the right way to do it. IMO, the requirements around SNAPSHOT releases is an important difference between open-source requirements for the release process and closed-source requirements... In this e-mail I want to describe an alternative release process (overlapping with the one Joakim described) that never uses SNAPSHOT and which is more appropriate to some organizations, perhaps especially closed-source ones. I think we all agree that it's bad (at least a little bit) to change things at the last minute before release (whether it be source code, binaries, or even your build process); one goal of
Re: The Future of the Release Process.
On Wednesday 13 December 2006 18:54, Tom Huybrechts wrote: 6) All source files in the sources.jar files will contain the following header. ---(snip)--- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the License); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ---(snip)--- Ditto for the pom itself ? Tom Yes, it should. However, there is a bug in Maven 2.0.4 that wipes out the headers on the deployed poms. Jason has a fix for that which will hopefully make it into 2.0.5. -- J. Daniel Kulp Principal Engineer IONA P: 781-902-8727C: 508-380-7194 [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
The Future of the Release Process.
This is just a synopsis email about the future of the release process. Disclaimer: I am not attempting to set policy, just to get discussion going, document it, and work towards the ideal toolchain that make the future of apache releases smooth, consistent, and resilient Desired End Goal: This discussion will revolve around the apache process for official releases of projects. 1) Releases are non-SNAPSHOT. 2) All releases shall be voted on. * Call a vote on the appropriate dev@ mailing list. * Wait 72 hours. * If unanimous +1 vote by 7 or more PMC members, then release can occur before 72 hour window is up. * Document the following in the vote: a) Project Name b) Project Version in Subversion. c) Desired Release Version ID. d) Expected Next Development Version ID. e) Age of development version (days since last non-snapshot release) f) Downstream snapshot dependencies that might cause problems. g) Jiras that have been closed/resolved for this release. f) Tasks that have been completed in SCM but do not appear in the Jira completion list. g) URL to jira completion report for this version. h) SCM revision being voted on. * If a change occurs to the project, the vote should be updated with the change and a new vote started. (this resets the 72 hours) This only consitutes changes that occur in the project itself, not downstream dependencies. 2) A Released project will contain the following files. Example directory contents of a project artifact 'foo', version '1.0' * foo-1.0.pom(pom / metadata for artifact) * foo-1.0.jar(actual binary artifact) * foo-1.0-sources.jar(source code for artifact) * foo-1.0-javadoc.jar(javadoc for artifact) 3) Each released file will be signed using the (apache.org) gpg signature of the commitor doing the release. 4) Each released file and gpg signature will have a hashcode generated for the file (managed by wagon) in SHA1 and MD5 format. 5) All jar files produced (binary, sources, and javadoc) will contain the following mandatory files: * META-INF/LICENSE.txt * META-INF/NOTICE.txt 6) All source files in the sources.jar files will contain the following header. ---(snip)--- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the License); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ---(snip)--- Techniques: 1) Original release plugin process. This is the release:prepare and release:perform technique. This technique does the following... a) Gather information about release IDs. * Release Version ID * Tag ID * Next Development Version ID a) Updates the poms in trunk to the provided Release Version ID. b) Tags these updated poms in subversion using the provided Tag ID. c) Updates the poms in trunk to the provided Next Development Version ID. d) Builds the release using the Subversion Tag. * clean * integration-test * install * site * deploy (binary and site) 2) Staged release process This process uses a staging workflow, there is no plugin for this process (yet). The benefits of this process is that a real binary is being blessed. This process also provides a way to resolve problems that occur during the release process. It goes like this... a) Call vote b) Wait on approval. c) Collect release information. * Release Version ID * Tag/Branch ID * Next Development Version ID c) 'prepare' release (occurs once) * Create a branch using provided Branch ID for this release. * Update poms in branch to provided Release Version ID. * Update poms in trunk to provided Next Development Version ID. d) 'stage' release (occurs 1 or more times) * Perform clean deploy. * Generate binary / sources / javadoc. * GPG Sign binary / sources / javadoc. * Generate Site. * Deploy
Re: The Future of the Release Process.
6) All source files in the sources.jar files will contain the following header. ---(snip)--- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the License); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ---(snip)--- Ditto for the pom itself ? Tom - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: The Future of the Release Process.
Joakim Erdfelt wrote: This is just a synopsis email about the future of the release process. Disclaimer: I am not attempting to set policy, just to get discussion going, document it, and work towards the ideal toolchain that make the future of apache releases smooth, consistent, and resilient Desired End Goal: This discussion will revolve around the apache process for official releases of projects. 1) Releases are non-SNAPSHOT. 2) All releases shall be voted on. * Call a vote on the appropriate dev@ mailing list. * Wait 72 hours. * If unanimous +1 vote by 7 or more PMC members, then release can occur before 72 hour window is up. * Document the following in the vote: a) Project Name b) Project Version in Subversion. c) Desired Release Version ID. d) Expected Next Development Version ID. e) Age of development version (days since last non-snapshot release) f) Downstream snapshot dependencies that might cause problems. g) Jiras that have been closed/resolved for this release. f) Tasks that have been completed in SCM but do not appear in the Jira completion list. g) URL to jira completion report for this version. h) SCM revision being voted on. * If a change occurs to the project, the vote should be updated with the change and a new vote started. (this resets the 72 hours) This only consitutes changes that occur in the project itself, not downstream dependencies. [snip/] I think we should have (2-g) and (2-f) above included as/in Release Notes. Release notes can be deployed to the repo along with other artifacts, so: * foo-1.0.pom(pom / metadata for artifact) * foo-1.0.jar(actual binary artifact) * foo-1.0-sources.jar(source code for artifact) * foo-1.0-javadoc.jar(javadoc for artifact) * foo-1.0-release-notes.txt (updates/features in the artifact) Cheers, Rahul - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Who should use SNAPSHOT when? RE: The Future of the Release Process.
Thanks for a great e-mail Joakim. I wanted to chime in with my two cents... (I've been off the radar for a couple of months while waiting for permission to sign my ICLA; it's in now, and I'm now back to paying more careful attention to this process... forgive me if some of this has already been covered.) One of the goals I know we've expressed before (but not explicitly listed here) is that Maven's release process should lead by example: other projects (whether open-source or closed-source) who haven't put as much thought into release engineering as we have should look to us as an example of the right way to do it. IMO, the requirements around SNAPSHOT releases is an important difference between open-source requirements for the release process and closed-source requirements... In this e-mail I want to describe an alternative release process (overlapping with the one Joakim described) that never uses SNAPSHOT and which is more appropriate to some organizations, perhaps especially closed-source ones. I think we all agree that it's bad (at least a little bit) to change things at the last minute before release (whether it be source code, binaries, or even your build process); one goal of the updated release process is that we should make as few last-minute changes as possible, and, to the greatest extent possible, bless binaries. But so long as you have the word SNAPSHOT embedded into your JARs during development, you'll have to change *something* at the last second, if only to remove the word SNAPSHOT. There is another way, which is better for at least some groups some of the time. If you never used SNAPSHOT, but Maven enforced a requirement that all JARs would have build numbers embedded in them (not appearing in the file name, but appearing in JAR manifest.mf and in the deployed POM), then the release process could be as little as copying the JARs into the right place and updating some metadata to call them released. Here's another way of saying the same thing. The release process Joakim described goes like this: a) Call vote b) Wait on approval. c) Collect release information. c) 'prepare' release (occurs once) d) 'stage' release (occurs 1 or more times) e) Wait on consensus from PMC for blessed artifacts. f) 'bless' release (occurs once) As this is described, it sounds as if projects would normally spend extremely little time in step D, stage. But if Maven provided more complete build numbering support for non-SNAPSHOT builds, you could imagine the project spending their entire development life in step D. After step E a decision was made to release, in step F the blessing would occur, and development would immediately begin on 1.1 in step D... no period of time spent in SNAPSHOT, so you wouldn't need to modify your code (prepare) right before release. Although I've highlighted one big advantage of not marking code under development as SNAPSHOT, the most significant disadvantage of doing it this way is that end users might confuse SNAPSHOT releases with the real official thing. (Perhaps especially if users just copy the relevant jars out of the repository and then leave Maven behind.) This can result in unnecessary support questions from users as they (unwittingly) complain about bugs in unreleased code, and can complicated support diagnostics as the person providing support may believe that the end-user has version 1.4, when they really have a developer snapshot of 1.4, never intended for release. With that said, I think most closed-source software development organizations don't have anywhere near as much fear of end-users grabbing under-development code and calling for support, since those binaries are typically kept a secret; in that case, the advantage of adding a SNAPSHOT marker may be outweighed by the disadvantage of requiring special changes right before release. Now that I've faxed in my ICLA, [heh] one of the goals I want to pursue as a Maven developer is to make the Maven release workflow support organizations that would want to work without ever using SNAPSHOT: to make that stage step a workable healthy period in a product's lifecycle that software companies could spend most of their time in. Specifically, I think that's how we'd want to maintain things at the place where I work, and that's how most closed-source ISVs should want to maintain their software. This shouldn't make it any harder to do open-source Maven releases; the fact that you *could* spend months or even years in step D doesn't mean that *we* should do so, or that we will. But I think a lot of users will benefit from a richer staging period, so it's worth putting in time and energy to make it really robust, IMO. -Dan ___ Notice: This email message, together with any attachments, may contain information of BEA Systems, Inc., its subsidiaries and affiliated entities, that
Re: Who should use SNAPSHOT when? RE: The Future of the Release Process.
Hi Dan, I think what you are describing is what you do at your work place, and I think it might be good if you could give us a full description of your system for folks here to help them understand exactly what it is you're talking about. I probably know a little more about it then most here but I still don't entirely grok the workflow you're describing. Maybe some simple examples of how artifacts names would change through the workflow you're describing versus our current approach would be helpful. Anything that makes releases easier is a good thing. Thanks, Jason. On 13 Dec 06, at 10:11 PM 13 Dec 06, Dan Fabulich wrote: Thanks for a great e-mail Joakim. I wanted to chime in with my two cents... (I've been off the radar for a couple of months while waiting for permission to sign my ICLA; it's in now, and I'm now back to paying more careful attention to this process... forgive me if some of this has already been covered.) One of the goals I know we've expressed before (but not explicitly listed here) is that Maven's release process should lead by example: other projects (whether open-source or closed-source) who haven't put as much thought into release engineering as we have should look to us as an example of the right way to do it. IMO, the requirements around SNAPSHOT releases is an important difference between open-source requirements for the release process and closed-source requirements... In this e-mail I want to describe an alternative release process (overlapping with the one Joakim described) that never uses SNAPSHOT and which is more appropriate to some organizations, perhaps especially closed-source ones. I think we all agree that it's bad (at least a little bit) to change things at the last minute before release (whether it be source code, binaries, or even your build process); one goal of the updated release process is that we should make as few last-minute changes as possible, and, to the greatest extent possible, bless binaries. But so long as you have the word SNAPSHOT embedded into your JARs during development, you'll have to change *something* at the last second, if only to remove the word SNAPSHOT. There is another way, which is better for at least some groups some of the time. If you never used SNAPSHOT, but Maven enforced a requirement that all JARs would have build numbers embedded in them (not appearing in the file name, but appearing in JAR manifest.mf and in the deployed POM), then the release process could be as little as copying the JARs into the right place and updating some metadata to call them released. Here's another way of saying the same thing. The release process Joakim described goes like this: a) Call vote b) Wait on approval. c) Collect release information. c) 'prepare' release (occurs once) d) 'stage' release (occurs 1 or more times) e) Wait on consensus from PMC for blessed artifacts. f) 'bless' release (occurs once) As this is described, it sounds as if projects would normally spend extremely little time in step D, stage. But if Maven provided more complete build numbering support for non-SNAPSHOT builds, you could imagine the project spending their entire development life in step D. After step E a decision was made to release, in step F the blessing would occur, and development would immediately begin on 1.1 in step D... no period of time spent in SNAPSHOT, so you wouldn't need to modify your code (prepare) right before release. Although I've highlighted one big advantage of not marking code under development as SNAPSHOT, the most significant disadvantage of doing it this way is that end users might confuse SNAPSHOT releases with the real official thing. (Perhaps especially if users just copy the relevant jars out of the repository and then leave Maven behind.) This can result in unnecessary support questions from users as they (unwittingly) complain about bugs in unreleased code, and can complicated support diagnostics as the person providing support may believe that the end-user has version 1.4, when they really have a developer snapshot of 1.4, never intended for release. With that said, I think most closed-source software development organizations don't have anywhere near as much fear of end-users grabbing under-development code and calling for support, since those binaries are typically kept a secret; in that case, the advantage of adding a SNAPSHOT marker may be outweighed by the disadvantage of requiring special changes right before release. Now that I've faxed in my ICLA, [heh] one of the goals I want to pursue as a Maven developer is to make the Maven release workflow support organizations that would want to work without ever using SNAPSHOT: to make that stage step a workable healthy period in a product's lifecycle that software companies could spend most of their time in. Specifically, I think that's how we'd want to
