+1 to deploying NARs to Maven. If it becomes a problem in the future, we could revisit but keeping the maven configuration as simple as possible is a huge plus.
-Joey On Thu, Jan 8, 2015 at 8:07 PM, Joe Witt <[email protected]> wrote: > I agree we need to be mindful of this. That said I am not actually sure > there is a problem. > > To provide some specific data to consider here are the Nars and their sizes > today (smallest to largest): > > 29K > ./standard-services/standard-services-api-nar/target/standard-services-api-nar-0.0.1-SNAPSHOT.nar > 158K > ./volatile-provenance-repository-bundle/nar/target/volatile-provenance-repository-nar-0.0.1-SNAPSHOT.nar > 537K > ./standard-services/ssl-context-bundle/nar/target/ssl-context-service-nar-0.0.1-SNAPSHOT.nar > 628K > ./standard-services/distributed-cache-services-bundle/distributed-cache-services-nar/target/distributed-cache-services-nar-0.0.1-SNAPSHOT.nar > 1.1M ./hadoop-bundle/nar/target/hadoop-nar-0.0.1-SNAPSHOT.nar > 4.3M > ./monitor-threshold-bundle/nar/target/monitor-threshold-nar-0.0.1-SNAPSHOT.nar > 4.4M > ./update-attribute-bundle/nar/target/update-attribute-nar-0.0.1-SNAPSHOT.nar > 4.5M ./jetty-bundle/target/nifi-jetty-bundle-0.0.1-SNAPSHOT.nar > 4.6M > ./persistent-provenance-repository-bundle/nar/target/persistent-provenance-repository-nar-0.0.1-SNAPSHOT.nar > 12M ./kafka-bundle/kafka-nar/target/kafka-nar-0.0.1-SNAPSHOT.nar > 12M ./standard-bundle/nar/target/nifi-standard-nar-0.0.1-SNAPSHOT.nar > 26M > ./hadoop-libraries-bundle/nar/target/hadoop-libraries-nar-0.0.1-SNAPSHOT.nar > 28M ./framework-bundle/nar/target/nifi-framework-nar-0.0.1-SNAPSHOT.nar > > Having the nars in a proper central repository would definitely make easier > for people to build their own assemblies of NiFi. We definitely want all > the Jar artifacts there. > > I am not convinced the added complexity (of modifying the build to > distinguish) is necessary. > > A quick look through Maven central for things like Wars (which is a very > analogous concept to Nars) suggests what we'd be doing is not at all > uncommon and the sizes are reasonable as well. > > I am of the view that we go as plain/keep it simple as we can here and if > it becomes an identifiable problem then we address it. > > Thanks > Joe > > On Thu, Jan 8, 2015 at 8:19 PM, Benson Margulies <[email protected]> > wrote: > >> To expand, >> >> On the one hand, people dump an amazing variety of crud onto Central. >> There's no special reason for NiFi to have more of a conscience than anyone >> else. >> >> On the other hand, if you want to use Maven to build things but not have >> them push to central, you control: >> >> the install plugin >> the deploy plugin >> >> and the 'attach' parameters of some other plugins, like the war plugin and >> the assembly plugin. Roughly, if you want to avoid treating the primary >> artifact of a project as a maven artifact, you have to mess with install >> and deploy. >> >> It would be good to sort this out before tackling any other release issues. >> >> >> >> On Thu, Jan 8, 2015 at 6:32 PM, Benson Margulies <[email protected]> >> wrote: >> >> > If you don't want to release it you need to not deploy it. That is a pom >> > change unrelated to the staging repo. >> > On Jan 8, 2015 5:37 PM, "Adam Taft" <[email protected]> wrote: >> > >> >> Benson, >> >> >> >> Quite a bit of NIFI's artifacts are more like application bundles of >> other >> >> common libraries. Most of the NIFI nars aren't really useful for >> >> application developers to bind to in their own code. These nars are >> more >> >> part of the application and less useful as a shared common library. >> >> >> >> One of my concerns was accidentally releasing unneeded nars into maven >> >> central. It sounds like the "purgatory" area solves this? Can you >> speak >> >> to this a little more? >> >> >> >> There's been discussion on whether we care that application nars escape >> >> into maven central - I'm not trying to weigh in on that topic (though, I >> >> think you can read my thoughts between the lines). I'm just asking for >> >> more clarity on the options when using the maven release plugin. >> >> Purgatory >> >> might very well allow us to have our cake and eat it too. >> >> >> >> Thanks, >> >> >> >> Adam >> >> >> >> >> >> On Thu, Jan 8, 2015 at 5:07 PM, Benson Margulies <[email protected] >> > >> >> wrote: >> >> >> >> > On Thu, Jan 8, 2015 at 4:53 PM, Joe Witt <[email protected]> wrote: >> >> > >> >> > > Benson, >> >> > > >> >> > > Well I think we have all the groundwork laid as far as I know for >> the >> >> > > release plugin. To be honest though (speaking for myself at least) >> >> I'm >> >> > not >> >> > > sure what the steps would be that the release plugin would do. I'd >> >> love >> >> > it >> >> > > to be as you describe but am also concerned about how much we'd be >> >> > bugging >> >> > > you to figure that out. I looked at what other projects seem to do >> >> and >> >> > it >> >> > > seemed shockingly manual. >> >> > > >> >> > > If you don't mind laying out the steps in more detail when you have >> >> time >> >> > > would love to learn from you on it. I can also try toying around >> with >> >> > it a >> >> > > bit more off-line. >> >> > > >> >> > >> >> > I don't know what projects you're looking at, but it shouldn't look so >> >> > manual for any project that has a Maven top-level build. >> >> > >> >> > Here's the basic workflow of the maven-release-plugin: >> >> > >> >> > release:prepare: >> >> > >> >> > 1. edit poms to contain 'release version'. >> >> > 2. make a commit. >> >> > 3. make a tag. >> >> > 4. edit poms to contain 'next snapshot' >> >> > 5. make a commit. >> >> > 6. push the whole business (unless you turn that off). >> >> > >> >> > release:perform: >> >> > >> >> > 7. check out from tag in target/checkout (using extra git clone) >> >> > 8. build >> >> > 9. deploy results to target of deploymentManagement >> >> > >> >> > Now comes the next trick. >> >> > >> >> > Apache runs a copy of Sonatype Nexus, that serves as a 'staging >> >> > repository'. So, when step 9 pushes things to the repository, it's >> >> pushing >> >> > them to a special 'purgatory' staging area. That area is available for >> >> > people to look at for testing, but does not propagate along to Maven >> >> > central. >> >> > >> >> > Given all this, then we add one ingredient: an additional maven module >> >> that >> >> > uses the maven-assembly-plugin to build a tarball that satisfies the >> >> > requirements of an Apache source release. This is not the same thing >> as >> >> > what the maven-source-plugin does at all. Since it will have >> >> > <attach>true</attach>, the resulting tarball swims upstream to the >> >> staging >> >> > repo with everything else. >> >> > >> >> > The vote email points people to that location, and supplies a sha1 for >> >> the >> >> > source ball. Voters download the source release from the staging repo, >> >> do >> >> > what they need to do, and vote. Three +1's later, and then you get to >> >> > publish. >> >> > >> >> > Does this help? The only manual steps are really copying to svn to >> push >> >> to >> >> > the Apache dist area. CXF and all the Maven plugins are examples. >> >> > >> >> > >> >> > >> >> > >> >> > > >> >> > > Thanks >> >> > > Joe >> >> > > >> >> > > On Thu, Jan 8, 2015 at 4:49 PM, Benson Margulies < >> >> [email protected]> >> >> > > wrote: >> >> > > >> >> > > > On Thu, Jan 8, 2015 at 4:41 PM, Joe Witt <[email protected]> >> >> wrote: >> >> > > > >> >> > > > > Benson, >> >> > > > > >> >> > > > > Yes that would be much appreciated. >> >> > > > > >> >> > > > > Here is a rough gamplan of what I *think* will be needed to do >> the >> >> > > > release: >> >> > > > > >> >> > > > > - Branch from develop against a ticket for the overall release >> >> > process >> >> > > > > - Update the version of all the artifacts and dependencies to >> >> > > > > 0.0.1-RC1-incubating or something like that >> >> > > > > - Create the tar.gz of the clean source tree. Sign that and >> place >> >> > both >> >> > > > the >> >> > > > > signed file and the tar.gz into my people's dir (if i have one >> of >> >> > > those) >> >> > > > > - Notify folks that this is available so they can pull it and >> >> attempt >> >> > > to >> >> > > > > build from that and validate the signature >> >> > > > > - Once folks agree it is good to merge that to master >> >> > > > > - Create a tar.gz of that and a signed file. Call a vote within >> >> the >> >> > > > team. >> >> > > > > If that is good call a vote within IPMC. >> >> > > > > - IF NO - fix whatever is wrong. >> >> > > > > - If all good then do a maven deploy of the binary artifacts, >> >> source >> >> > > > > bundles, and javadocs >> >> > > > > - Upload the tar.gz of source and the signed file to our dist >> dir. >> >> > And >> >> > > > > upload convenience binary build tar.gz and zip with sigs to our >> >> dist >> >> > > > > folder. Then create links to these from our downloads page. >> >> > > > > - Update JIRA that the release is closed. >> >> > > > > - Wash/Rinse/Repeat >> >> > > > > >> >> > > > > <note this all is with the understanding that we've done all the >> >> > > > necessary >> >> > > > > review of dependencies, licenses, properly documented them, have >> >> our >> >> > > > > ECCN/encryption stuff properly filed> >> >> > > > > >> >> > > > >> >> > > > Sure, except that I think we can make the maven-release-plugin do >> a >> >> lot >> >> > > of >> >> > > > the work. >> >> > > > >> >> > > > The idea is that you run the release plugin, and it delivers all >> >> these >> >> > > > goodies to repository.apache.org. Then you call the vote. If the >> >> vote >> >> > > > passes, you (a) push the promote button to push to Maven Central, >> >> and >> >> > (b) >> >> > > > check the official source release tarball into the svn area for >> >> > > > distribution. >> >> > > > >> >> > > > I will take a whack at a PR for point (a). >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > > >> >> > > > > Might be missing a detail or two but does that sound roughly on >> >> the >> >> > > right >> >> > > > > rack? >> >> > > > > >> >> > > > > I had though the maven release plugin would be useful but looks >> >> like >> >> > > > we'll >> >> > > > > just be able to use the versions plugin to update them and can >> do >> >> the >> >> > > > other >> >> > > > > simple steps manually. >> >> > > > > >> >> > > > > Thanks >> >> > > > > Joe >> >> > > > > >> >> > > > > On Thu, Jan 8, 2015 at 3:46 PM, Benson Margulies < >> >> > > [email protected]> >> >> > > > > wrote: >> >> > > > > >> >> > > > > > Typically people set the maven assembly plugin for this and >> >> include >> >> > > it >> >> > > > in >> >> > > > > > the build. Would you like me to pitch this in? >> >> > > > > > >> >> > > > > > >> >> > > > > > On Thu, Jan 8, 2015 at 2:48 PM, Mike Drob < >> [email protected]> >> >> > > wrote: >> >> > > > > > >> >> > > > > > > On Wed, Jan 7, 2015 at 9:22 PM, Joe Witt < >> [email protected]> >> >> > > wrote: >> >> > > > > > > >> >> > > > > > > > Josh >> >> > > > > > > > >> >> > > > > > > > Awesome. >> >> > > > > > > > >> >> > > > > > > > Honestly I believe we're good on the LICENSE, NOTICE, >> >> > DISCLAIMER, >> >> > > > > > README. >> >> > > > > > > > All dependencies have been reviewed and checked for ASLv2 >> >> > > > > > compatibility. >> >> > > > > > > > >> >> > > > > > > > Will submit the infra ticket now for the dist/ path for >> keys >> >> > > files >> >> > > > > and >> >> > > > > > > > such. >> >> > > > > > > > >> >> > > > > > > > My guess is that the release artifact should be a tarball >> of >> >> > all >> >> > > > > > source. >> >> > > > > > > > Could we literally just package up a clean source tree? >> >> Anyone >> >> > > > else >> >> > > > > > have >> >> > > > > > > > views on this? >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > > git archive -o [nifi-0.0.1-incubating.tar.gz] [release-tag] >> >> is a >> >> > > > > > perfectly >> >> > > > > > > reasonable thing to do. >> >> > > > > > > >> >> > > > > > > > >> >> > > > > > > > Ideally with this release we'd do it all properly >> including >> >> > maven >> >> > > > > > > > artifacts, sources/javadocs, and so on. The Maven build >> >> does >> >> > > > already >> >> > > > > > > > operate now off a single command at the root to build >> >> > everything >> >> > > > > > > > (build-order is gone) and inherits from the apache parent. >> >> > > > > > > > >> >> > > > > > > > Will need to incorporate RAT. >> >> > > > > > > > >> >> > > > > > > > Thanks for all that - definitely gives some stuff to work >> on >> >> > and >> >> > > > look >> >> > > > > > > into. >> >> > > > > > > > >> >> > > > > > > > Thanks >> >> > > > > > > > Joe >> >> > > > > > > > >> >> > > > > > > > On Wed, Jan 7, 2015 at 10:10 PM, Josh Elser < >> >> > > [email protected]> >> >> > > > > > > wrote: >> >> > > > > > > > >> >> > > > > > > > > Regardless of what you call it, writing down exactly >> what >> >> was >> >> > > > done >> >> > > > > to >> >> > > > > > > > make >> >> > > > > > > > > a RC is extremely important early on (I know that I sure >> >> > can't >> >> > > > > > remember >> >> > > > > > > > > what I did last week, much less last release). I'm not >> >> sure >> >> > if >> >> > > > > > release >> >> > > > > > > > > guide is formally defined somewhere, but enough >> >> information >> >> > > that >> >> > > > > any >> >> > > > > > > > other >> >> > > > > > > > > committer can come by after "you get hit by a train" and >> >> > make a >> >> > > > > > release >> >> > > > > > > > is >> >> > > > > > > > > extremely important. Using the CMS and making a page on >> >> the >> >> > > > website >> >> > > > > > for >> >> > > > > > > > > this is extremely easy to do, IMO. >> >> > > > > > > > > >> >> > > > > > > > > Another concern for how to actually make a release is >> the >> >> > type >> >> > > of >> >> > > > > > > > > packaging that is released. I not talking about the >> >> > > source/binary >> >> > > > > > > release >> >> > > > > > > > > here, literally "what is Apache NiFi 0.0.1-incubating"? >> Is >> >> > it a >> >> > > > > > > tarball? >> >> > > > > > > > > Are there jars in Maven? etc. A tarball is pretty easy >> to >> >> > make, >> >> > > > and >> >> > > > > > > > likely >> >> > > > > > > > > the closest to what the build-order.sh script already >> does >> >> > > (with >> >> > > > > > > Maven's >> >> > > > > > > > > help). I'm not sure if maven-released artifacts are >> quite >> >> as >> >> > > > > > important >> >> > > > > > > at >> >> > > > > > > > > this point -- if it's not, punting on that will help. >> >> If/when >> >> > > you >> >> > > > > get >> >> > > > > > > to >> >> > > > > > > > > that point, look into the apache parent pom[1]. It is >> >> > extremely >> >> > > > > well >> >> > > > > > > > > automated to use the existing infrastructure to do it >> all >> >> for >> >> > > > you. >> >> > > > > > > > > >> >> > > > > > > > > Without looking at official documentation (which is me >> >> being >> >> > > > lazy, >> >> > > > > > > > sorry), >> >> > > > > > > > > some other things that will need to be thought about: >> >> > > > > > > > > >> >> > > > > > > > > Start with the releases page [2] >> >> > > > > > > > > >> >> > > > > > > > > * You *must* include a source artifact. It cannot have >> >> > > binaries. >> >> > > > No >> >> > > > > > > Java >> >> > > > > > > > > class files, no jars. A user must be able to download >> the >> >> > > source >> >> > > > > > > artifact >> >> > > > > > > > > and build it all themselves. Artifacts which include >> >> binaries >> >> > > are >> >> > > > > for >> >> > > > > > > > > user-convenience only and can optionally be provided as >> >> well. >> >> > > > > > > > > >> >> > > > > > > > > * LICENSE, NOTICE and README must all be included in the >> >> top >> >> > > > level >> >> > > > > of >> >> > > > > > > the >> >> > > > > > > > > artifact. The NOTICE is the most painful and must >> include >> >> any >> >> > > > > > required >> >> > > > > > > > > third-party notices. [3] >> >> > > > > > > > > >> >> > > > > > > > > * You will need to audit your dependencies to make sure >> >> that >> >> > > they >> >> > > > > are >> >> > > > > > > > > ASLv2 compatible [4] >> >> > > > > > > > > >> >> > > > > > > > > * Releases must be cryptographically signed (PGP) [5]. >> >> Your >> >> > > > public >> >> > > > > > key >> >> > > > > > > > > should be published to known sites (e.g. pgp.mit.edu) >> and >> >> > must >> >> > > > be >> >> > > > > > > listed >> >> > > > > > > > > in NiFi's KEYS file [6] (which does not yet exist, >> >> probably >> >> > > needs >> >> > > > > an >> >> > > > > > > > infra >> >> > > > > > > > > ticket to create your dist/ directory?). >> >> > > > > > > > > >> >> > > > > > > > > * Verify that every source file contain the proper ASL >> >> > header. >> >> > > > The >> >> > > > > > > > > release-audit-tool[7] (`mvn apache-rat:check`) is a >> >> wonderful >> >> > > > tool >> >> > > > > > that >> >> > > > > > > > can >> >> > > > > > > > > (and should, IMO) be integrated into your build process >> to >> >> > > > prevent >> >> > > > > > > people >> >> > > > > > > > > from accidentally committing new files between releases >> >> that >> >> > do >> >> > > > not >> >> > > > > > > have >> >> > > > > > > > > the correct header. >> >> > > > > > > > > >> >> > > > > > > > > And suddenly, this is really long :). Short answer: >> decide >> >> > what >> >> > > > the >> >> > > > > > > > > release should look like (just a tarball?), start >> vetting >> >> > your >> >> > > > > source >> >> > > > > > > > code >> >> > > > > > > > > for license headers, and looking into NiFi's >> dependencies >> >> and >> >> > > > their >> >> > > > > > > > > licenses. >> >> > > > > > > > > >> >> > > > > > > > > [1] http://maven.apache.org/pom/asf/ >> >> > > > > > > > > [2] http://www.apache.org/dev/release.html >> >> > > > > > > > > [3] http://www.apache.org/legal/src-headers.html >> >> > > > > > > > > [4] >> http://www.apache.org/legal/resolved.html#category-x >> >> > > > > > > > > [5] http://www.apache.org/dev/release-signing.html >> >> > > > > > > > > [6] http://www.apache.org/dist/incubator/nifi/KEYS >> >> > > > > > > > > [7] http://creadur.apache.org/rat/ >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > Tony Kurc wrote: >> >> > > > > > > > > >> >> > > > > > > > >> I read the guide Joe linked and a lot of the sticky >> parts >> >> > are >> >> > > > > marked >> >> > > > > > > > >> "TODO" >> >> > > > > > > > >> and it looks like a work in progress >> >> > > > > > > > >> >> >> > > > > > > > >> "Podlings can short circuit this process by starting >> out >> >> > with >> >> > > > > > written >> >> > > > > > > > >> release documentation. It is strongly recommended that >> >> > > Podlings >> >> > > > > > invest >> >> > > > > > > > >> time >> >> > > > > > > > >> looking at the best practices recommended in this >> >> document. >> >> > By >> >> > > > > > > selection >> >> > > > > > > > >> and modification, sections of this document can be used >> >> to >> >> > > > quickly >> >> > > > > > and >> >> > > > > > > > >> easily bootstrap a release guide. " >> >> > > > > > > > >> >> >> > > > > > > > >> Is step one putting together a release guide? >> >> > > > > > > > >> >> >> > > > > > > > >> >> >> > > > > > > > >> >> >> > > > > > > > >> On Wed, Jan 7, 2015 at 9:26 PM, Joe Witt< >> >> [email protected] >> >> > > >> >> > > > > > wrote: >> >> > > > > > > > >> >> >> > > > > > > > >> Hello >> >> > > > > > > > >>> >> >> > > > > > > > >>> Just wanted to stir this one up a bit. Looks like all >> >> > > tickets >> >> > > > > > pegged >> >> > > > > > > > as >> >> > > > > > > > >>> 0.0.1 are resolved (2 remain as of now but seem likely >> >> to >> >> > be >> >> > > > > > resolved >> >> > > > > > > > >>> shortly based on their comments). So working through >> >> the >> >> > > > release >> >> > > > > > > steps >> >> > > > > > > > >>> available on apache.org and via the link Brock sent. >> >> > > > > > > > >>> >> >> > > > > > > > >>> Anyone interested in this part of the process or who >> has >> >> > > advice >> >> > > > > to >> >> > > > > > > help >> >> > > > > > > > >>> us >> >> > > > > > > > >>> avoid landmines we're happy to hear it. >> >> > > > > > > > >>> >> >> > > > > > > > >>> Thanks >> >> > > > > > > > >>> Joe >> >> > > > > > > > >>> >> >> > > > > > > > >>> On Thu, Dec 18, 2014 at 1:31 PM, Joe Witt< >> >> > [email protected] >> >> > > > >> >> > > > > > > wrote: >> >> > > > > > > > >>> >> >> > > > > > > > >>> Thanks Brock this is very helpful. >> >> > > > > > > > >>>> >> >> > > > > > > > >>>> On Thu, Dec 18, 2014 at 1:27 PM, Brock Noland< >> >> > > > > [email protected]> >> >> > > > > > > > >>>> >> >> > > > > > > > >>> wrote: >> >> > > > > > > > >>> >> >> > > > > > > > >>>> Hi, >> >> > > > > > > > >>>>> >> >> > > > > > > > >>>>> This is a decent guide which can be copied: >> >> > > > > > > > >>>>> >> >> > > > > > > > >>>>> http://mrunit.apache.org/pmc/how_to_release.html >> >> > > > > > > > >>>>> >> >> > > > > > > > >>>>> Brock >> >> > > > > > > > >>>>> >> >> > > > > > > > >>>>> On Thu, Dec 18, 2014 at 10:17 AM, Joe Witt< >> >> > > > [email protected]> >> >> > > > > > > > wrote: >> >> > > > > > > > >>>>> >> >> > > > > > > > >>>>>> Folks, >> >> > > > > > > > >>>>>> >> >> > > > > > > > >>>>>> Looking at the tickets that remain which are >> >> presently >> >> > > tied >> >> > > > to >> >> > > > > > > 0.0.1 >> >> > > > > > > > >>>>>> >> >> > > > > > > > >>>>> we're >> >> > > > > > > > >>>>> >> >> > > > > > > > >>>>>> probably 1-2 weeks out from this initial release. >> >> Can >> >> > you >> >> > > > > > provide >> >> > > > > > > > >>>>>> >> >> > > > > > > > >>>>> some >> >> > > > > > > > >>> >> >> > > > > > > > >>>> pointers/references or pointers on how to get this >> ball >> >> > > > rolling >> >> > > > > > and >> >> > > > > > > > >>>>>> >> >> > > > > > > > >>>>> any >> >> > > > > > > > >>> >> >> > > > > > > > >>>> rocks we must move before going down this path? >> >> > > > > > > > >>>>>> >> >> > > > > > > > >>>>>> >> >> > http://incubator.apache.org/guides/releasemanagement.html >> >> > > > > > > > >>>>>> >> >> > > > > > > > >>>>>> That link seems really helpful but has a lot of >> TODOs >> >> > for >> >> > > > > areas >> >> > > > > > > > which >> >> > > > > > > > >>>>>> >> >> > > > > > > > >>>>> need >> >> > > > > > > > >>>>> >> >> > > > > > > > >>>>>> more explanation. >> >> > > > > > > > >>>>>> >> >> > > > > > > > >>>>>> >> >> > > > > > > > >>>>>> Thanks >> >> > > > > > > > >>>>>> Joe >> >> > > > > > > > >>>>>> >> >> > > > > > > > >>>>>> >> >> > > > > > > > >> >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> > >> -- Joey Echeverria
