I agree we need to be mindful of this. That said I am not actually sure there is a problem.
To provide some specific data to consider here are the Nars and their sizes today (smallest to largest): 29K ./standard-services/standard-services-api-nar/target/standard-services-api-nar-0.0.1-SNAPSHOT.nar 158K ./volatile-provenance-repository-bundle/nar/target/volatile-provenance-repository-nar-0.0.1-SNAPSHOT.nar 537K ./standard-services/ssl-context-bundle/nar/target/ssl-context-service-nar-0.0.1-SNAPSHOT.nar 628K ./standard-services/distributed-cache-services-bundle/distributed-cache-services-nar/target/distributed-cache-services-nar-0.0.1-SNAPSHOT.nar 1.1M ./hadoop-bundle/nar/target/hadoop-nar-0.0.1-SNAPSHOT.nar 4.3M ./monitor-threshold-bundle/nar/target/monitor-threshold-nar-0.0.1-SNAPSHOT.nar 4.4M ./update-attribute-bundle/nar/target/update-attribute-nar-0.0.1-SNAPSHOT.nar 4.5M ./jetty-bundle/target/nifi-jetty-bundle-0.0.1-SNAPSHOT.nar 4.6M ./persistent-provenance-repository-bundle/nar/target/persistent-provenance-repository-nar-0.0.1-SNAPSHOT.nar 12M ./kafka-bundle/kafka-nar/target/kafka-nar-0.0.1-SNAPSHOT.nar 12M ./standard-bundle/nar/target/nifi-standard-nar-0.0.1-SNAPSHOT.nar 26M ./hadoop-libraries-bundle/nar/target/hadoop-libraries-nar-0.0.1-SNAPSHOT.nar 28M ./framework-bundle/nar/target/nifi-framework-nar-0.0.1-SNAPSHOT.nar Having the nars in a proper central repository would definitely make easier for people to build their own assemblies of NiFi. We definitely want all the Jar artifacts there. I am not convinced the added complexity (of modifying the build to distinguish) is necessary. A quick look through Maven central for things like Wars (which is a very analogous concept to Nars) suggests what we'd be doing is not at all uncommon and the sizes are reasonable as well. I am of the view that we go as plain/keep it simple as we can here and if it becomes an identifiable problem then we address it. Thanks Joe On Thu, Jan 8, 2015 at 8:19 PM, Benson Margulies <[email protected]> wrote: > To expand, > > On the one hand, people dump an amazing variety of crud onto Central. > There's no special reason for NiFi to have more of a conscience than anyone > else. > > On the other hand, if you want to use Maven to build things but not have > them push to central, you control: > > the install plugin > the deploy plugin > > and the 'attach' parameters of some other plugins, like the war plugin and > the assembly plugin. Roughly, if you want to avoid treating the primary > artifact of a project as a maven artifact, you have to mess with install > and deploy. > > It would be good to sort this out before tackling any other release issues. > > > > On Thu, Jan 8, 2015 at 6:32 PM, Benson Margulies <[email protected]> > wrote: > > > If you don't want to release it you need to not deploy it. That is a pom > > change unrelated to the staging repo. > > On Jan 8, 2015 5:37 PM, "Adam Taft" <[email protected]> wrote: > > > >> Benson, > >> > >> Quite a bit of NIFI's artifacts are more like application bundles of > other > >> common libraries. Most of the NIFI nars aren't really useful for > >> application developers to bind to in their own code. These nars are > more > >> part of the application and less useful as a shared common library. > >> > >> One of my concerns was accidentally releasing unneeded nars into maven > >> central. It sounds like the "purgatory" area solves this? Can you > speak > >> to this a little more? > >> > >> There's been discussion on whether we care that application nars escape > >> into maven central - I'm not trying to weigh in on that topic (though, I > >> think you can read my thoughts between the lines). I'm just asking for > >> more clarity on the options when using the maven release plugin. > >> Purgatory > >> might very well allow us to have our cake and eat it too. > >> > >> Thanks, > >> > >> Adam > >> > >> > >> On Thu, Jan 8, 2015 at 5:07 PM, Benson Margulies <[email protected] > > > >> wrote: > >> > >> > On Thu, Jan 8, 2015 at 4:53 PM, Joe Witt <[email protected]> wrote: > >> > > >> > > Benson, > >> > > > >> > > Well I think we have all the groundwork laid as far as I know for > the > >> > > release plugin. To be honest though (speaking for myself at least) > >> I'm > >> > not > >> > > sure what the steps would be that the release plugin would do. I'd > >> love > >> > it > >> > > to be as you describe but am also concerned about how much we'd be > >> > bugging > >> > > you to figure that out. I looked at what other projects seem to do > >> and > >> > it > >> > > seemed shockingly manual. > >> > > > >> > > If you don't mind laying out the steps in more detail when you have > >> time > >> > > would love to learn from you on it. I can also try toying around > with > >> > it a > >> > > bit more off-line. > >> > > > >> > > >> > I don't know what projects you're looking at, but it shouldn't look so > >> > manual for any project that has a Maven top-level build. > >> > > >> > Here's the basic workflow of the maven-release-plugin: > >> > > >> > release:prepare: > >> > > >> > 1. edit poms to contain 'release version'. > >> > 2. make a commit. > >> > 3. make a tag. > >> > 4. edit poms to contain 'next snapshot' > >> > 5. make a commit. > >> > 6. push the whole business (unless you turn that off). > >> > > >> > release:perform: > >> > > >> > 7. check out from tag in target/checkout (using extra git clone) > >> > 8. build > >> > 9. deploy results to target of deploymentManagement > >> > > >> > Now comes the next trick. > >> > > >> > Apache runs a copy of Sonatype Nexus, that serves as a 'staging > >> > repository'. So, when step 9 pushes things to the repository, it's > >> pushing > >> > them to a special 'purgatory' staging area. That area is available for > >> > people to look at for testing, but does not propagate along to Maven > >> > central. > >> > > >> > Given all this, then we add one ingredient: an additional maven module > >> that > >> > uses the maven-assembly-plugin to build a tarball that satisfies the > >> > requirements of an Apache source release. This is not the same thing > as > >> > what the maven-source-plugin does at all. Since it will have > >> > <attach>true</attach>, the resulting tarball swims upstream to the > >> staging > >> > repo with everything else. > >> > > >> > The vote email points people to that location, and supplies a sha1 for > >> the > >> > source ball. Voters download the source release from the staging repo, > >> do > >> > what they need to do, and vote. Three +1's later, and then you get to > >> > publish. > >> > > >> > Does this help? The only manual steps are really copying to svn to > push > >> to > >> > the Apache dist area. CXF and all the Maven plugins are examples. > >> > > >> > > >> > > >> > > >> > > > >> > > Thanks > >> > > Joe > >> > > > >> > > On Thu, Jan 8, 2015 at 4:49 PM, Benson Margulies < > >> [email protected]> > >> > > wrote: > >> > > > >> > > > On Thu, Jan 8, 2015 at 4:41 PM, Joe Witt <[email protected]> > >> wrote: > >> > > > > >> > > > > Benson, > >> > > > > > >> > > > > Yes that would be much appreciated. > >> > > > > > >> > > > > Here is a rough gamplan of what I *think* will be needed to do > the > >> > > > release: > >> > > > > > >> > > > > - Branch from develop against a ticket for the overall release > >> > process > >> > > > > - Update the version of all the artifacts and dependencies to > >> > > > > 0.0.1-RC1-incubating or something like that > >> > > > > - Create the tar.gz of the clean source tree. Sign that and > place > >> > both > >> > > > the > >> > > > > signed file and the tar.gz into my people's dir (if i have one > of > >> > > those) > >> > > > > - Notify folks that this is available so they can pull it and > >> attempt > >> > > to > >> > > > > build from that and validate the signature > >> > > > > - Once folks agree it is good to merge that to master > >> > > > > - Create a tar.gz of that and a signed file. Call a vote within > >> the > >> > > > team. > >> > > > > If that is good call a vote within IPMC. > >> > > > > - IF NO - fix whatever is wrong. > >> > > > > - If all good then do a maven deploy of the binary artifacts, > >> source > >> > > > > bundles, and javadocs > >> > > > > - Upload the tar.gz of source and the signed file to our dist > dir. > >> > And > >> > > > > upload convenience binary build tar.gz and zip with sigs to our > >> dist > >> > > > > folder. Then create links to these from our downloads page. > >> > > > > - Update JIRA that the release is closed. > >> > > > > - Wash/Rinse/Repeat > >> > > > > > >> > > > > <note this all is with the understanding that we've done all the > >> > > > necessary > >> > > > > review of dependencies, licenses, properly documented them, have > >> our > >> > > > > ECCN/encryption stuff properly filed> > >> > > > > > >> > > > > >> > > > Sure, except that I think we can make the maven-release-plugin do > a > >> lot > >> > > of > >> > > > the work. > >> > > > > >> > > > The idea is that you run the release plugin, and it delivers all > >> these > >> > > > goodies to repository.apache.org. Then you call the vote. If the > >> vote > >> > > > passes, you (a) push the promote button to push to Maven Central, > >> and > >> > (b) > >> > > > check the official source release tarball into the svn area for > >> > > > distribution. > >> > > > > >> > > > I will take a whack at a PR for point (a). > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > > >> > > > > Might be missing a detail or two but does that sound roughly on > >> the > >> > > right > >> > > > > rack? > >> > > > > > >> > > > > I had though the maven release plugin would be useful but looks > >> like > >> > > > we'll > >> > > > > just be able to use the versions plugin to update them and can > do > >> the > >> > > > other > >> > > > > simple steps manually. > >> > > > > > >> > > > > Thanks > >> > > > > Joe > >> > > > > > >> > > > > On Thu, Jan 8, 2015 at 3:46 PM, Benson Margulies < > >> > > [email protected]> > >> > > > > wrote: > >> > > > > > >> > > > > > Typically people set the maven assembly plugin for this and > >> include > >> > > it > >> > > > in > >> > > > > > the build. Would you like me to pitch this in? > >> > > > > > > >> > > > > > > >> > > > > > On Thu, Jan 8, 2015 at 2:48 PM, Mike Drob < > [email protected]> > >> > > wrote: > >> > > > > > > >> > > > > > > On Wed, Jan 7, 2015 at 9:22 PM, Joe Witt < > [email protected]> > >> > > wrote: > >> > > > > > > > >> > > > > > > > Josh > >> > > > > > > > > >> > > > > > > > Awesome. > >> > > > > > > > > >> > > > > > > > Honestly I believe we're good on the LICENSE, NOTICE, > >> > DISCLAIMER, > >> > > > > > README. > >> > > > > > > > All dependencies have been reviewed and checked for ASLv2 > >> > > > > > compatibility. > >> > > > > > > > > >> > > > > > > > Will submit the infra ticket now for the dist/ path for > keys > >> > > files > >> > > > > and > >> > > > > > > > such. > >> > > > > > > > > >> > > > > > > > My guess is that the release artifact should be a tarball > of > >> > all > >> > > > > > source. > >> > > > > > > > Could we literally just package up a clean source tree? > >> Anyone > >> > > > else > >> > > > > > have > >> > > > > > > > views on this? > >> > > > > > > > > >> > > > > > > > >> > > > > > > git archive -o [nifi-0.0.1-incubating.tar.gz] [release-tag] > >> is a > >> > > > > > perfectly > >> > > > > > > reasonable thing to do. > >> > > > > > > > >> > > > > > > > > >> > > > > > > > Ideally with this release we'd do it all properly > including > >> > maven > >> > > > > > > > artifacts, sources/javadocs, and so on. The Maven build > >> does > >> > > > already > >> > > > > > > > operate now off a single command at the root to build > >> > everything > >> > > > > > > > (build-order is gone) and inherits from the apache parent. > >> > > > > > > > > >> > > > > > > > Will need to incorporate RAT. > >> > > > > > > > > >> > > > > > > > Thanks for all that - definitely gives some stuff to work > on > >> > and > >> > > > look > >> > > > > > > into. > >> > > > > > > > > >> > > > > > > > Thanks > >> > > > > > > > Joe > >> > > > > > > > > >> > > > > > > > On Wed, Jan 7, 2015 at 10:10 PM, Josh Elser < > >> > > [email protected]> > >> > > > > > > wrote: > >> > > > > > > > > >> > > > > > > > > Regardless of what you call it, writing down exactly > what > >> was > >> > > > done > >> > > > > to > >> > > > > > > > make > >> > > > > > > > > a RC is extremely important early on (I know that I sure > >> > can't > >> > > > > > remember > >> > > > > > > > > what I did last week, much less last release). I'm not > >> sure > >> > if > >> > > > > > release > >> > > > > > > > > guide is formally defined somewhere, but enough > >> information > >> > > that > >> > > > > any > >> > > > > > > > other > >> > > > > > > > > committer can come by after "you get hit by a train" and > >> > make a > >> > > > > > release > >> > > > > > > > is > >> > > > > > > > > extremely important. Using the CMS and making a page on > >> the > >> > > > website > >> > > > > > for > >> > > > > > > > > this is extremely easy to do, IMO. > >> > > > > > > > > > >> > > > > > > > > Another concern for how to actually make a release is > the > >> > type > >> > > of > >> > > > > > > > > packaging that is released. I not talking about the > >> > > source/binary > >> > > > > > > release > >> > > > > > > > > here, literally "what is Apache NiFi 0.0.1-incubating"? > Is > >> > it a > >> > > > > > > tarball? > >> > > > > > > > > Are there jars in Maven? etc. A tarball is pretty easy > to > >> > make, > >> > > > and > >> > > > > > > > likely > >> > > > > > > > > the closest to what the build-order.sh script already > does > >> > > (with > >> > > > > > > Maven's > >> > > > > > > > > help). I'm not sure if maven-released artifacts are > quite > >> as > >> > > > > > important > >> > > > > > > at > >> > > > > > > > > this point -- if it's not, punting on that will help. > >> If/when > >> > > you > >> > > > > get > >> > > > > > > to > >> > > > > > > > > that point, look into the apache parent pom[1]. It is > >> > extremely > >> > > > > well > >> > > > > > > > > automated to use the existing infrastructure to do it > all > >> for > >> > > > you. > >> > > > > > > > > > >> > > > > > > > > Without looking at official documentation (which is me > >> being > >> > > > lazy, > >> > > > > > > > sorry), > >> > > > > > > > > some other things that will need to be thought about: > >> > > > > > > > > > >> > > > > > > > > Start with the releases page [2] > >> > > > > > > > > > >> > > > > > > > > * You *must* include a source artifact. It cannot have > >> > > binaries. > >> > > > No > >> > > > > > > Java > >> > > > > > > > > class files, no jars. A user must be able to download > the > >> > > source > >> > > > > > > artifact > >> > > > > > > > > and build it all themselves. Artifacts which include > >> binaries > >> > > are > >> > > > > for > >> > > > > > > > > user-convenience only and can optionally be provided as > >> well. > >> > > > > > > > > > >> > > > > > > > > * LICENSE, NOTICE and README must all be included in the > >> top > >> > > > level > >> > > > > of > >> > > > > > > the > >> > > > > > > > > artifact. The NOTICE is the most painful and must > include > >> any > >> > > > > > required > >> > > > > > > > > third-party notices. [3] > >> > > > > > > > > > >> > > > > > > > > * You will need to audit your dependencies to make sure > >> that > >> > > they > >> > > > > are > >> > > > > > > > > ASLv2 compatible [4] > >> > > > > > > > > > >> > > > > > > > > * Releases must be cryptographically signed (PGP) [5]. > >> Your > >> > > > public > >> > > > > > key > >> > > > > > > > > should be published to known sites (e.g. pgp.mit.edu) > and > >> > must > >> > > > be > >> > > > > > > listed > >> > > > > > > > > in NiFi's KEYS file [6] (which does not yet exist, > >> probably > >> > > needs > >> > > > > an > >> > > > > > > > infra > >> > > > > > > > > ticket to create your dist/ directory?). > >> > > > > > > > > > >> > > > > > > > > * Verify that every source file contain the proper ASL > >> > header. > >> > > > The > >> > > > > > > > > release-audit-tool[7] (`mvn apache-rat:check`) is a > >> wonderful > >> > > > tool > >> > > > > > that > >> > > > > > > > can > >> > > > > > > > > (and should, IMO) be integrated into your build process > to > >> > > > prevent > >> > > > > > > people > >> > > > > > > > > from accidentally committing new files between releases > >> that > >> > do > >> > > > not > >> > > > > > > have > >> > > > > > > > > the correct header. > >> > > > > > > > > > >> > > > > > > > > And suddenly, this is really long :). Short answer: > decide > >> > what > >> > > > the > >> > > > > > > > > release should look like (just a tarball?), start > vetting > >> > your > >> > > > > source > >> > > > > > > > code > >> > > > > > > > > for license headers, and looking into NiFi's > dependencies > >> and > >> > > > their > >> > > > > > > > > licenses. > >> > > > > > > > > > >> > > > > > > > > [1] http://maven.apache.org/pom/asf/ > >> > > > > > > > > [2] http://www.apache.org/dev/release.html > >> > > > > > > > > [3] http://www.apache.org/legal/src-headers.html > >> > > > > > > > > [4] > http://www.apache.org/legal/resolved.html#category-x > >> > > > > > > > > [5] http://www.apache.org/dev/release-signing.html > >> > > > > > > > > [6] http://www.apache.org/dist/incubator/nifi/KEYS > >> > > > > > > > > [7] http://creadur.apache.org/rat/ > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > Tony Kurc wrote: > >> > > > > > > > > > >> > > > > > > > >> I read the guide Joe linked and a lot of the sticky > parts > >> > are > >> > > > > marked > >> > > > > > > > >> "TODO" > >> > > > > > > > >> and it looks like a work in progress > >> > > > > > > > >> > >> > > > > > > > >> "Podlings can short circuit this process by starting > out > >> > with > >> > > > > > written > >> > > > > > > > >> release documentation. It is strongly recommended that > >> > > Podlings > >> > > > > > invest > >> > > > > > > > >> time > >> > > > > > > > >> looking at the best practices recommended in this > >> document. > >> > By > >> > > > > > > selection > >> > > > > > > > >> and modification, sections of this document can be used > >> to > >> > > > quickly > >> > > > > > and > >> > > > > > > > >> easily bootstrap a release guide. " > >> > > > > > > > >> > >> > > > > > > > >> Is step one putting together a release guide? > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > > >> On Wed, Jan 7, 2015 at 9:26 PM, Joe Witt< > >> [email protected] > >> > > > >> > > > > > wrote: > >> > > > > > > > >> > >> > > > > > > > >> Hello > >> > > > > > > > >>> > >> > > > > > > > >>> Just wanted to stir this one up a bit. Looks like all > >> > > tickets > >> > > > > > pegged > >> > > > > > > > as > >> > > > > > > > >>> 0.0.1 are resolved (2 remain as of now but seem likely > >> to > >> > be > >> > > > > > resolved > >> > > > > > > > >>> shortly based on their comments). So working through > >> the > >> > > > release > >> > > > > > > steps > >> > > > > > > > >>> available on apache.org and via the link Brock sent. > >> > > > > > > > >>> > >> > > > > > > > >>> Anyone interested in this part of the process or who > has > >> > > advice > >> > > > > to > >> > > > > > > help > >> > > > > > > > >>> us > >> > > > > > > > >>> avoid landmines we're happy to hear it. > >> > > > > > > > >>> > >> > > > > > > > >>> Thanks > >> > > > > > > > >>> Joe > >> > > > > > > > >>> > >> > > > > > > > >>> On Thu, Dec 18, 2014 at 1:31 PM, Joe Witt< > >> > [email protected] > >> > > > > >> > > > > > > wrote: > >> > > > > > > > >>> > >> > > > > > > > >>> Thanks Brock this is very helpful. > >> > > > > > > > >>>> > >> > > > > > > > >>>> On Thu, Dec 18, 2014 at 1:27 PM, Brock Noland< > >> > > > > [email protected]> > >> > > > > > > > >>>> > >> > > > > > > > >>> wrote: > >> > > > > > > > >>> > >> > > > > > > > >>>> Hi, > >> > > > > > > > >>>>> > >> > > > > > > > >>>>> This is a decent guide which can be copied: > >> > > > > > > > >>>>> > >> > > > > > > > >>>>> http://mrunit.apache.org/pmc/how_to_release.html > >> > > > > > > > >>>>> > >> > > > > > > > >>>>> Brock > >> > > > > > > > >>>>> > >> > > > > > > > >>>>> On Thu, Dec 18, 2014 at 10:17 AM, Joe Witt< > >> > > > [email protected]> > >> > > > > > > > wrote: > >> > > > > > > > >>>>> > >> > > > > > > > >>>>>> Folks, > >> > > > > > > > >>>>>> > >> > > > > > > > >>>>>> Looking at the tickets that remain which are > >> presently > >> > > tied > >> > > > to > >> > > > > > > 0.0.1 > >> > > > > > > > >>>>>> > >> > > > > > > > >>>>> we're > >> > > > > > > > >>>>> > >> > > > > > > > >>>>>> probably 1-2 weeks out from this initial release. > >> Can > >> > you > >> > > > > > provide > >> > > > > > > > >>>>>> > >> > > > > > > > >>>>> some > >> > > > > > > > >>> > >> > > > > > > > >>>> pointers/references or pointers on how to get this > ball > >> > > > rolling > >> > > > > > and > >> > > > > > > > >>>>>> > >> > > > > > > > >>>>> any > >> > > > > > > > >>> > >> > > > > > > > >>>> rocks we must move before going down this path? > >> > > > > > > > >>>>>> > >> > > > > > > > >>>>>> > >> > http://incubator.apache.org/guides/releasemanagement.html > >> > > > > > > > >>>>>> > >> > > > > > > > >>>>>> That link seems really helpful but has a lot of > TODOs > >> > for > >> > > > > areas > >> > > > > > > > which > >> > > > > > > > >>>>>> > >> > > > > > > > >>>>> need > >> > > > > > > > >>>>> > >> > > > > > > > >>>>>> more explanation. > >> > > > > > > > >>>>>> > >> > > > > > > > >>>>>> > >> > > > > > > > >>>>>> Thanks > >> > > > > > > > >>>>>> Joe > >> > > > > > > > >>>>>> > >> > > > > > > > >>>>>> > >> > > > > > > > >> > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > >
